2024-04-26T01:14:42.631 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T01:14:42.631 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235 --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} --name leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/orig.config.yaml 2024-04-26T01:14:42.693 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T01:14:42.801 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} email: null first_in_suite: false job_id: '7674235' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: afa1933b0fdbb6c99c947d1eda34d661d23cd327 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3448 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: afa1933b0fdbb6c99c947d1eda34d661d23cd327 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLo5ySqCtwejdXjowKaoioj1jhXs8OyavPeXWNu4uesAxD9iJ1xy2bkqb3bS9sAzr4/vsiz6LdgOzBgl8GVvhVc= smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBH0CAbYukTmxxBRGplu/YJpHfq0PpyXRbfWS68BntPvvyKiRPX6vA7kuP0E79O/udz5E9lyE7KtnL2D/nuPCT4= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApA1IpWDbjAwBm0DgsXFeZ9YfP569BS90v2+vnZ1P7gaIKfjb192gH4qyIbw7WXNbySvGj6eiqUkItuhEzV3I4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_01:02:28 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T01:14:42.801 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa; will attempt to use it 2024-04-26T01:14:42.802 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks 2024-04-26T01:14:42.802 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T01:14:42.804 INFO:teuthology.task.internal:Checking packages... 2024-04-26T01:14:42.828 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-26T01:14:42.828 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T01:14:42.828 INFO:teuthology.packaging:ref: None 2024-04-26T01:14:42.828 INFO:teuthology.packaging:tag: None 2024-04-26T01:14:42.828 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-26T01:14:42.828 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:14:42.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-26T01:14:43.080 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-26T01:14:43.082 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T01:14:43.088 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T01:14:43.089 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T01:14:43.101 INFO:teuthology.task.internal:Saving configuration 2024-04-26T01:14:43.125 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T01:14:43.131 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T01:14:43.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:09:24.805897', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLo5ySqCtwejdXjowKaoioj1jhXs8OyavPeXWNu4uesAxD9iJ1xy2bkqb3bS9sAzr4/vsiz6LdgOzBgl8GVvhVc='} 2024-04-26T01:14:43.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:09:24.807718', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBH0CAbYukTmxxBRGplu/YJpHfq0PpyXRbfWS68BntPvvyKiRPX6vA7kuP0E79O/udz5E9lyE7KtnL2D/nuPCT4='} 2024-04-26T01:14:43.199 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:09:24.809314', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApA1IpWDbjAwBm0DgsXFeZ9YfP569BS90v2+vnZ1P7gaIKfjb192gH4qyIbw7WXNbySvGj6eiqUkItuhEzV3I4='} 2024-04-26T01:14:43.199 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T01:14:43.210 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T01:14:43.210 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T01:14:43.210 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T01:14:43.210 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T01:14:43.317 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2d108f70d0>, signals=[15]) 2024-04-26T01:14:43.317 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T01:14:43.323 INFO:teuthology.task.internal:Opening connections... 2024-04-26T01:14:43.324 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:14:43.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:14:43.405 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-04-26T01:14:43.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:14:43.480 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-04-26T01:14:43.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:14:43.553 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T01:14:43.560 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-04-26T01:14:43.579 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-04-26T01:14:43.579 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-04-26T01:14:43.637 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-04-26T01:14:43.637 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-04-26T01:14:43.637 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-04-26T01:14:43.637 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:14:43.637 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:14:43.638 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:14:43.639 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-04-26T01:14:43.657 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-04-26T01:14:43.675 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-04-26T01:14:43.676 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-04-26T01:14:43.733 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:14:43.734 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:14:43.735 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:14:43.735 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:14:43.735 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-04-26T01:14:43.757 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-04-26T01:14:43.775 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-04-26T01:14:43.775 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-04-26T01:14:43.831 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-04-26T01:14:43.831 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:14:43.832 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:14:43.833 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-04-26T01:14:43.857 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T01:14:43.867 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T01:14:43.916 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T01:14:43.916 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:14:43.919 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:14:43.921 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:14:43.938 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T01:14:43.944 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T01:14:43.944 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:14:43.977 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:14:43.979 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:14:44.061 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T01:14:44.133 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T01:14:44.147 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-26T01:14:44.147 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T01:14:44.147 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T01:14:44.148 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:14:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:14:44.165 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T01:14:44.350 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:14:44.350 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T01:14:44.350 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-26T01:14:44.368 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-04-26T01:14:44.369 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:14:44.369 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T01:14:44.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum upgrade -y linux-firmware 2024-04-26T01:14:46.844 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-04-26T01:14:50.146 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 19 MB 00:02 2024-04-26T01:14:54.554 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.2 MB 00:00 2024-04-26T01:14:56.002 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-04-26T01:14:59.903 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 21 MB 00:03 2024-04-26T01:15:05.095 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-26T01:15:07.199 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-26T01:15:07.200 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-26T01:15:07.200 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-26T01:15:07.295 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T01:15:07.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:15:07.442 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T01:15:07.442 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T01:15:07.445 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-04-26T01:15:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26 01:15:22 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T01:15:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:FINISHED --2024-04-26 01:15:22-- 2024-04-26T01:15:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:Total wall clock time: 15s 2024-04-26T01:15:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:Downloaded: 1 files, 341M in 14s (23.9 MB/s) 2024-04-26T01:15:22.083 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T01:15:22.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T01:15:24.276 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying... ######################################## 2024-04-26T01:15:24.407 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing... ######################################## 2024-04-26T01:15:24.514 INFO:teuthology.orchestra.run.smithi028.stdout:Updating / installing... 2024-04-26T01:15:32.910 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T01:16:12.267 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T01:16:12.268 DEBUG:teuthology.orchestra.run.smithi028:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T01:16:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T01:16:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T01:16:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T01:16:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T01:16:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T01:16:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T01:16:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T01:16:12.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T01:16:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T01:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T01:16:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T01:16:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T01:16:12.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T01:16:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T01:16:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T01:16:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T01:16:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T01:16:12.367 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T01:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T01:16:12.369 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T01:16:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T01:16:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T01:16:12.372 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T01:16:12.373 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T01:16:12.374 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T01:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T01:16:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T01:16:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T01:16:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T01:16:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T01:16:12.380 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T01:16:12.381 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T01:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T01:16:12.383 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T01:16:12.384 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T01:16:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T01:16:12.386 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T01:16:12.387 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T01:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T01:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T01:16:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T01:16:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T01:16:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T01:16:12.393 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T01:16:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T01:16:12.395 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T01:16:12.396 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T01:16:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T01:16:12.398 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T01:16:12.399 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T01:16:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T01:16:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T01:16:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T01:16:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T01:16:12.404 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T01:16:12.405 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T01:16:12.406 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T01:16:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T01:16:12.408 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T01:16:12.408 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T01:16:12.408 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T01:16:12.408 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T01:16:12.408 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T01:16:12.409 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T01:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T01:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T01:16:12.412 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T01:16:12.413 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T01:16:12.414 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T01:16:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T01:16:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T01:16:12.417 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T01:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T01:16:12.419 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T01:16:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T01:16:12.421 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T01:16:12.422 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T01:16:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T01:16:12.424 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T01:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T01:16:12.426 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T01:16:12.427 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T01:16:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T01:16:12.429 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T01:16:12.430 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T01:16:12.431 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T01:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T01:16:12.433 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T01:16:12.434 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T01:16:12.435 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T01:16:12.436 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T01:16:12.437 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T01:16:12.438 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T01:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T01:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T01:16:12.441 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T01:16:12.442 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T01:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T01:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T01:16:12.445 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T01:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T01:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T01:16:12.448 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T01:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T01:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T01:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T01:16:12.452 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T01:16:12.453 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T01:16:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T01:16:12.455 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T01:16:12.456 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T01:16:12.457 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T01:16:12.458 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T01:16:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T01:16:12.460 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T01:16:12.461 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T01:16:12.462 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T01:16:12.463 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T01:16:12.464 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T01:16:12.465 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T01:16:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T01:16:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T01:16:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T01:16:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T01:16:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T01:16:12.471 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T01:16:12.472 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T01:16:12.473 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T01:16:12.474 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T01:16:12.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T01:16:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T01:16:12.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T01:16:12.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T01:16:12.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T01:16:12.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T01:16:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T01:16:12.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T01:16:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T01:16:12.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T01:16:12.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T01:16:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T01:16:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T01:16:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T01:16:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T01:16:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T01:16:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T01:16:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T01:16:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T01:16:12.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T01:16:12.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T01:16:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T01:16:12.500 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T01:16:12.501 DEBUG:teuthology.orchestra.run.smithi028:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T01:16:12.526 INFO:teuthology.orchestra.run.smithi028.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout:for file in vmlinuz System.map config; do 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout: fi 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout:done 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T01:16:12.527 INFO:teuthology.orchestra.run.smithi028.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T01:16:12.528 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-26T01:16:12.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-04-26T01:16:12.600 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-04-26T01:16:12.600 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-04-26T01:16:12.600 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-04-26T01:16:12.600 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-04-26T01:16:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-04-26T01:16:12.602 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:16:12.602 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T01:16:12.602 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T01:16:12.602 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T01:16:12.602 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-04-26T01:16:12.602 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:16:12.602 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T01:16:12.603 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T01:16:12.603 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T01:16:12.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T01:16:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-04-26T01:16:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T01:16:14.564 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-04-26T01:16:14.567 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-04-26T01:16:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.jAx9RkBmpc 2024-04-26T01:16:14.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jAx9RkBmpc 2024-04-26T01:16:14.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.jAx9RkBmpc 2024-04-26T01:16:14.805 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.jAx9RkBmpc is 6KB 2024-04-26T01:16:14.854 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.jAx9RkBmpc 2024-04-26T01:16:14.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T01:16:14.930 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T01:16:14.930 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T01:16:14.930 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T01:16:14.931 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T01:16:14.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T01:16:15.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-04-26T01:16:45.078 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T01:16:45.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-26T01:16:45.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:17:03.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-04-26T01:17:12.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-26T01:17:12.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:18:12.652 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T01:18:24.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-26T01:18:24.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:18:25.034 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-26T01:18:25.716 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-26T01:18:25.716 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T01:18:25.716 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-26T01:18:25.770 INFO:teuthology.orchestra.run.smithi028.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T01:18:25.770 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T01:18:25.770 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T01:18:25.771 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T01:18:26.771 DEBUG:teuthology.parallel:result is None 2024-04-26T01:18:26.772 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T01:18:26.779 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T01:18:26.779 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:18:26.787 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:18:26.790 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:18:26.841 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T01:18:26.848 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T01:18:26.859 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T01:18:26.859 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:18:26.865 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:18:26.868 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:18:27.001 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T01:18:27.007 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T01:18:27.008 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:18:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:18:27.025 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:18:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:18:27.042 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:18:27.050 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:18:27.068 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:18:27.112 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:18:27.123 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:18:27.124 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:18:27.147 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:18:27.170 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:18:27.225 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:18:27.230 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T01:18:27.237 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T01:18:27.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:18:27.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:18:27.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:18:27.324 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T01:18:27.336 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T01:18:27.336 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:18:27.344 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:18:27.346 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:18:27.392 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:18:27.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:18:27.597 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:18:27.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:18:27.779 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:18:27.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:18:27.865 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:18:27.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:18:27.965 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:18:28.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:18:28.094 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:18:28.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:18:28.162 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:18:28.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:18:28.255 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:18:28.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:18:28.381 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:18:28.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:18:28.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-04-26T01:18:28.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-04-26T01:18:28.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-04-26T01:18:28.495 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:18:28.525 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:18:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:18:29.006 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T01:18:29.016 INFO:teuthology.task.internal:Starting timer... 2024-04-26T01:18:29.016 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T01:18:29.027 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T01:18:29.038 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T01:18:29.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-04-26T01:18:29.172 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-04-26T01:18:29.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-04-26T01:18:29.229 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-04-26T01:18:29.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-04-26T01:18:29.297 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-04-26T01:18:29.299 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T01:18:29.299 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-04-26T01:18:29.345 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-04-26T01:18:29.346 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-04-26T01:18:29.367 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-04-26T01:18:29.367 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-04-26T01:18:29.387 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-04-26T01:18:29.388 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-04-26T01:18:29.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:18:29.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:18:29.467 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:18:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:18:29.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:18:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:18:29.519 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T01:18:29.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:18:29.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:18:29.619 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:18:29.644 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T01:18:29.654 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T01:18:29.726 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T01:18:29.728 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi053.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T01:23:44.452 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-04-26T01:23:44.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-26T01:23:44.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:23:44.567 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-26T01:23:44.720 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-26T01:23:44.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-26T01:23:44.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:23:44.798 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-04-26T01:23:44.888 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-26T01:23:44.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-04-26T01:23:44.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:23:44.962 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-04-26T01:23:45.044 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-04-26T01:23:45.044 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T01:23:45.055 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T01:23:45.055 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:23:45.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:23:45.063 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:23:45.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:23:45.065 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:23:45.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:23:45.096 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:23:45.112 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:23:45.115 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:23:45.130 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:23:45.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-04-26T01:23:45.149 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-04-26T01:23:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:23:45.154 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-04-26T01:23:45.164 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:23:45.166 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-04-26T01:23:45.178 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:23:45.179 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:23:45.193 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:23:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:23:45.234 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:23:45.237 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:23:45.237 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-04-26T01:23:45.237 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.238 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.238 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.238 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.248 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:23:45.251 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:23:45.251 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-04-26T01:23:45.251 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.251 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.251 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.251 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-04-26T01:23:45.316 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-04-26T01:23:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:23:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:23:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:23:45.560 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:23:45.560 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-26T01:23:45.560 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.560 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.561 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.561 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:23:45.562 INFO:teuthology.run_tasks:Running task install... 2024-04-26T01:23:45.573 DEBUG:teuthology.task.install:project ceph 2024-04-26T01:23:45.573 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-26T01:23:45.573 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-26T01:23:45.574 INFO:teuthology.task.install:Using flavor: default 2024-04-26T01:23:45.581 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T01:23:45.581 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T01:23:45.581 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:23:45.582 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:23:45.585 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:23:45.585 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:23:45.587 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:23:45.587 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:23:45.811 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:23:45.811 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:23:45.820 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:23:45.820 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:23:45.823 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:23:45.823 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:23:45.928 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:23:45.929 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:23:45.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:23:45.931 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:23:45.931 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:23:45.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:23:45.958 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:23:45.958 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:23:45.995 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:23:45.995 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:23:45.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:23:46.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:23:46.026 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:23:46.026 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:23:46.028 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:23:46.028 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:23:46.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:23:46.100 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:23:46.163 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-04-26T01:23:46.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-04-26T01:23:46.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:23:46.175 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:23:46.241 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-04-26T01:23:46.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-04-26T01:23:46.286 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:23:46.371 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-04-26T01:23:46.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-04-26T01:23:46.391 INFO:teuthology.orchestra.run.smithi169.stdout:48 files removed 2024-04-26T01:23:46.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:23:46.479 INFO:teuthology.orchestra.run.smithi053.stdout:48 files removed 2024-04-26T01:23:46.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:23:46.711 INFO:teuthology.orchestra.run.smithi028.stdout:48 files removed 2024-04-26T01:23:46.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:23:47.147 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-04-26T01:23:47.206 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 196 kB/s | 83 kB 00:00 2024-04-26T01:23:47.480 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 40 kB/s | 12 kB 00:00 2024-04-26T01:23:47.571 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 222 kB/s | 83 kB 00:00 2024-04-26T01:23:47.649 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-26T01:23:47.847 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-04-26T01:23:47.966 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-04-26T01:23:48.071 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-26T01:23:48.597 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-04-26T01:23:48.616 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-26T01:23:48.860 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-04-26T01:23:49.464 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-26T01:23:50.768 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-26T01:23:50.987 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-26T01:23:51.802 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-26T01:23:54.935 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-26T01:23:55.263 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-26T01:23:56.412 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-26T01:23:56.609 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 16 kB 00:00 2024-04-26T01:23:58.223 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-26T01:23:58.560 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-26T01:23:58.979 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 21 MB 00:02 2024-04-26T01:24:00.030 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-26T01:24:03.635 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-26T01:24:04.221 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-04-26T01:24:05.770 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:24:05.770 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:24:05.781 INFO:teuthology.orchestra.run.smithi169.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:24:05.782 INFO:teuthology.orchestra.run.smithi169.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:24:05.783 INFO:teuthology.orchestra.run.smithi169.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:24:05.784 INFO:teuthology.orchestra.run.smithi169.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:24:05.785 INFO:teuthology.orchestra.run.smithi169.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:24:05.789 INFO:teuthology.orchestra.run.smithi169.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:24:05.789 INFO:teuthology.orchestra.run.smithi169.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:24:05.850 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-26T01:24:05.871 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2024-04-26T01:24:05.871 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-26T01:24:05.871 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2024-04-26T01:24:05.872 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-26T01:24:05.872 INFO:teuthology.orchestra.run.smithi169.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:24:05.872 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:24:05.872 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:24:05.872 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:24:05.872 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:24:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:24:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:24:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:24:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:24:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:24:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:24:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:24:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:24:05.874 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:24:05.874 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:24:05.874 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:24:05.874 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:24:05.874 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:24:05.874 INFO:teuthology.orchestra.run.smithi169.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:24:05.875 INFO:teuthology.orchestra.run.smithi169.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:24:05.875 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:24:05.875 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:24:05.875 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:24:05.875 INFO:teuthology.orchestra.run.smithi169.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:24:05.875 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:24:05.876 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:24:05.876 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:24:05.876 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:24:05.876 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:24:05.876 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:24:05.876 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:24:05.877 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:24:05.877 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:24:05.877 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:24:05.877 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:24:05.877 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:24:05.877 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:24:05.878 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-04-26T01:24:05.878 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:24:05.878 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:24:05.878 INFO:teuthology.orchestra.run.smithi169.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:24:05.878 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:24:05.878 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-26T01:24:05.879 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:24:05.879 INFO:teuthology.orchestra.run.smithi169.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:24:05.879 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:24:05.880 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:24:05.880 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:24:05.880 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:24:05.880 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:24:05.880 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:24:05.881 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:24:05.881 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:24:05.881 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:24:05.882 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:24:05.882 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:24:05.882 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:24:05.882 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:24:05.882 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:24:05.883 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:24:05.883 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:24:05.883 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:24:05.883 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:24:05.883 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:24:05.883 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:24:05.883 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:24:05.884 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:24:05.884 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:24:05.884 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:24:05.884 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:24:05.884 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:24:05.884 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:24:05.884 INFO:teuthology.orchestra.run.smithi169.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:24:05.884 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:24:05.885 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:24:05.885 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:24:05.885 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:24:05.885 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:24:05.885 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:24:05.885 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:24:05.885 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:24:05.885 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:24:05.886 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:24:05.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:24:05.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:24:05.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:24:05.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:24:05.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:24:05.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:24:05.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:24:05.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:24:05.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:24:05.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:24:05.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:24:05.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:24:05.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:24:05.888 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:24:05.888 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:24:05.888 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:24:05.888 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:24:05.888 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:24:05.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:24:05.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:24:05.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:24:05.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:24:05.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:24:05.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:24:05.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:24:05.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:24:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:24:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:24:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:24:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:24:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:24:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:24:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:24:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:24:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:24:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:24:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:24:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:24:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:24:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:24:05.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:24:05.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:24:05.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:24:05.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:24:05.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:24:05.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:24:05.893 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:24:05.893 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:24:05.893 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:24:05.893 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:24:05.893 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:24:05.893 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:24:05.893 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:24:05.893 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:24:05.894 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:24:05.894 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:24:05.894 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:24:05.894 INFO:teuthology.orchestra.run.smithi169.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:24:05.894 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:24:05.894 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:24:05.894 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:24:05.895 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:24:05.895 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:24:05.895 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:24:05.895 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:24:05.895 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:24:05.895 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:24:05.896 INFO:teuthology.orchestra.run.smithi169.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:24:05.896 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:24:05.896 INFO:teuthology.orchestra.run.smithi169.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:24:05.896 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-04-26T01:24:05.896 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:24:05.896 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:24:05.896 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-26T01:24:05.897 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2024-04-26T01:24:05.897 INFO:teuthology.orchestra.run.smithi169.stdout:Install 148 Packages 2024-04-26T01:24:05.897 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 4 Packages 2024-04-26T01:24:05.897 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:24:05.898 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 222 M 2024-04-26T01:24:05.899 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-26T01:24:06.233 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-04-26T01:24:06.482 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:24:06.482 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:24:06.494 INFO:teuthology.orchestra.run.smithi053.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:24:06.495 INFO:teuthology.orchestra.run.smithi053.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:24:06.495 INFO:teuthology.orchestra.run.smithi053.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:24:06.496 INFO:teuthology.orchestra.run.smithi053.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:24:06.498 INFO:teuthology.orchestra.run.smithi053.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:24:06.500 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:24:06.501 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:24:06.566 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-26T01:24:06.587 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-26T01:24:06.588 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-04-26T01:24:06.588 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-26T01:24:06.588 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-04-26T01:24:06.588 INFO:teuthology.orchestra.run.smithi053.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:24:06.588 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:24:06.588 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:24:06.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:24:06.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:24:06.590 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:24:06.590 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:24:06.590 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:24:06.590 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:24:06.590 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:24:06.590 INFO:teuthology.orchestra.run.smithi053.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:24:06.590 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:24:06.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:24:06.592 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:24:06.593 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:24:06.594 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:24:06.595 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:24:06.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:24:06.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:24:06.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:24:06.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:24:06.600 INFO:teuthology.orchestra.run.smithi053.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout:Install 148 Packages 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 4 Packages 2024-04-26T01:24:06.601 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:24:06.602 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 222 M 2024-04-26T01:24:06.602 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-04-26T01:24:07.319 INFO:teuthology.orchestra.run.smithi169.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-26T01:24:07.573 INFO:teuthology.orchestra.run.smithi053.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 26 kB/s | 6.6 kB 00:00 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi053.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 962 kB/s | 890 kB 00:00 2024-04-26T01:24:08.640 INFO:teuthology.orchestra.run.smithi053.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 1.0 MB/s | 145 kB 00:00 2024-04-26T01:24:08.677 INFO:teuthology.orchestra.run.smithi169.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 655 kB/s | 890 kB 00:01 2024-04-26T01:24:08.819 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:24:08.821 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:24:08.843 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:24:08.845 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:24:08.846 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:24:08.848 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:24:08.850 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:24:08.854 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:24:08.855 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:24:08.932 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-26T01:24:08.946 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-26T01:24:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-26T01:24:08.946 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:24:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:24:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:24:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:24:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:24:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:24:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:24:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:24:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:24:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 4 Packages 2024-04-26T01:24:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:24:08.959 INFO:teuthology.orchestra.run.smithi169.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 526 kB/s | 145 kB 00:00 2024-04-26T01:24:08.959 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 222 M 2024-04-26T01:24:08.959 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-26T01:24:10.057 INFO:teuthology.orchestra.run.smithi053.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 1.7 MB/s | 2.4 MB 00:01 2024-04-26T01:24:10.393 INFO:teuthology.orchestra.run.smithi028.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 14 kB/s | 6.6 kB 00:00 2024-04-26T01:24:10.661 INFO:teuthology.orchestra.run.smithi169.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 1.4 MB/s | 2.4 MB 00:01 2024-04-26T01:24:11.245 INFO:teuthology.orchestra.run.smithi169.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 2.6 MB/s | 1.5 MB 00:00 2024-04-26T01:24:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 791 kB/s | 890 kB 00:01 2024-04-26T01:24:11.573 INFO:teuthology.orchestra.run.smithi053.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 1.0 MB/s | 1.5 MB 00:01 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi028.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 728 kB/s | 145 kB 00:00 2024-04-26T01:24:12.078 INFO:teuthology.orchestra.run.smithi169.stdout:(6/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.0 MB/s | 5.4 MB 00:05 2024-04-26T01:24:12.445 INFO:teuthology.orchestra.run.smithi169.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 3.9 MB/s | 4.7 MB 00:01 2024-04-26T01:24:13.418 INFO:teuthology.orchestra.run.smithi028.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 1.4 MB/s | 2.4 MB 00:01 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi028.stdout:(5/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.4 MB/s | 5.4 MB 00:03 2024-04-26T01:24:14.302 INFO:teuthology.orchestra.run.smithi028.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 1.7 MB/s | 1.5 MB 00:00 2024-04-26T01:24:14.965 INFO:teuthology.orchestra.run.smithi053.stdout:(6/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 2.8 MB/s | 21 MB 00:07 2024-04-26T01:24:15.240 INFO:teuthology.orchestra.run.smithi053.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 1.3 MB/s | 4.7 MB 00:03 2024-04-26T01:24:15.316 INFO:teuthology.orchestra.run.smithi053.stdout:(8/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 687 kB/s | 5.4 MB 00:07 2024-04-26T01:24:15.345 INFO:teuthology.orchestra.run.smithi169.stdout:(8/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 3.7 MB/s | 11 MB 00:02 2024-04-26T01:24:15.399 INFO:teuthology.orchestra.run.smithi053.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 303 kB/s | 25 kB 00:00 2024-04-26T01:24:15.462 INFO:teuthology.orchestra.run.smithi169.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 216 kB/s | 25 kB 00:00 2024-04-26T01:24:15.545 INFO:teuthology.orchestra.run.smithi028.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 2.6 MB/s | 4.7 MB 00:01 2024-04-26T01:24:15.828 INFO:teuthology.orchestra.run.smithi028.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 3.6 MB/s | 21 MB 00:05 2024-04-26T01:24:15.911 INFO:teuthology.orchestra.run.smithi028.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 307 kB/s | 25 kB 00:00 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi053.stdout:(10/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 3.8 MB/s | 17 MB 00:04 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi028.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 2.8 MB/s | 11 MB 00:03 2024-04-26T01:24:19.458 INFO:teuthology.orchestra.run.smithi053.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.6 MB/s | 225 kB 00:00 2024-04-26T01:24:19.541 INFO:teuthology.orchestra.run.smithi053.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 326 kB/s | 27 kB 00:00 2024-04-26T01:24:19.561 INFO:teuthology.orchestra.run.smithi028.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 1.3 MB/s | 225 kB 00:00 2024-04-26T01:24:19.616 INFO:teuthology.orchestra.run.smithi053.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 422 kB/s | 32 kB 00:00 2024-04-26T01:24:19.653 INFO:teuthology.orchestra.run.smithi028.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 298 kB/s | 27 kB 00:00 2024-04-26T01:24:19.754 INFO:teuthology.orchestra.run.smithi028.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 320 kB/s | 32 kB 00:00 2024-04-26T01:24:19.841 INFO:teuthology.orchestra.run.smithi053.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 3.2 MB/s | 747 kB 00:00 2024-04-26T01:24:19.933 INFO:teuthology.orchestra.run.smithi053.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.7 MB/s | 163 kB 00:00 2024-04-26T01:24:20.017 INFO:teuthology.orchestra.run.smithi053.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-26T01:24:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 2.6 MB/s | 747 kB 00:00 2024-04-26T01:24:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.8 MB/s | 163 kB 00:00 2024-04-26T01:24:20.159 INFO:teuthology.orchestra.run.smithi053.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 3.4 MB/s | 500 kB 00:00 2024-04-26T01:24:20.203 INFO:teuthology.orchestra.run.smithi169.stdout:(10/152): ceph-common-19.0.0-3156.gd69f13d8.el7 1.6 MB/s | 21 MB 00:13 2024-04-26T01:24:20.213 INFO:teuthology.orchestra.run.smithi028.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-26T01:24:20.428 INFO:teuthology.orchestra.run.smithi169.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 1.0 MB/s | 225 kB 00:00 2024-04-26T01:24:20.463 INFO:teuthology.orchestra.run.smithi028.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 2.0 MB/s | 500 kB 00:00 2024-04-26T01:24:20.520 INFO:teuthology.orchestra.run.smithi169.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 295 kB/s | 27 kB 00:00 2024-04-26T01:24:20.612 INFO:teuthology.orchestra.run.smithi169.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 346 kB/s | 32 kB 00:00 2024-04-26T01:24:20.926 INFO:teuthology.orchestra.run.smithi053.stdout:(18/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 1.9 MB/s | 11 MB 00:05 2024-04-26T01:24:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 494 kB/s | 45 kB 00:00 2024-04-26T01:24:21.095 INFO:teuthology.orchestra.run.smithi169.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 1.5 MB/s | 747 kB 00:00 2024-04-26T01:24:21.143 INFO:teuthology.orchestra.run.smithi053.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.1 MB/s | 135 kB 00:00 2024-04-26T01:24:21.271 INFO:teuthology.orchestra.run.smithi169.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 933 kB/s | 163 kB 00:00 2024-04-26T01:24:21.285 INFO:teuthology.orchestra.run.smithi053.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.1 MB/s | 161 kB 00:00 2024-04-26T01:24:21.371 INFO:teuthology.orchestra.run.smithi169.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.2 MB/s | 126 kB 00:00 2024-04-26T01:24:21.444 INFO:teuthology.orchestra.run.smithi053.stdout:(22/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 4.2 MB/s | 5.4 MB 00:01 2024-04-26T01:24:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:(23/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.4 MB/s | 320 kB 00:00 2024-04-26T01:24:21.544 INFO:teuthology.orchestra.run.smithi053.stdout:(24/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.9 MB/s | 298 kB 00:00 2024-04-26T01:24:21.586 INFO:teuthology.orchestra.run.smithi053.stdout:(25/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.3 MB/s | 99 kB 00:00 2024-04-26T01:24:21.636 INFO:teuthology.orchestra.run.smithi053.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 929 kB/s | 85 kB 00:00 2024-04-26T01:24:21.663 INFO:teuthology.orchestra.run.smithi169.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 1.7 MB/s | 500 kB 00:00 2024-04-26T01:24:21.737 INFO:teuthology.orchestra.run.smithi053.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.7 MB/s | 170 kB 00:00 2024-04-26T01:24:21.829 INFO:teuthology.orchestra.run.smithi053.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 374 kB/s | 34 kB 00:00 2024-04-26T01:24:21.954 INFO:teuthology.orchestra.run.smithi053.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.1 MB/s | 142 kB 00:00 2024-04-26T01:24:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:(18/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 2.1 MB/s | 17 MB 00:07 2024-04-26T01:24:22.372 INFO:teuthology.orchestra.run.smithi028.stdout:(19/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 2.8 MB/s | 5.4 MB 00:01 2024-04-26T01:24:22.397 INFO:teuthology.orchestra.run.smithi028.stdout:(20/152): python3-ceph-argparse-19.0.0-3156.gd6 288 kB/s | 45 kB 00:00 2024-04-26T01:24:22.531 INFO:teuthology.orchestra.run.smithi028.stdout:(21/152): python3-ceph-common-19.0.0-3156.gd69f 855 kB/s | 135 kB 00:00 2024-04-26T01:24:22.564 INFO:teuthology.orchestra.run.smithi028.stdout:(22/152): python3-cephfs-19.0.0-3156.gd69f13d8. 971 kB/s | 161 kB 00:00 2024-04-26T01:24:22.605 INFO:teuthology.orchestra.run.smithi169.stdout:(18/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 1.6 MB/s | 17 MB 00:10 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi169.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 544 kB/s | 45 kB 00:00 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi028.stdout:(23/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.8 MB/s | 320 kB 00:00 2024-04-26T01:24:22.740 INFO:teuthology.orchestra.run.smithi028.stdout:(24/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 1.7 MB/s | 298 kB 00:00 2024-04-26T01:24:22.747 INFO:teuthology.orchestra.run.smithi169.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 2.3 MB/s | 135 kB 00:00 2024-04-26T01:24:22.763 INFO:teuthology.orchestra.run.smithi053.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 4.4 MB/s | 3.5 MB 00:00 2024-04-26T01:24:22.782 INFO:teuthology.orchestra.run.smithi028.stdout:(25/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.3 MB/s | 99 kB 00:00 2024-04-26T01:24:22.815 INFO:teuthology.orchestra.run.smithi028.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.1 MB/s | 85 kB 00:00 2024-04-26T01:24:22.821 INFO:teuthology.orchestra.run.smithi053.stdout:(31/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 2.5 MB/s | 3.1 MB 00:01 2024-04-26T01:24:22.864 INFO:teuthology.orchestra.run.smithi169.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.4 MB/s | 161 kB 00:00 2024-04-26T01:24:22.899 INFO:teuthology.orchestra.run.smithi028.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.0 MB/s | 170 kB 00:00 2024-04-26T01:24:22.955 INFO:teuthology.orchestra.run.smithi053.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.8 MB/s | 249 kB 00:00 2024-04-26T01:24:22.999 INFO:teuthology.orchestra.run.smithi028.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 344 kB/s | 34 kB 00:00 2024-04-26T01:24:23.038 INFO:teuthology.orchestra.run.smithi053.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 593 kB/s | 49 kB 00:00 2024-04-26T01:24:23.056 INFO:teuthology.orchestra.run.smithi169.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.6 MB/s | 320 kB 00:00 2024-04-26T01:24:23.100 INFO:teuthology.orchestra.run.smithi028.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.4 MB/s | 142 kB 00:00 2024-04-26T01:24:23.105 INFO:teuthology.orchestra.run.smithi053.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 247 kB/s | 16 kB 00:00 2024-04-26T01:24:23.240 INFO:teuthology.orchestra.run.smithi169.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 1.6 MB/s | 298 kB 00:00 2024-04-26T01:24:23.247 INFO:teuthology.orchestra.run.smithi053.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 1.9 MB/s | 273 kB 00:00 2024-04-26T01:24:23.309 INFO:teuthology.orchestra.run.smithi169.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.4 MB/s | 99 kB 00:00 2024-04-26T01:24:23.401 INFO:teuthology.orchestra.run.smithi169.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 925 kB/s | 85 kB 00:00 2024-04-26T01:24:23.591 INFO:teuthology.orchestra.run.smithi053.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 2.2 MB/s | 762 kB 00:00 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi053.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 168 kB/s | 18 kB 00:00 2024-04-26T01:24:23.917 INFO:teuthology.orchestra.run.smithi028.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 2.7 MB/s | 3.1 MB 00:01 2024-04-26T01:24:24.260 INFO:teuthology.orchestra.run.smithi169.stdout:(26/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 2.1 MB/s | 5.4 MB 00:02 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi169.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.5 MB/s | 170 kB 00:00 2024-04-26T01:24:24.469 INFO:teuthology.orchestra.run.smithi169.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 341 kB/s | 34 kB 00:00 2024-04-26T01:24:24.525 INFO:teuthology.orchestra.run.smithi053.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 97 kB/s | 80 kB 00:00 2024-04-26T01:24:24.577 INFO:teuthology.orchestra.run.smithi169.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.3 MB/s | 142 kB 00:00 2024-04-26T01:24:24.609 INFO:teuthology.orchestra.run.smithi053.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 803 kB 00:00 2024-04-26T01:24:24.684 INFO:teuthology.orchestra.run.smithi028.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 2.2 MB/s | 3.5 MB 00:01 2024-04-26T01:24:24.701 INFO:teuthology.orchestra.run.smithi053.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 3.8 MB/s | 7.4 MB 00:01 2024-04-26T01:24:24.726 INFO:teuthology.orchestra.run.smithi053.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-04-26T01:24:24.751 INFO:teuthology.orchestra.run.smithi053.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-26T01:24:24.777 INFO:teuthology.orchestra.run.smithi169.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 2.2 MB/s | 3.1 MB 00:01 2024-04-26T01:24:24.801 INFO:teuthology.orchestra.run.smithi053.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-04-26T01:24:24.817 INFO:teuthology.orchestra.run.smithi028.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.8 MB/s | 249 kB 00:00 2024-04-26T01:24:24.827 INFO:teuthology.orchestra.run.smithi053.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-26T01:24:24.852 INFO:teuthology.orchestra.run.smithi053.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi053.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.5 MB/s | 125 kB 00:00 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi028.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 665 kB/s | 49 kB 00:00 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi053.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi053.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-26T01:24:24.970 INFO:teuthology.orchestra.run.smithi053.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-26T01:24:24.976 INFO:teuthology.orchestra.run.smithi028.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 199 kB/s | 16 kB 00:00 2024-04-26T01:24:24.995 INFO:teuthology.orchestra.run.smithi053.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T01:24:25.012 INFO:teuthology.orchestra.run.smithi053.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 519 kB/s | 21 kB 00:00 2024-04-26T01:24:25.037 INFO:teuthology.orchestra.run.smithi053.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 313 kB 00:00 2024-04-26T01:24:25.054 INFO:teuthology.orchestra.run.smithi053.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 793 kB/s | 33 kB 00:00 2024-04-26T01:24:25.126 INFO:teuthology.orchestra.run.smithi028.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 1.8 MB/s | 273 kB 00:00 2024-04-26T01:24:25.130 INFO:teuthology.orchestra.run.smithi053.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-04-26T01:24:25.147 INFO:teuthology.orchestra.run.smithi053.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 201 kB/s | 18 kB 00:00 2024-04-26T01:24:25.163 INFO:teuthology.orchestra.run.smithi053.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 334 kB/s | 11 kB 00:00 2024-04-26T01:24:25.189 INFO:teuthology.orchestra.run.smithi053.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-04-26T01:24:25.214 INFO:teuthology.orchestra.run.smithi053.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 910 kB/s | 45 kB 00:00 2024-04-26T01:24:25.248 INFO:teuthology.orchestra.run.smithi053.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-26T01:24:25.273 INFO:teuthology.orchestra.run.smithi053.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 10 MB/s | 582 kB 00:00 2024-04-26T01:24:25.290 INFO:teuthology.orchestra.run.smithi053.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 624 kB/s | 25 kB 00:00 2024-04-26T01:24:25.307 INFO:teuthology.orchestra.run.smithi053.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-26T01:24:25.341 INFO:teuthology.orchestra.run.smithi053.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-26T01:24:25.366 INFO:teuthology.orchestra.run.smithi053.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-26T01:24:25.400 INFO:teuthology.orchestra.run.smithi053.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 604 kB/s | 35 kB 00:00 2024-04-26T01:24:25.444 INFO:teuthology.orchestra.run.smithi028.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 2.4 MB/s | 762 kB 00:00 2024-04-26T01:24:25.535 INFO:teuthology.orchestra.run.smithi028.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 201 kB/s | 18 kB 00:00 2024-04-26T01:24:25.542 INFO:teuthology.orchestra.run.smithi053.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-26T01:24:25.600 INFO:teuthology.orchestra.run.smithi053.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-04-26T01:24:25.642 INFO:teuthology.orchestra.run.smithi053.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-26T01:24:25.676 INFO:teuthology.orchestra.run.smithi053.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-26T01:24:25.694 INFO:teuthology.orchestra.run.smithi028.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 511 kB/s | 80 kB 00:00 2024-04-26T01:24:25.726 INFO:teuthology.orchestra.run.smithi053.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-26T01:24:25.786 INFO:teuthology.orchestra.run.smithi028.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 8.7 MB/s | 803 kB 00:00 2024-04-26T01:24:25.810 INFO:teuthology.orchestra.run.smithi053.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-26T01:24:25.853 INFO:teuthology.orchestra.run.smithi028.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-26T01:24:25.885 INFO:teuthology.orchestra.run.smithi053.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 28 MB/s | 5.8 MB 00:00 2024-04-26T01:24:25.895 INFO:teuthology.orchestra.run.smithi028.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.1 MB/s | 253 kB 00:00 2024-04-26T01:24:25.910 INFO:teuthology.orchestra.run.smithi053.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 885 kB/s | 87 kB 00:00 2024-04-26T01:24:25.961 INFO:teuthology.orchestra.run.smithi053.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.9 MB/s | 244 kB 00:00 2024-04-26T01:24:25.979 INFO:teuthology.orchestra.run.smithi028.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-04-26T01:24:26.002 INFO:teuthology.orchestra.run.smithi053.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 6.2 MB/s | 248 kB 00:00 2024-04-26T01:24:26.078 INFO:teuthology.orchestra.run.smithi053.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 6.0 MB 00:00 2024-04-26T01:24:26.103 INFO:teuthology.orchestra.run.smithi053.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 262 kB/s | 26 kB 00:00 2024-04-26T01:24:26.104 INFO:teuthology.orchestra.run.smithi028.stdout:(43/152): ceph-mgr-diskprediction-local-19.0.0- 3.4 MB/s | 7.4 MB 00:02 2024-04-26T01:24:26.121 INFO:teuthology.orchestra.run.smithi028.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 755 kB/s | 106 kB 00:00 2024-04-26T01:24:26.128 INFO:teuthology.orchestra.run.smithi053.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-26T01:24:26.146 INFO:teuthology.orchestra.run.smithi028.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.3 MB/s | 135 kB 00:00 2024-04-26T01:24:26.154 INFO:teuthology.orchestra.run.smithi053.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-26T01:24:26.170 INFO:teuthology.orchestra.run.smithi169.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 2.2 MB/s | 3.5 MB 00:01 2024-04-26T01:24:26.171 INFO:teuthology.orchestra.run.smithi028.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.5 MB/s | 125 kB 00:00 2024-04-26T01:24:26.179 INFO:teuthology.orchestra.run.smithi053.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-04-26T01:24:26.205 INFO:teuthology.orchestra.run.smithi028.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 215 kB 00:00 2024-04-26T01:24:26.247 INFO:teuthology.orchestra.run.smithi028.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 9.3 MB/s | 689 kB 00:00 2024-04-26T01:24:26.286 INFO:teuthology.orchestra.run.smithi169.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.1 MB/s | 249 kB 00:00 2024-04-26T01:24:26.314 INFO:teuthology.orchestra.run.smithi028.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 9.1 MB/s | 943 kB 00:00 2024-04-26T01:24:26.321 INFO:teuthology.orchestra.run.smithi053.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 250 kB 00:00 2024-04-26T01:24:26.339 INFO:teuthology.orchestra.run.smithi028.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 107 kB 00:00 2024-04-26T01:24:26.353 INFO:teuthology.orchestra.run.smithi169.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 742 kB/s | 49 kB 00:00 2024-04-26T01:24:26.364 INFO:teuthology.orchestra.run.smithi028.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 445 kB/s | 21 kB 00:00 2024-04-26T01:24:26.396 INFO:teuthology.orchestra.run.smithi053.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-04-26T01:24:26.398 INFO:teuthology.orchestra.run.smithi028.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-04-26T01:24:26.412 INFO:teuthology.orchestra.run.smithi169.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 283 kB/s | 16 kB 00:00 2024-04-26T01:24:26.415 INFO:teuthology.orchestra.run.smithi028.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 703 kB/s | 33 kB 00:00 2024-04-26T01:24:26.422 INFO:teuthology.orchestra.run.smithi053.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-26T01:24:26.447 INFO:teuthology.orchestra.run.smithi053.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T01:24:26.463 INFO:teuthology.orchestra.run.smithi053.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-26T01:24:26.489 INFO:teuthology.orchestra.run.smithi053.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 6.3 MB/s | 269 kB 00:00 2024-04-26T01:24:26.514 INFO:teuthology.orchestra.run.smithi053.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-26T01:24:26.520 INFO:teuthology.orchestra.run.smithi169.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.5 MB/s | 273 kB 00:00 2024-04-26T01:24:26.524 INFO:teuthology.orchestra.run.smithi028.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-04-26T01:24:26.540 INFO:teuthology.orchestra.run.smithi053.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-26T01:24:26.540 INFO:teuthology.orchestra.run.smithi028.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 183 kB/s | 18 kB 00:00 2024-04-26T01:24:26.556 INFO:teuthology.orchestra.run.smithi053.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-26T01:24:26.573 INFO:teuthology.orchestra.run.smithi053.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-04-26T01:24:26.607 INFO:teuthology.orchestra.run.smithi053.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.3 MB/s | 42 kB 00:00 2024-04-26T01:24:26.612 INFO:teuthology.orchestra.run.smithi169.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 4.0 MB/s | 7.4 MB 00:01 2024-04-26T01:24:26.640 INFO:teuthology.orchestra.run.smithi053.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.4 MB/s | 44 kB 00:00 2024-04-26T01:24:26.654 INFO:teuthology.orchestra.run.smithi169.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 440 kB/s | 18 kB 00:00 2024-04-26T01:24:26.691 INFO:teuthology.orchestra.run.smithi053.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.5 MB/s | 221 kB 00:00 2024-04-26T01:24:26.733 INFO:teuthology.orchestra.run.smithi053.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 925 kB/s | 36 kB 00:00 2024-04-26T01:24:26.767 INFO:teuthology.orchestra.run.smithi053.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.5 MB/s | 47 kB 00:00 2024-04-26T01:24:26.771 INFO:teuthology.orchestra.run.smithi169.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 692 kB/s | 80 kB 00:00 2024-04-26T01:24:26.813 INFO:teuthology.orchestra.run.smithi169.stdout:(39/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 2.5 MB/s | 762 kB 00:00 2024-04-26T01:24:26.817 INFO:teuthology.orchestra.run.smithi053.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.2 MB/s | 305 kB 00:00 2024-04-26T01:24:26.872 INFO:teuthology.orchestra.run.smithi169.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.8 MB/s | 803 kB 00:00 2024-04-26T01:24:26.897 INFO:teuthology.orchestra.run.smithi169.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-26T01:24:26.922 INFO:teuthology.orchestra.run.smithi169.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-26T01:24:26.964 INFO:teuthology.orchestra.run.smithi169.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-26T01:24:26.976 INFO:teuthology.orchestra.run.smithi053.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-04-26T01:24:26.989 INFO:teuthology.orchestra.run.smithi169.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-26T01:24:27.006 INFO:teuthology.orchestra.run.smithi169.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-26T01:24:27.032 INFO:teuthology.orchestra.run.smithi169.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-26T01:24:27.057 INFO:teuthology.orchestra.run.smithi169.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-26T01:24:27.091 INFO:teuthology.orchestra.run.smithi169.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-26T01:24:27.124 INFO:teuthology.orchestra.run.smithi169.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-26T01:24:27.150 INFO:teuthology.orchestra.run.smithi169.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T01:24:27.160 INFO:teuthology.orchestra.run.smithi053.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 170 kB/s | 58 kB 00:00 2024-04-26T01:24:27.166 INFO:teuthology.orchestra.run.smithi169.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 515 kB/s | 21 kB 00:00 2024-04-26T01:24:27.185 INFO:teuthology.orchestra.run.smithi053.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 449 kB/s | 94 kB 00:00 2024-04-26T01:24:27.200 INFO:teuthology.orchestra.run.smithi169.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-26T01:24:27.210 INFO:teuthology.orchestra.run.smithi053.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-26T01:24:27.222 INFO:teuthology.orchestra.run.smithi028.stdout:(56/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 4.3 MB/s | 49 MB 00:11 2024-04-26T01:24:27.225 INFO:teuthology.orchestra.run.smithi169.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 557 kB/s | 33 kB 00:00 2024-04-26T01:24:27.236 INFO:teuthology.orchestra.run.smithi053.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-04-26T01:24:27.250 INFO:teuthology.orchestra.run.smithi028.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 16 kB/s | 11 kB 00:00 2024-04-26T01:24:27.261 INFO:teuthology.orchestra.run.smithi053.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-04-26T01:24:27.275 INFO:teuthology.orchestra.run.smithi028.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 219 kB/s | 160 kB 00:00 2024-04-26T01:24:27.300 INFO:teuthology.orchestra.run.smithi028.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 599 kB/s | 45 kB 00:00 2024-04-26T01:24:27.344 INFO:teuthology.orchestra.run.smithi053.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 7.8 MB/s | 863 kB 00:00 2024-04-26T01:24:27.367 INFO:teuthology.orchestra.run.smithi028.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 7.2 MB/s | 662 kB 00:00 2024-04-26T01:24:27.401 INFO:teuthology.orchestra.run.smithi028.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 4.6 MB/s | 582 kB 00:00 2024-04-26T01:24:27.426 INFO:teuthology.orchestra.run.smithi028.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 204 kB/s | 25 kB 00:00 2024-04-26T01:24:27.460 INFO:teuthology.orchestra.run.smithi028.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 830 kB/s | 47 kB 00:00 2024-04-26T01:24:27.467 INFO:teuthology.orchestra.run.smithi169.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-04-26T01:24:27.484 INFO:teuthology.orchestra.run.smithi028.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.5 MB/s | 292 kB 00:00 2024-04-26T01:24:27.493 INFO:teuthology.orchestra.run.smithi169.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 69 kB/s | 18 kB 00:00 2024-04-26T01:24:27.503 INFO:teuthology.orchestra.run.smithi053.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.2 MB/s | 548 kB 00:00 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi169.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi028.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.1 MB/s | 188 kB 00:00 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi053.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.7 MB/s | 309 kB 00:00 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi169.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi053.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi028.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 436 kB/s | 35 kB 00:00 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi169.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 771 kB/s | 45 kB 00:00 2024-04-26T01:24:27.603 INFO:teuthology.orchestra.run.smithi169.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-26T01:24:27.636 INFO:teuthology.orchestra.run.smithi169.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 582 kB 00:00 2024-04-26T01:24:27.661 INFO:teuthology.orchestra.run.smithi169.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 443 kB/s | 25 kB 00:00 2024-04-26T01:24:27.687 INFO:teuthology.orchestra.run.smithi169.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-04-26T01:24:27.712 INFO:teuthology.orchestra.run.smithi169.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-26T01:24:27.727 INFO:teuthology.orchestra.run.smithi028.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-04-26T01:24:27.745 INFO:teuthology.orchestra.run.smithi169.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-26T01:24:27.753 INFO:teuthology.orchestra.run.smithi053.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-04-26T01:24:27.771 INFO:teuthology.orchestra.run.smithi169.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-04-26T01:24:27.822 INFO:teuthology.orchestra.run.smithi053.stdout:(108/152): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-26T01:24:27.846 INFO:teuthology.orchestra.run.smithi053.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 544 kB/s | 49 kB 00:00 2024-04-26T01:24:27.852 INFO:teuthology.orchestra.run.smithi028.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-26T01:24:27.871 INFO:teuthology.orchestra.run.smithi053.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-26T01:24:27.896 INFO:teuthology.orchestra.run.smithi053.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-04-26T01:24:27.911 INFO:teuthology.orchestra.run.smithi028.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-04-26T01:24:27.929 INFO:teuthology.orchestra.run.smithi169.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-04-26T01:24:27.938 INFO:teuthology.orchestra.run.smithi053.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-04-26T01:24:27.944 INFO:teuthology.orchestra.run.smithi028.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 4.5 MB/s | 911 kB 00:00 2024-04-26T01:24:27.970 INFO:teuthology.orchestra.run.smithi028.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-26T01:24:27.972 INFO:teuthology.orchestra.run.smithi053.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 7.2 MB/s | 548 kB 00:00 2024-04-26T01:24:27.988 INFO:teuthology.orchestra.run.smithi169.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-04-26T01:24:28.006 INFO:teuthology.orchestra.run.smithi053.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 649 kB/s | 43 kB 00:00 2024-04-26T01:24:28.031 INFO:teuthology.orchestra.run.smithi053.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-04-26T01:24:28.038 INFO:teuthology.orchestra.run.smithi169.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-26T01:24:28.064 INFO:teuthology.orchestra.run.smithi053.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-04-26T01:24:28.080 INFO:teuthology.orchestra.run.smithi169.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 9.9 MB/s | 911 kB 00:00 2024-04-26T01:24:28.097 INFO:teuthology.orchestra.run.smithi169.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-26T01:24:28.114 INFO:teuthology.orchestra.run.smithi053.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-26T01:24:28.117 INFO:teuthology.orchestra.run.smithi028.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.9 MB/s | 1.0 MB 00:00 2024-04-26T01:24:28.140 INFO:teuthology.orchestra.run.smithi053.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-26T01:24:28.165 INFO:teuthology.orchestra.run.smithi053.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 6.0 MB/s | 303 kB 00:00 2024-04-26T01:24:28.172 INFO:teuthology.orchestra.run.smithi169.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 14 MB/s | 1.0 MB 00:00 2024-04-26T01:24:28.190 INFO:teuthology.orchestra.run.smithi053.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 705 kB/s | 35 kB 00:00 2024-04-26T01:24:28.224 INFO:teuthology.orchestra.run.smithi053.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.5 MB/s | 206 kB 00:00 2024-04-26T01:24:28.230 INFO:teuthology.orchestra.run.smithi028.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-04-26T01:24:28.239 INFO:teuthology.orchestra.run.smithi169.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-26T01:24:28.254 INFO:teuthology.orchestra.run.smithi028.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 309 kB/s | 87 kB 00:00 2024-04-26T01:24:28.315 INFO:teuthology.orchestra.run.smithi169.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 25 MB/s | 5.8 MB 00:00 2024-04-26T01:24:28.341 INFO:teuthology.orchestra.run.smithi053.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-26T01:24:28.357 INFO:teuthology.orchestra.run.smithi053.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-26T01:24:28.373 INFO:teuthology.orchestra.run.smithi169.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.3 MB/s | 244 kB 00:00 2024-04-26T01:24:28.400 INFO:teuthology.orchestra.run.smithi053.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-26T01:24:28.421 INFO:teuthology.orchestra.run.smithi028.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-04-26T01:24:28.425 INFO:teuthology.orchestra.run.smithi053.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 263 kB/s | 18 kB 00:00 2024-04-26T01:24:28.449 INFO:teuthology.orchestra.run.smithi169.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 6.0 MB 00:00 2024-04-26T01:24:28.450 INFO:teuthology.orchestra.run.smithi053.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-04-26T01:24:28.454 INFO:teuthology.orchestra.run.smithi028.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.3 MB/s | 244 kB 00:00 2024-04-26T01:24:28.474 INFO:teuthology.orchestra.run.smithi169.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.5 MB/s | 248 kB 00:00 2024-04-26T01:24:28.476 INFO:teuthology.orchestra.run.smithi053.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-04-26T01:24:28.479 INFO:teuthology.orchestra.run.smithi028.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 248 kB 00:00 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi169.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi053.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi028.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 321 kB/s | 26 kB 00:00 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi169.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi028.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 392 kB/s | 19 kB 00:00 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi169.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi053.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi169.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 591 kB/s | 35 kB 00:00 2024-04-26T01:24:28.588 INFO:teuthology.orchestra.run.smithi028.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-04-26T01:24:28.614 INFO:teuthology.orchestra.run.smithi028.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 327 kB/s | 35 kB 00:00 2024-04-26T01:24:28.643 INFO:teuthology.orchestra.run.smithi053.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 327 kB/s | 46 kB 00:00 2024-04-26T01:24:28.668 INFO:teuthology.orchestra.run.smithi053.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 861 kB/s | 79 kB 00:00 2024-04-26T01:24:28.689 INFO:teuthology.orchestra.run.smithi028.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.5 MB/s | 250 kB 00:00 2024-04-26T01:24:28.702 INFO:teuthology.orchestra.run.smithi053.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 990 kB/s | 58 kB 00:00 2024-04-26T01:24:28.744 INFO:teuthology.orchestra.run.smithi053.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-26T01:24:28.769 INFO:teuthology.orchestra.run.smithi053.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 248 kB/s | 16 kB 00:00 2024-04-26T01:24:28.794 INFO:teuthology.orchestra.run.smithi053.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-26T01:24:28.797 INFO:teuthology.orchestra.run.smithi028.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-04-26T01:24:28.823 INFO:teuthology.orchestra.run.smithi028.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 222 kB 00:00 2024-04-26T01:24:28.828 INFO:teuthology.orchestra.run.smithi053.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 485 kB/s | 28 kB 00:00 2024-04-26T01:24:28.841 INFO:teuthology.orchestra.run.smithi028.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 573 kB/s | 77 kB 00:00 2024-04-26T01:24:28.853 INFO:teuthology.orchestra.run.smithi053.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 525 kB/s | 31 kB 00:00 2024-04-26T01:24:28.856 INFO:teuthology.orchestra.run.smithi028.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 1.3 MB/s | 42 kB 00:00 2024-04-26T01:24:28.879 INFO:teuthology.orchestra.run.smithi053.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-26T01:24:28.882 INFO:teuthology.orchestra.run.smithi028.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 269 kB 00:00 2024-04-26T01:24:28.904 INFO:teuthology.orchestra.run.smithi053.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-26T01:24:28.907 INFO:teuthology.orchestra.run.smithi028.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.5 MB/s | 159 kB 00:00 2024-04-26T01:24:28.938 INFO:teuthology.orchestra.run.smithi053.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 614 kB/s | 36 kB 00:00 2024-04-26T01:24:28.941 INFO:teuthology.orchestra.run.smithi028.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 279 kB 00:00 2024-04-26T01:24:28.963 INFO:teuthology.orchestra.run.smithi053.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 872 kB/s | 51 kB 00:00 2024-04-26T01:24:28.966 INFO:teuthology.orchestra.run.smithi028.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 51 kB 00:00 2024-04-26T01:24:28.988 INFO:teuthology.orchestra.run.smithi053.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-26T01:24:29.009 INFO:teuthology.orchestra.run.smithi169.stdout:(81/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 3.6 MB/s | 49 MB 00:13 2024-04-26T01:24:29.013 INFO:teuthology.orchestra.run.smithi053.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-26T01:24:29.022 INFO:teuthology.orchestra.run.smithi028.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 485 kB/s | 54 kB 00:00 2024-04-26T01:24:29.034 INFO:teuthology.orchestra.run.smithi169.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 558 kB/s | 250 kB 00:00 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi053.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 8.4 MB/s | 427 kB 00:00 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi053.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 488 kB/s | 20 kB 00:00 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi028.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 337 kB/s | 42 kB 00:00 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi053.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi169.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.1 MB/s | 5.1 MB 00:00 2024-04-26T01:24:29.143 INFO:teuthology.orchestra.run.smithi169.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-04-26T01:24:29.168 INFO:teuthology.orchestra.run.smithi169.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 591 kB/s | 77 kB 00:00 2024-04-26T01:24:29.185 INFO:teuthology.orchestra.run.smithi169.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-26T01:24:29.210 INFO:teuthology.orchestra.run.smithi169.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 269 kB 00:00 2024-04-26T01:24:29.235 INFO:teuthology.orchestra.run.smithi169.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-26T01:24:29.239 INFO:teuthology.orchestra.run.smithi053.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-04-26T01:24:29.261 INFO:teuthology.orchestra.run.smithi169.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-26T01:24:29.286 INFO:teuthology.orchestra.run.smithi169.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-26T01:24:29.311 INFO:teuthology.orchestra.run.smithi169.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 709 kB/s | 54 kB 00:00 2024-04-26T01:24:29.392 INFO:teuthology.orchestra.run.smithi028.stdout:(93/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-04-26T01:24:29.417 INFO:teuthology.orchestra.run.smithi028.stdout:(94/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 117 kB/s | 44 kB 00:00 2024-04-26T01:24:29.420 INFO:teuthology.orchestra.run.smithi169.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 397 kB/s | 42 kB 00:00 2024-04-26T01:24:29.445 INFO:teuthology.orchestra.run.smithi169.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 340 kB/s | 44 kB 00:00 2024-04-26T01:24:29.483 INFO:teuthology.orchestra.run.smithi028.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 590 kB/s | 221 kB 00:00 2024-04-26T01:24:29.509 INFO:teuthology.orchestra.run.smithi028.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 399 kB/s | 36 kB 00:00 2024-04-26T01:24:29.513 INFO:teuthology.orchestra.run.smithi169.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-04-26T01:24:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 412 kB/s | 47 kB 00:00 2024-04-26T01:24:29.537 INFO:teuthology.orchestra.run.smithi169.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 403 kB/s | 36 kB 00:00 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi028.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi028.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi028.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi028.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 866 kB/s | 64 kB 00:00 2024-04-26T01:24:29.669 INFO:teuthology.orchestra.run.smithi028.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 914 kB/s | 45 kB 00:00 2024-04-26T01:24:29.694 INFO:teuthology.orchestra.run.smithi028.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 301 kB/s | 22 kB 00:00 2024-04-26T01:24:29.713 INFO:teuthology.orchestra.run.smithi169.stdout:(96/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-04-26T01:24:29.736 INFO:teuthology.orchestra.run.smithi028.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.3 MB/s | 863 kB 00:00 2024-04-26T01:24:29.738 INFO:teuthology.orchestra.run.smithi169.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 213 kB/s | 47 kB 00:00 2024-04-26T01:24:29.763 INFO:teuthology.orchestra.run.smithi169.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-04-26T01:24:29.789 INFO:teuthology.orchestra.run.smithi169.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 776 kB/s | 58 kB 00:00 2024-04-26T01:24:29.814 INFO:teuthology.orchestra.run.smithi169.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi169.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 848 kB/s | 64 kB 00:00 2024-04-26T01:24:29.854 INFO:teuthology.orchestra.run.smithi028.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 168 kB/s | 19 kB 00:00 2024-04-26T01:24:29.856 INFO:teuthology.orchestra.run.smithi169.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 669 kB/s | 45 kB 00:00 2024-04-26T01:24:29.873 INFO:teuthology.orchestra.run.smithi169.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 670 kB/s | 22 kB 00:00 2024-04-26T01:24:29.899 INFO:teuthology.orchestra.run.smithi169.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-04-26T01:24:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-26T01:24:29.998 INFO:teuthology.orchestra.run.smithi053.stdout:(148/152): ceph-test-19.0.0-3156.gd69f13d8.el7. 3.4 MB/s | 49 MB 00:14 2024-04-26T01:24:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-04-26T01:24:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 349 kB/s | 26 kB 00:00 2024-04-26T01:24:30.056 INFO:teuthology.orchestra.run.smithi053.stdout:(149/152): librados2-19.0.0-3156.gd69f13d8.el7. 3.5 MB/s | 3.4 MB 00:00 2024-04-26T01:24:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 734 kB/s | 49 kB 00:00 2024-04-26T01:24:30.082 INFO:teuthology.orchestra.run.smithi169.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-04-26T01:24:30.098 INFO:teuthology.orchestra.run.smithi053.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 14 MB/s | 1.3 MB 00:00 2024-04-26T01:24:30.104 INFO:teuthology.orchestra.run.smithi028.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-26T01:24:30.130 INFO:teuthology.orchestra.run.smithi028.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.6 MB/s | 151 kB 00:00 2024-04-26T01:24:30.165 INFO:teuthology.orchestra.run.smithi053.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 3.4 MB/s | 3.1 MB 00:00 2024-04-26T01:24:30.214 INFO:teuthology.orchestra.run.smithi028.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 6.5 MB/s | 548 kB 00:00 2024-04-26T01:24:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-04-26T01:24:30.220 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:24:30.220 INFO:teuthology.orchestra.run.smithi053.stdout:Total 9.4 MB/s | 222 MB 00:23 2024-04-26T01:24:30.255 INFO:teuthology.orchestra.run.smithi028.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-04-26T01:24:30.281 INFO:teuthology.orchestra.run.smithi028.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 657 kB/s | 43 kB 00:00 2024-04-26T01:24:30.314 INFO:teuthology.orchestra.run.smithi028.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 558 kB/s | 32 kB 00:00 2024-04-26T01:24:30.340 INFO:teuthology.orchestra.run.smithi028.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 259 kB/s | 15 kB 00:00 2024-04-26T01:24:30.365 INFO:teuthology.orchestra.run.smithi169.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 629 kB/s | 309 kB 00:00 2024-04-26T01:24:30.440 INFO:teuthology.orchestra.run.smithi028.stdout:(117/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-04-26T01:24:30.441 INFO:teuthology.orchestra.run.smithi169.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-04-26T01:24:30.474 INFO:teuthology.orchestra.run.smithi028.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-04-26T01:24:30.507 INFO:teuthology.orchestra.run.smithi028.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-04-26T01:24:30.524 INFO:teuthology.orchestra.run.smithi169.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-04-26T01:24:30.532 INFO:teuthology.orchestra.run.smithi028.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 608 kB/s | 35 kB 00:00 2024-04-26T01:24:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-26T01:24:30.599 INFO:teuthology.orchestra.run.smithi169.stdout:(109/152): abseil-cpp-20211102.0-3.el9.x86_64.r 738 kB/s | 548 kB 00:00 2024-04-26T01:24:30.600 INFO:teuthology.orchestra.run.smithi028.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-04-26T01:24:30.633 INFO:teuthology.orchestra.run.smithi028.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-26T01:24:30.659 INFO:teuthology.orchestra.run.smithi028.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 185 kB/s | 11 kB 00:00 2024-04-26T01:24:30.666 INFO:teuthology.orchestra.run.smithi169.stdout:(110/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 4.4 MB 00:00 2024-04-26T01:24:30.692 INFO:teuthology.orchestra.run.smithi169.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 402 kB/s | 67 kB 00:00 2024-04-26T01:24:30.692 INFO:teuthology.orchestra.run.smithi028.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 306 kB/s | 18 kB 00:00 2024-04-26T01:24:30.718 INFO:teuthology.orchestra.run.smithi028.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 400 kB/s | 23 kB 00:00 2024-04-26T01:24:30.743 INFO:teuthology.orchestra.run.smithi028.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 393 kB/s | 19 kB 00:00 2024-04-26T01:24:30.776 INFO:teuthology.orchestra.run.smithi028.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 340 kB/s | 20 kB 00:00 2024-04-26T01:24:30.784 INFO:teuthology.orchestra.run.smithi169.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 820 kB/s | 151 kB 00:00 2024-04-26T01:24:30.809 INFO:teuthology.orchestra.run.smithi169.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-26T01:24:30.851 INFO:teuthology.orchestra.run.smithi169.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 650 kB/s | 43 kB 00:00 2024-04-26T01:24:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.4 MB/s | 2.0 MB 00:00 2024-04-26T01:24:30.877 INFO:teuthology.orchestra.run.smithi028.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 467 kB/s | 46 kB 00:00 2024-04-26T01:24:30.918 INFO:teuthology.orchestra.run.smithi169.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-26T01:24:30.919 INFO:teuthology.orchestra.run.smithi028.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-04-26T01:24:30.936 INFO:teuthology.orchestra.run.smithi028.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 996 kB/s | 58 kB 00:00 2024-04-26T01:24:30.943 INFO:teuthology.orchestra.run.smithi169.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-26T01:24:30.961 INFO:teuthology.orchestra.run.smithi028.stdout:(133/152): python3-more-itertools-8.12.0-2.el9. 728 kB/s | 79 kB 00:00 2024-04-26T01:24:30.995 INFO:teuthology.orchestra.run.smithi028.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-04-26T01:24:31.012 INFO:teuthology.orchestra.run.smithi028.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-04-26T01:24:31.035 INFO:teuthology.orchestra.run.smithi169.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-26T01:24:31.037 INFO:teuthology.orchestra.run.smithi028.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T01:24:31.062 INFO:teuthology.orchestra.run.smithi028.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 426 kB/s | 28 kB 00:00 2024-04-26T01:24:31.088 INFO:teuthology.orchestra.run.smithi028.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 411 kB/s | 31 kB 00:00 2024-04-26T01:24:31.110 INFO:teuthology.orchestra.run.smithi169.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 894 kB/s | 172 kB 00:00 2024-04-26T01:24:31.121 INFO:teuthology.orchestra.run.smithi028.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-26T01:24:31.136 INFO:teuthology.orchestra.run.smithi169.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-26T01:24:31.146 INFO:teuthology.orchestra.run.smithi028.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 618 kB/s | 36 kB 00:00 2024-04-26T01:24:31.172 INFO:teuthology.orchestra.run.smithi028.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-04-26T01:24:31.202 INFO:teuthology.orchestra.run.smithi169.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 384 kB/s | 35 kB 00:00 2024-04-26T01:24:31.222 INFO:teuthology.orchestra.run.smithi028.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 512 kB/s | 51 kB 00:00 2024-04-26T01:24:31.236 INFO:teuthology.orchestra.run.smithi169.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-26T01:24:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-26T01:24:31.286 INFO:teuthology.orchestra.run.smithi169.stdout:(122/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-04-26T01:24:31.297 INFO:teuthology.orchestra.run.smithi028.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-04-26T01:24:31.323 INFO:teuthology.orchestra.run.smithi028.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 268 kB/s | 20 kB 00:00 2024-04-26T01:24:31.326 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-04-26T01:24:31.328 INFO:teuthology.orchestra.run.smithi169.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-26T01:24:31.353 INFO:teuthology.orchestra.run.smithi169.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-04-26T01:24:31.356 INFO:teuthology.orchestra.run.smithi028.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-26T01:24:31.390 INFO:teuthology.orchestra.run.smithi028.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-04-26T01:24:31.395 INFO:teuthology.orchestra.run.smithi169.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-04-26T01:24:31.421 INFO:teuthology.orchestra.run.smithi169.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-04-26T01:24:31.457 INFO:teuthology.orchestra.run.smithi028.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-26T01:24:31.462 INFO:teuthology.orchestra.run.smithi169.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 291 kB/s | 19 kB 00:00 2024-04-26T01:24:31.486 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-04-26T01:24:31.486 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-04-26T01:24:31.488 INFO:teuthology.orchestra.run.smithi169.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 293 kB/s | 20 kB 00:00 2024-04-26T01:24:31.524 INFO:teuthology.orchestra.run.smithi028.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 21 MB/s | 1.3 MB 00:00 2024-04-26T01:24:31.555 INFO:teuthology.orchestra.run.smithi169.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 694 kB/s | 46 kB 00:00 2024-04-26T01:24:31.616 INFO:teuthology.orchestra.run.smithi028.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-04-26T01:24:31.630 INFO:teuthology.orchestra.run.smithi169.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-04-26T01:24:31.680 INFO:teuthology.orchestra.run.smithi169.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-04-26T01:24:31.715 INFO:teuthology.orchestra.run.smithi169.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 691 kB/s | 58 kB 00:00 2024-04-26T01:24:31.756 INFO:teuthology.orchestra.run.smithi169.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-26T01:24:31.781 INFO:teuthology.orchestra.run.smithi169.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-04-26T01:24:31.832 INFO:teuthology.orchestra.run.smithi169.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T01:24:31.874 INFO:teuthology.orchestra.run.smithi169.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-04-26T01:24:31.924 INFO:teuthology.orchestra.run.smithi169.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-04-26T01:24:32.033 INFO:teuthology.orchestra.run.smithi169.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-26T01:24:32.058 INFO:teuthology.orchestra.run.smithi169.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-04-26T01:24:32.100 INFO:teuthology.orchestra.run.smithi169.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 533 kB/s | 36 kB 00:00 2024-04-26T01:24:32.134 INFO:teuthology.orchestra.run.smithi169.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-26T01:24:32.141 INFO:teuthology.orchestra.run.smithi028.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 4.2 MB/s | 3.1 MB 00:00 2024-04-26T01:24:32.234 INFO:teuthology.orchestra.run.smithi169.stdout:(142/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.0 MB/s | 2.0 MB 00:01 2024-04-26T01:24:32.259 INFO:teuthology.orchestra.run.smithi169.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 715 kB/s | 90 kB 00:00 2024-04-26T01:24:32.284 INFO:teuthology.orchestra.run.smithi169.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-26T01:24:32.327 INFO:teuthology.orchestra.run.smithi169.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-04-26T01:24:32.333 INFO:teuthology.orchestra.run.smithi028.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 3.5 MB/s | 3.4 MB 00:00 2024-04-26T01:24:32.341 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:24:32.342 INFO:teuthology.orchestra.run.smithi028.stdout:Total 9.5 MB/s | 222 MB 00:23 2024-04-26T01:24:32.485 INFO:teuthology.orchestra.run.smithi169.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.7 MB/s | 427 kB 00:00 2024-04-26T01:24:32.585 INFO:teuthology.orchestra.run.smithi169.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 638 kB/s | 192 kB 00:00 2024-04-26T01:24:32.627 INFO:teuthology.orchestra.run.smithi169.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-04-26T01:24:32.694 INFO:teuthology.orchestra.run.smithi169.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 20 MB/s | 1.3 MB 00:00 2024-04-26T01:24:32.770 INFO:teuthology.orchestra.run.smithi169.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 32 MB/s | 2.4 MB 00:00 2024-04-26T01:24:33.086 INFO:teuthology.orchestra.run.smithi169.stdout:(151/152): librados2-19.0.0-3156.gd69f13d8.el7. 5.7 MB/s | 3.4 MB 00:00 2024-04-26T01:24:33.136 INFO:teuthology.orchestra.run.smithi169.stdout:(152/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 5.7 MB/s | 3.1 MB 00:00 2024-04-26T01:24:33.140 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:24:33.140 INFO:teuthology.orchestra.run.smithi169.stdout:Total 8.1 MB/s | 222 MB 00:27 2024-04-26T01:24:33.360 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-04-26T01:24:33.360 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-04-26T01:24:34.257 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-26T01:24:34.402 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-26T01:24:34.402 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-26T01:24:35.302 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-26T01:24:35.563 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-26T01:24:35.563 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-26T01:24:36.170 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-26T01:24:36.171 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-26T01:24:38.387 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-26T01:24:38.396 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-26T01:24:39.492 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-04-26T01:24:39.594 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:24:39.633 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:24:39.687 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:24:39.789 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:24:40.133 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:24:40.379 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:24:40.406 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:24:40.490 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:24:40.527 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:24:40.684 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:24:40.733 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:24:40.777 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:24:40.874 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:24:40.903 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:24:40.966 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:24:40.987 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:24:41.008 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:24:41.180 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:24:41.212 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:24:41.324 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:24:41.427 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:24:41.470 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:24:41.505 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:24:41.580 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:24:41.620 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:24:41.664 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:24:41.696 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:24:41.724 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:24:41.804 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:24:41.856 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:24:41.885 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:24:41.921 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:24:41.999 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:24:42.031 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:24:42.075 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:24:42.099 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-26T01:24:42.103 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:24:42.176 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:24:42.190 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:24:42.227 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:24:42.228 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:24:42.282 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:24:42.304 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:24:42.326 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:24:42.349 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:24:42.393 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:24:42.449 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:24:42.505 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:24:42.778 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:24:42.834 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:24:42.955 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:24:42.985 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:24:43.076 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:24:43.106 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:24:43.187 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:24:43.230 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:24:43.254 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:24:43.268 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:24:43.303 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:24:43.310 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:24:43.331 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:24:43.416 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:24:43.444 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:24:43.517 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:24:43.601 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:24:43.675 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:24:43.713 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:24:43.749 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:24:43.758 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:24:43.791 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:24:43.813 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:24:43.849 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:24:43.892 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:24:43.899 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:24:43.934 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:24:43.992 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:24:44.123 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:24:44.153 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:24:44.157 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:24:44.191 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:24:44.194 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:24:44.236 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:24:44.267 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:24:44.295 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:24:44.327 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:24:44.365 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:24:44.367 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:24:44.393 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:24:44.417 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:24:44.427 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:24:44.465 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:24:44.501 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:24:44.509 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:24:44.585 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:24:44.738 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:24:44.760 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:24:44.778 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:24:44.805 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:24:44.808 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:24:44.838 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:24:44.873 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:24:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:24:44.906 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:24:44.948 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:24:44.964 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-26T01:24:44.982 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:24:45.010 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:24:45.065 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:24:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:24:45.071 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:24:45.121 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:24:45.139 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:24:45.152 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:24:45.320 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:24:45.337 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:24:45.371 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:24:45.375 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:24:45.498 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:24:45.582 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:24:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:24:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:24:45.956 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:24:45.993 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:24:46.050 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:24:46.081 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:24:46.104 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:24:46.130 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:24:46.175 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:24:46.176 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:24:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:24:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:24:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:24:46.368 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:24:46.371 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:24:46.413 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:24:46.459 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:24:46.494 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:24:46.503 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:24:46.551 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:24:46.555 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:24:46.602 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:24:46.632 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:24:46.671 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:24:46.673 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:24:46.692 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:24:46.770 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:24:46.772 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:24:46.799 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:24:46.839 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:24:46.865 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:24:46.889 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:24:46.939 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:24:46.963 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:24:47.028 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:24:47.086 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:24:47.090 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:24:47.130 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:24:47.158 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:24:47.192 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:24:47.231 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:24:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:24:47.263 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:24:47.288 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:24:47.293 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:24:47.327 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:24:47.361 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:24:47.376 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:24:47.402 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:24:47.405 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:24:47.437 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:24:47.455 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:24:47.475 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:24:47.522 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:24:47.569 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:24:47.575 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:24:47.607 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:24:47.653 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:24:47.750 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:24:47.803 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:24:47.816 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:24:47.857 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:24:48.054 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:24:48.183 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:24:48.222 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:24:48.269 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:24:48.364 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:24:48.449 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:24:48.521 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:24:48.525 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:24:48.572 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:24:48.576 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:24:48.606 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:24:48.614 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:24:48.702 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:24:48.737 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:24:48.755 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:24:48.815 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:24:48.844 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:24:48.862 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:24:48.868 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:24:48.875 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:24:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:24:48.955 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:24:48.983 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:24:49.016 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:24:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:24:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:24:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:24:49.087 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:24:49.137 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:24:49.159 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:24:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:24:49.301 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:24:49.344 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:24:49.382 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:24:49.386 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:24:49.478 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:24:49.507 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:24:49.508 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:24:49.544 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:24:49.686 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:24:49.796 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:24:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:24:50.042 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:24:50.073 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:24:50.127 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:24:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:24:50.482 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:24:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:24:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:24:50.712 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:24:50.827 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:24:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:24:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:24:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:24:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:24:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:24:51.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:24:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:24:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:24:51.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:24:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:24:51.602 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:24:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:24:51.682 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:24:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:24:51.751 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:24:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:24:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:24:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:24:52.237 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:24:52.295 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:24:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:24:52.400 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:24:52.422 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:24:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:24:52.485 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:24:52.521 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:24:52.573 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:24:52.618 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:24:52.669 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:24:52.709 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:24:52.741 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:24:52.926 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:24:53.089 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:24:53.112 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:24:53.132 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:24:53.156 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:24:53.194 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:24:53.224 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:24:53.224 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:24:53.224 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:24:53.224 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:24:53.241 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:24:53.250 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:24:53.301 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:24:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:24:53.453 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:24:53.525 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:24:53.601 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:24:53.705 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:24:53.782 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:24:53.993 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:24:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:24:54.154 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:24:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:24:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:24:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:24:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:24:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:24:55.271 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:24:55.305 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:24:55.391 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:24:55.391 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:24:55.391 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:24:55.391 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:24:55.423 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:24:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:24:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:24:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:24:55.927 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:24:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:24:56.226 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:24:56.532 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:24:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:24:56.771 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:24:57.706 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:24:57.884 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:24:59.587 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:24:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:24:59.759 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:24:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:25:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:25:00.257 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:25:00.380 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:25:00.414 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:25:00.470 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:25:01.064 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:25:01.094 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:25:01.194 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:25:01.220 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:25:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:25:04.092 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:25:04.769 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:25:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:25:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:25:04.979 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:25:04.979 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:25:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:05.587 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:25:05.602 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:25:05.603 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:05.603 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:25:05.603 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:06.905 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:25:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-04-26T01:25:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-04-26T01:25:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-04-26T01:25:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-04-26T01:25:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-04-26T01:25:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-04-26T01:25:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-04-26T01:25:06.906 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:07.139 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:25:07.166 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:25:07.166 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:07.166 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:25:07.166 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:25:07.166 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:25:07.166 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:07.408 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:25:07.432 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:25:07.432 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:07.432 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:25:07.432 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:25:07.432 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:25:07.432 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:07.901 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:25:07.924 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:25:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:25:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:25:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:25:07.925 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:08.040 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:25:08.098 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:25:08.161 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:25:08.357 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:25:08.402 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:25:08.482 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:25:08.571 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:25:08.606 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:25:08.631 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:25:08.631 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:08.631 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:25:08.631 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:25:08.631 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:25:08.631 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:08.722 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:25:08.760 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:25:08.792 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:25:08.820 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:25:08.864 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:25:08.887 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:25:08.887 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:08.887 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:25:08.887 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:25:08.887 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:25:08.888 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:09.265 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:25:09.293 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:25:09.325 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:25:09.348 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:25:09.371 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:25:09.371 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:09.371 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:25:09.371 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:25:09.371 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:25:09.372 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:09.388 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:25:09.416 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:25:09.478 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:25:09.526 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:25:09.582 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:25:09.767 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:25:09.815 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:25:09.885 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:25:09.980 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:25:10.121 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:25:10.166 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:25:10.198 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:25:10.226 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:25:10.551 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:25:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:25:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:25:10.758 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:25:10.813 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:25:11.565 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:25:11.593 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:25:11.848 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:25:11.872 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:25:11.883 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:25:11.903 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:25:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:25:11.928 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:11.928 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:25:11.928 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:25:11.928 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:25:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:11.979 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:25:11.990 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:25:12.092 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:25:12.146 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:25:12.853 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:25:12.976 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:25:13.121 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:25:13.157 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:25:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:25:13.180 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:25:13.180 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:13.180 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:25:13.181 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:25:13.181 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:25:13.181 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:25:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:25:13.208 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:13.208 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:25:13.208 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:25:13.208 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:25:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:13.224 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:25:13.247 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:25:13.247 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:13.247 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:25:13.247 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:13.252 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:25:13.264 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:25:13.291 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:25:13.315 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:25:13.315 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:13.315 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:25:13.315 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:25:13.316 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:25:13.316 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:13.361 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:25:13.417 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:25:13.634 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:25:13.657 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:25:13.658 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:13.658 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:25:13.658 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:25:13.658 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:25:13.658 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:14.340 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:25:14.371 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:25:14.394 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:25:14.394 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:14.395 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:25:14.395 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:25:14.395 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:25:14.395 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:14.439 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:25:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:25:14.461 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:14.461 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:25:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:14.506 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:25:14.528 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:25:14.529 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:14.529 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:25:14.529 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:25:14.529 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:25:14.529 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:14.840 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:25:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:25:14.863 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:14.863 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:25:14.863 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:25:14.864 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:25:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:25:18.546 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:25:18.595 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:25:18.627 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:25:18.654 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:25:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:25:18.727 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:25:18.727 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:18.727 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:25:18.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:18.848 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:25:18.908 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:25:18.938 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:25:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:25:19.510 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:25:19.514 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:19.550 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:25:19.560 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:19.581 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:19.582 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:25:19.610 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:25:19.792 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:25:19.840 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:25:19.869 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:25:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:25:20.484 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:25:20.493 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:25:20.504 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:20.504 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:25:20.504 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:25:20.504 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:25:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:20.582 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:25:20.624 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:25:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:25:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:25:20.861 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:25:20.862 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:25:20.879 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:25:20.879 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:20.879 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:25:20.879 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:25:20.879 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:25:20.879 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:20.954 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:25:20.995 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:25:20.995 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:25:21.088 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:25:21.164 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:21.394 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:25:21.421 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:25:21.481 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:25:21.521 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:25:21.521 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:25:21.521 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:21.521 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:25:21.522 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:25:21.522 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:25:21.522 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:21.602 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:25:21.634 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:25:21.635 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:25:21.674 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:25:21.719 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:25:21.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:25:21.760 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:25:21.760 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:25:21.823 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:25:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:25:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:25:22.154 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:25:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:25:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:25:22.576 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:25:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:25:22.676 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:25:22.708 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:25:23.716 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:25:23.752 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:25:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:25:23.928 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:25:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:25:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:25:25.373 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:25:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:25:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:25:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:25:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:25:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:25:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:25:27.085 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:27.085 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:25:27.086 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:25:27.086 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:25:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:25:27.130 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:25:27.131 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:25:27.132 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:25:27.134 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:25:27.135 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:25:27.136 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:25:27.137 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:25:27.137 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:25:27.137 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:25:27.137 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:25:27.137 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:25:27.137 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:25:27.137 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:25:27.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:25:27.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:25:27.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:25:27.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:25:27.142 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:25:27.142 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:25:27.142 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:25:27.142 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:25:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:25:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:25:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:25:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:25:27.759 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:27.759 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:25:27.759 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:25:27.759 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:25:27.759 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:25:27.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:25:27.761 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:25:27.762 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:25:27.762 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:25:27.776 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:25:27.777 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:25:27.777 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:25:27.777 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:25:27.777 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:25:27.777 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:25:27.777 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:25:27.778 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:25:27.778 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:25:27.778 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:25:27.778 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:25:27.778 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:25:27.779 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:25:27.780 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:25:27.781 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:25:27.782 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:25:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:25:27.784 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:25:27.785 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:25:27.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:25:27.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:25:27.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:25:27.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:25:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:25:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:25:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:25:28.665 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:28.665 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:25:28.665 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:25:28.665 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:25:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:25:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:25:28.805 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:28.805 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:25:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:25:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:25:28.986 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:28.986 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:25:28.986 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:25:28.986 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:25:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:29.223 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:29.223 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.224 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.225 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:25:29.226 INFO:teuthology.orchestra.run.smithi053.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:25:29.227 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:25:29.228 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:25:29.229 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:25:29.230 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:25:29.231 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:25:29.232 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:25:29.233 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.234 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-26T01:25:29.235 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-04-26T01:25:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:25:29.452 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:25:29.453 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:25:29.453 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:25:29.453 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:25:29.453 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:25:29.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:29.747 DEBUG:teuthology.parallel:result is None 2024-04-26T01:25:29.856 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:29.856 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.857 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.858 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:25:29.859 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:25:29.860 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:25:29.861 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:25:29.862 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:25:29.863 INFO:teuthology.orchestra.run.smithi169.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:25:29.864 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:25:29.865 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:25:29.866 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:25:29.867 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:25:29.868 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:25:29.869 INFO:teuthology.orchestra.run.smithi169.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:25:29.869 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:25:29.869 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-26T01:25:30.246 DEBUG:teuthology.parallel:result is None 2024-04-26T01:25:34.779 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:25:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:25:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:25:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:25:34.996 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:25:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:25:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:25:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:25:36.512 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:36.559 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:36.616 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:25:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:25:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:25:38.318 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:25:38.508 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:25:38.556 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:25:38.556 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:25:38.749 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:25:38.796 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:25:38.797 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:25:38.947 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:25:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:39.437 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:39.437 INFO:teuthology.orchestra.run.smithi028.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T01:25:39.437 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:45.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:25:45.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:25:45.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:25:45.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:25:45.309 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:25:45.310 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:25:45.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:25:45.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:25:45.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:25:45.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:25:45.322 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:25:45.322 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:25:45.322 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:25:45.323 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:25:45.324 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:25:45.325 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:25:45.326 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:25:45.327 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:25:45.328 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:25:45.329 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:25:45.330 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:25:45.331 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:25:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:25:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:25:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:25:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:25:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:25:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:25:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:25:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:25:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:25:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:25:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:25:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:25:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:25:47.777 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-26T01:25:48.445 DEBUG:teuthology.parallel:result is None 2024-04-26T01:25:48.445 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:25:48.732 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:25:48.792 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:25:48.792 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:25:48.792 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:25:48.794 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:25:49.115 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:25:49.142 INFO:teuthology.orchestra.run.smithi053.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:25:49.142 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:25:49.142 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:25:49.144 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:25:49.472 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:25:49.498 INFO:teuthology.orchestra.run.smithi169.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:25:49.498 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:25:49.498 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:25:49.500 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T01:25:49.501 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:25:49.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:25:49.570 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:25:49.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:25:49.603 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:25:49.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:25:49.634 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T01:25:49.635 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:25:49.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:25:49.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:25:49.804 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:25:49.804 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:25:49.830 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:25:49.895 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:25:49.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:25:49.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:25:49.991 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T01:25:49.991 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:25:49.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:25:50.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:25:50.167 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:25:50.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:25:50.192 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:25:50.257 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:25:50.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:25:50.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:25:50.345 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T01:25:50.345 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:25:50.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:25:50.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:25:50.519 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:25:50.519 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:25:50.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:25:50.607 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:25:50.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:25:50.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:25:50.694 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T01:25:50.791 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-26T01:25:50.791 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:25:50.793 INFO:tasks.cephadm:Cluster fsid is ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:25:50.793 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T01:25:50.793 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.53', 'mon.c': '172.21.15.169'} 2024-04-26T01:25:50.794 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-04-26T01:25:50.794 INFO:tasks.cephadm:First mgr is x 2024-04-26T01:25:50.794 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T01:25:50.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-04-26T01:25:50.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2024-04-26T01:25:50.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2024-04-26T01:25:50.909 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T01:25:50.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:25:51.140 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T01:25:51.355 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:25:51.356 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-26T01:25:51.356 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:25 /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:51.995 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:52.650 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:25 /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:52.650 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:53.080 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:25 /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:53.081 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:53.126 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:53.145 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:25:53.180 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-26T01:25:53.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:25:53.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:25:53.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:25:54.592 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:25:54.774 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:25:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:26:35.219 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:26:35.220 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:26:35.220 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:26:35.220 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-04-26T01:26:35.220 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:26:35.220 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:26:35.220 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:27:01.293 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-26T01:27:01.294 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:27:01.294 INFO:teuthology.orchestra.run.smithi053.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:27:01.294 INFO:teuthology.orchestra.run.smithi053.stdout: "repo_digests": [ 2024-04-26T01:27:01.294 INFO:teuthology.orchestra.run.smithi053.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:27:01.294 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-04-26T01:27:01.294 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-26T01:27:03.362 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-04-26T01:27:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:27:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:27:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "repo_digests": [ 2024-04-26T01:27:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:27:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-04-26T01:27:03.363 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-04-26T01:27:03.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-04-26T01:27:03.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2024-04-26T01:27:03.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2024-04-26T01:27:03.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-04-26T01:27:03.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2024-04-26T01:27:03.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2024-04-26T01:27:03.687 INFO:tasks.cephadm:Writing seed config... 2024-04-26T01:27:03.688 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T01:27:03.688 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T01:27:03.689 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T01:27:03.690 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T01:27:03.691 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T01:27:03.692 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T01:27:03.692 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T01:27:03.692 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:27:03.692 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T01:27:03.753 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ead407d4-036b-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T01:27:03.754 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T01:27:03.801 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T01:27:03.847 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T01:27:03.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid ead407d4-036b-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:27:04.123 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:27:04.123 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', 'ead407d4-036b-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-04-26T01:27:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T01:27:04.124 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-04-26T01:27:04.181 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T01:27:04.181 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-04-26T01:27:04.181 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-04-26T01:27:04.200 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T01:27:04.201 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T01:27:04.219 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T01:27:04.219 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-26T01:27:04.244 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-26T01:27:04.266 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-26T01:27:04.267 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-26T01:27:04.267 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-04-26T01:27:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T01:27:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T01:27:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-04-26T01:27:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-04-26T01:27:04.350 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T01:27:04.350 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T01:27:04.368 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T01:27:04.368 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-26T01:27:04.387 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-26T01:27:04.406 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-26T01:27:04.406 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-26T01:27:04.407 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-04-26T01:27:04.407 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:04.407 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 140382858947744 on /run/cephadm/ead407d4-036b-11ef-bc93-c7b262605968.lock 2024-04-26T01:27:04.407 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 140382858947744 acquired on /run/cephadm/ead407d4-036b-11ef-bc93-c7b262605968.lock 2024-04-26T01:27:04.407 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-04-26T01:27:04.408 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-04-26T01:27:04.408 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-26T01:27:04.417 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-04-26T01:27:04.417 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-04-26T01:27:04.425 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T01:27:04.425 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T01:27:04.425 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T01:27:04.432 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T01:27:04.432 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T01:27:04.432 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T01:27:04.432 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T01:27:04.432 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-26T01:27:04.432 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T01:27:04.433 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-26T01:27:04.433 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-26T01:27:04.433 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T01:27:04.434 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T01:27:04.435 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:27:05.156 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-26T01:27:05.156 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:27:05.156 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T01:27:05.156 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-26T01:27:05.156 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-26T01:27:05.156 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-26T01:27:05.157 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T01:27:06.421 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:27:06.421 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:27:06.421 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T01:27:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-04-26T01:27:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-04-26T01:27:08.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDsAitmmXg2BhAAxDAscRlfe/igeOQkPAKCDA== 2024-04-26T01:27:09.804 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDtAitmT5bJDRAA+PHpsprLOh6XzTZZzttYyg== 2024-04-26T01:27:11.123 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDuAitmK/U1EhAAvsktZs8AQaWtSPvgwvlsWQ== 2024-04-26T01:27:11.126 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-04-26T01:27:12.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T01:27:12.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T01:27:12.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:12.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T01:27:12.333 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T01:27:12.333 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-04-26T01:27:12.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:12.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T01:27:12.334 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:27:12.334 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-04-26T01:27:13.546 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-04-26T01:27:13.919 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T01:27:14.159 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T01:27:14.454 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ead407d4-036b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-ead407d4-036b-11ef-bc93-c7b262605968.target. 2024-04-26T01:27:14.454 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ead407d4-036b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-ead407d4-036b-11ef-bc93-c7b262605968.target. 2024-04-26T01:27:14.808 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a 2024-04-26T01:27:14.808 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T01:27:15.133 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ead407d4-036b-11ef-bc93-c7b262605968.target.wants/ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-ead407d4-036b-11ef-bc93-c7b262605968@.service. 2024-04-26T01:27:15.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:15 smithi028 podman[27861]: 2024-04-26 01:27:15.448853548 +0000 UTC m=+0.032447178 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:27:15.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:15 smithi028 podman[27861]: 2024-04-26 01:27:15.794243003 +0000 UTC m=+0.377836641 container init 28d0d260348ee485489e7a590d70670e0f4cd63f95bc894f7d7e1ff61330b087 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T01:27:15.926 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:27:15.926 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-26T01:27:15.945 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:27:15.945 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-26T01:27:15.945 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-26T01:27:15.945 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T01:27:15.946 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-04-26T01:27:15.946 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-04-26T01:27:16.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:15 smithi028 podman[27861]: 2024-04-26 01:27:15.804663996 +0000 UTC m=+0.388257657 container start 28d0d260348ee485489e7a590d70670e0f4cd63f95bc894f7d7e1ff61330b087 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:27:16.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:15 smithi028 ceph-mon[27879]: mkfs ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:16.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:15 smithi028 ceph-mon[27879]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:27:16.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:15 smithi028 bash[27861]: 28d0d260348ee485489e7a590d70670e0f4cd63f95bc894f7d7e1ff61330b087 2024-04-26T01:27:16.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:15 smithi028 systemd[1]: Started Ceph mon.a for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:27:17.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:27:17.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: monmap epoch 1 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: last_changed 2024-04-26T01:27:11.642338+0000 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: min_mon_release 19 (squid) 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: election_strategy: 1 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: fsmap 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: mgrmap e1: no daemons active 2024-04-26T01:27:17.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:16 smithi028 ceph-mon[27879]: from='client.? 172.21.15.28:0/1497868993' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T01:27:17.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T01:27:17.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:17.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T01:27:17.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:17.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-04-26T01:27:17.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.811251s) 2024-04-26T01:27:17.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.600+0000 7f163d400700 1 Processor -- start 2024-04-26T01:27:17.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.601+0000 7f163d400700 1 -- start start 2024-04-26T01:27:17.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.601+0000 7f163d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f1638110430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:17.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.601+0000 7f163d400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1638110970 con 0x7f1638110010 2024-04-26T01:27:17.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.602+0000 7f1636800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f1638110430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:17.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.602+0000 7f1636800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f1638110430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39098/0 (socket says 172.21.15.28:39098) 2024-04-26T01:27:17.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.602+0000 7f1636800700 1 -- 172.21.15.28:0/2444122967 learned_addr learned my addr 172.21.15.28:0/2444122967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:17.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.603+0000 7f1636800700 1 -- 172.21.15.28:0/2444122967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1638110af0 con 0x7f1638110010 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.603+0000 7f1636800700 1 --2- 172.21.15.28:0/2444122967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f1638110430 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f16280098a0 tx=0x7f1628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615637fb419493f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.603+0000 7f1635e00700 1 -- 172.21.15.28:0/2444122967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1628004160 con 0x7f1638110010 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.603+0000 7f1635e00700 1 -- 172.21.15.28:0/2444122967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1628004300 con 0x7f1638110010 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.603+0000 7f1635e00700 1 -- 172.21.15.28:0/2444122967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16280036a0 con 0x7f1638110010 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.604+0000 7f163d400700 1 -- 172.21.15.28:0/2444122967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 msgr2=0x7f1638110430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.604+0000 7f163d400700 1 --2- 172.21.15.28:0/2444122967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f1638110430 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f16280098a0 tx=0x7f1628009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.604+0000 7f163d400700 1 -- 172.21.15.28:0/2444122967 shutdown_connections 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.604+0000 7f163d400700 1 --2- 172.21.15.28:0/2444122967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f1638110430 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.604+0000 7f163d400700 1 -- 172.21.15.28:0/2444122967 >> 172.21.15.28:0/2444122967 conn(0x7f163807da60 msgr2=0x7f163807fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.604+0000 7f163d400700 1 -- 172.21.15.28:0/2444122967 shutdown_connections 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.604+0000 7f163d400700 1 -- 172.21.15.28:0/2444122967 wait complete. 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.605+0000 7f163d400700 1 Processor -- start 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.606+0000 7f163d400700 1 -- start start 2024-04-26T01:27:17.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.606+0000 7f163d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f16381a2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.606+0000 7f163d400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f163810efc0 con 0x7f1638110010 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.606+0000 7f1636800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f16381a2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.606+0000 7f1636800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f16381a2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39100/0 (socket says 172.21.15.28:39100) 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.606+0000 7f1636800700 1 -- 172.21.15.28:0/1497868993 learned_addr learned my addr 172.21.15.28:0/1497868993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.607+0000 7f1636800700 1 -- 172.21.15.28:0/1497868993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16381a2580 con 0x7f1638110010 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.607+0000 7f1636800700 1 --2- 172.21.15.28:0/1497868993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f16381a2040 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1628000c00 tx=0x7f1628004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.607+0000 7f162fe00700 1 -- 172.21.15.28:0/1497868993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1628003db0 con 0x7f1638110010 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.607+0000 7f162fe00700 1 -- 172.21.15.28:0/1497868993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1628011630 con 0x7f1638110010 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.607+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16381a2810 con 0x7f1638110010 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.607+0000 7f162fe00700 1 -- 172.21.15.28:0/1497868993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1628011950 con 0x7f1638110010 2024-04-26T01:27:17.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.607+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16381a5500 con 0x7f1638110010 2024-04-26T01:27:17.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.608+0000 7f162fe00700 1 -- 172.21.15.28:0/1497868993 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1628011af0 con 0x7f1638110010 2024-04-26T01:27:17.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.608+0000 7f162fe00700 1 -- 172.21.15.28:0/1497868993 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f162801c450 con 0x7f1638110010 2024-04-26T01:27:17.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.609+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16380640a0 con 0x7f1638110010 2024-04-26T01:27:17.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.612+0000 7f162fe00700 1 -- 172.21.15.28:0/1497868993 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1628032020 con 0x7f1638110010 2024-04-26T01:27:17.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.692+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f163819b500 con 0x7f1638110010 2024-04-26T01:27:17.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.693+0000 7f162fe00700 1 -- 172.21.15.28:0/1497868993 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f163819b500 con 0x7f1638110010 2024-04-26T01:27:17.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.695+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 msgr2=0x7f16381a2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:17.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.695+0000 7f163d400700 1 --2- 172.21.15.28:0/1497868993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f16381a2040 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1628000c00 tx=0x7f1628004750 comp rx=0 tx=0).stop 2024-04-26T01:27:17.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.695+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 shutdown_connections 2024-04-26T01:27:17.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.695+0000 7f163d400700 1 --2- 172.21.15.28:0/1497868993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1638110010 0x7f16381a2040 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:17.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.695+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 >> 172.21.15.28:0/1497868993 conn(0x7f163807da60 msgr2=0x7f1638196500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:17.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.696+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 shutdown_connections 2024-04-26T01:27:17.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:16.696+0000 7f163d400700 1 -- 172.21.15.28:0/1497868993 wait complete. 2024-04-26T01:27:17.351 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-04-26T01:27:17.351 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T01:27:18.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:18 smithi028 ceph-mon[27879]: from='client.? 172.21.15.28:0/3908847071' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T01:27:18.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:18 smithi028 ceph-mon[27879]: from='client.? 172.21.15.28:0/3908847071' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T01:27:18.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:18.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T01:27:18.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:18.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T01:27:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-26T01:27:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T01:27:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T01:27:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T01:27:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T01:27:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:18.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T01:27:18.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T01:27:18.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:18.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T01:27:18.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T01:27:18.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T01:27:18.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T01:27:18.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.978+0000 7f86f6200700 1 Processor -- start 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.978+0000 7f86f6200700 1 -- start start 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.978+0000 7f86f6200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f010a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.978+0000 7f86f6200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86f010af50 con 0x7f86f010a560 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.979+0000 7f86ef400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f010a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.979+0000 7f86ef400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f010a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39108/0 (socket says 172.21.15.28:39108) 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.979+0000 7f86ef400700 1 -- 172.21.15.28:0/2624582776 learned_addr learned my addr 172.21.15.28:0/2624582776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:18.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.980+0000 7f86ef400700 1 -- 172.21.15.28:0/2624582776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f010b760 con 0x7f86f010a560 2024-04-26T01:27:18.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.980+0000 7f86ef400700 1 --2- 172.21.15.28:0/2624582776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f010a980 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f86e00098a0 tx=0x7f86e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9859af8d6ec48f38 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:18.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.980+0000 7f86eea00700 1 -- 172.21.15.28:0/2624582776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86e0004160 con 0x7f86f010a560 2024-04-26T01:27:18.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.981+0000 7f86eea00700 1 -- 172.21.15.28:0/2624582776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86e0004300 con 0x7f86f010a560 2024-04-26T01:27:18.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.981+0000 7f86eea00700 1 -- 172.21.15.28:0/2624582776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86e00036a0 con 0x7f86f010a560 2024-04-26T01:27:18.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.981+0000 7f86f6200700 1 -- 172.21.15.28:0/2624582776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 msgr2=0x7f86f010a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:18.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.981+0000 7f86f6200700 1 --2- 172.21.15.28:0/2624582776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f010a980 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f86e00098a0 tx=0x7f86e0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:18.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.981+0000 7f86f6200700 1 -- 172.21.15.28:0/2624582776 shutdown_connections 2024-04-26T01:27:18.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.981+0000 7f86f6200700 1 --2- 172.21.15.28:0/2624582776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f010a980 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:18.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.981+0000 7f86f6200700 1 -- 172.21.15.28:0/2624582776 >> 172.21.15.28:0/2624582776 conn(0x7f86f007dbf0 msgr2=0x7f86f01084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:18.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.982+0000 7f86f6200700 1 -- 172.21.15.28:0/2624582776 shutdown_connections 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.982+0000 7f86f6200700 1 -- 172.21.15.28:0/2624582776 wait complete. 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.983+0000 7f86f6200700 1 Processor -- start 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.983+0000 7f86f6200700 1 -- start start 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.984+0000 7f86f6200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.984+0000 7f86f6200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86f010bc90 con 0x7f86f010a560 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.984+0000 7f86ef400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.984+0000 7f86ef400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f019db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39124/0 (socket says 172.21.15.28:39124) 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.984+0000 7f86ef400700 1 -- 172.21.15.28:0/3908847071 learned_addr learned my addr 172.21.15.28:0/3908847071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.985+0000 7f86ef400700 1 -- 172.21.15.28:0/3908847071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f019e0d0 con 0x7f86f010a560 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.985+0000 7f86ef400700 1 --2- 172.21.15.28:0/3908847071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f019db90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f86e0000c00 tx=0x7f86e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.985+0000 7f86ecc00700 1 -- 172.21.15.28:0/3908847071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86e0003db0 con 0x7f86f010a560 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.985+0000 7f86ecc00700 1 -- 172.21.15.28:0/3908847071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86e0011630 con 0x7f86f010a560 2024-04-26T01:27:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.985+0000 7f86ecc00700 1 -- 172.21.15.28:0/3908847071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86e0011950 con 0x7f86f010a560 2024-04-26T01:27:18.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.985+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f019e360 con 0x7f86f010a560 2024-04-26T01:27:18.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.985+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f01a1050 con 0x7f86f010a560 2024-04-26T01:27:18.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.986+0000 7f86ecc00700 1 -- 172.21.15.28:0/3908847071 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f86e0011af0 con 0x7f86f010a560 2024-04-26T01:27:18.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.986+0000 7f86ecc00700 1 -- 172.21.15.28:0/3908847071 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f86e001c450 con 0x7f86f010a560 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.987+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f00640a0 con 0x7f86f010a560 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:17.991+0000 7f86ecc00700 1 -- 172.21.15.28:0/3908847071 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f86e0032020 con 0x7f86f010a560 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.068+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f86f002e180 con 0x7f86f010a560 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.074+0000 7f86ecc00700 1 -- 172.21.15.28:0/3908847071 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f86e0004260 con 0x7f86f010a560 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.075+0000 7f86ecc00700 1 -- 172.21.15.28:0/3908847071 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f86e001c6a0 con 0x7f86f010a560 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.077+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 msgr2=0x7f86f019db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.077+0000 7f86f6200700 1 --2- 172.21.15.28:0/3908847071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f019db90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f86e0000c00 tx=0x7f86e0004750 comp rx=0 tx=0).stop 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.077+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 shutdown_connections 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.077+0000 7f86f6200700 1 --2- 172.21.15.28:0/3908847071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86f010a560 0x7f86f019db90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.077+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 >> 172.21.15.28:0/3908847071 conn(0x7f86f007dbf0 msgr2=0x7f86f0194870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.077+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 shutdown_connections 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:18.077+0000 7f86f6200700 1 -- 172.21.15.28:0/3908847071 wait complete. 2024-04-26T01:27:18.691 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-04-26T01:27:19.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:19 smithi028 ceph-mon[27879]: from='client.? 172.21.15.28:0/1119243854' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:27:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.345+0000 7f7837e00700 1 Processor -- start 2024-04-26T01:27:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.346+0000 7f7837e00700 1 -- start start 2024-04-26T01:27:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.346+0000 7f7837e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783010a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.346+0000 7f7837e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f783010af60 con 0x7f783010a570 2024-04-26T01:27:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.346+0000 7f7835400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783010a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.347+0000 7f7835400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783010a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48040/0 (socket says 172.21.15.28:48040) 2024-04-26T01:27:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.347+0000 7f7835400700 1 -- 172.21.15.28:0/2135908055 learned_addr learned my addr 172.21.15.28:0/2135908055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.347+0000 7f7835400700 1 -- 172.21.15.28:0/2135908055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783010b770 con 0x7f783010a570 2024-04-26T01:27:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.347+0000 7f7835400700 1 --2- 172.21.15.28:0/2135908055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783010a990 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f78240098a0 tx=0x7f7824009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2541033f0a0265 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.347+0000 7f782fe00700 1 -- 172.21.15.28:0/2135908055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7824004160 con 0x7f783010a570 2024-04-26T01:27:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.348+0000 7f782fe00700 1 -- 172.21.15.28:0/2135908055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7824004300 con 0x7f783010a570 2024-04-26T01:27:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.348+0000 7f782fe00700 1 -- 172.21.15.28:0/2135908055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7824011af0 con 0x7f783010a570 2024-04-26T01:27:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.348+0000 7f7837e00700 1 -- 172.21.15.28:0/2135908055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 msgr2=0x7f783010a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.348+0000 7f7837e00700 1 --2- 172.21.15.28:0/2135908055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783010a990 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f78240098a0 tx=0x7f7824009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.348+0000 7f7837e00700 1 -- 172.21.15.28:0/2135908055 shutdown_connections 2024-04-26T01:27:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.348+0000 7f7837e00700 1 --2- 172.21.15.28:0/2135908055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783010a990 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.348+0000 7f7837e00700 1 -- 172.21.15.28:0/2135908055 >> 172.21.15.28:0/2135908055 conn(0x7f783007da30 msgr2=0x7f78301084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.348+0000 7f7837e00700 1 -- 172.21.15.28:0/2135908055 shutdown_connections 2024-04-26T01:27:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.349+0000 7f7837e00700 1 -- 172.21.15.28:0/2135908055 wait complete. 2024-04-26T01:27:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.349+0000 7f7837e00700 1 Processor -- start 2024-04-26T01:27:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.350+0000 7f7837e00700 1 -- start start 2024-04-26T01:27:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.350+0000 7f7837e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.350+0000 7f7837e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f783010eb20 con 0x7f783010a570 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.350+0000 7f7835400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.350+0000 7f7835400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48042/0 (socket says 172.21.15.28:48042) 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.350+0000 7f7835400700 1 -- 172.21.15.28:0/1119243854 learned_addr learned my addr 172.21.15.28:0/1119243854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.351+0000 7f7835400700 1 -- 172.21.15.28:0/1119243854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783019e920 con 0x7f783010a570 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.351+0000 7f7835400700 1 --2- 172.21.15.28:0/1119243854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783019e3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7824000c00 tx=0x7f7824011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.351+0000 7f782e000700 1 -- 172.21.15.28:0/1119243854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7824003b20 con 0x7f783010a570 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.351+0000 7f782e000700 1 -- 172.21.15.28:0/1119243854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f782401a8c0 con 0x7f783010a570 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.351+0000 7f782e000700 1 -- 172.21.15.28:0/1119243854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7824024940 con 0x7f783010a570 2024-04-26T01:27:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.351+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783019ebb0 con 0x7f783010a570 2024-04-26T01:27:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.352+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78301a18a0 con 0x7f783010a570 2024-04-26T01:27:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.352+0000 7f782e000700 1 -- 172.21.15.28:0/1119243854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7824022020 con 0x7f783010a570 2024-04-26T01:27:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.352+0000 7f782e000700 1 -- 172.21.15.28:0/1119243854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f782402ddd0 con 0x7f783010a570 2024-04-26T01:27:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.353+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78300640a0 con 0x7f783010a570 2024-04-26T01:27:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.356+0000 7f782e000700 1 -- 172.21.15.28:0/1119243854 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f782401aa70 con 0x7f783010a570 2024-04-26T01:27:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.434+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f78301a1de0 con 0x7f783010a570 2024-04-26T01:27:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.435+0000 7f782e000700 1 -- 172.21.15.28:0/1119243854 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f7824032030 con 0x7f783010a570 2024-04-26T01:27:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.436+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 msgr2=0x7f783019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.436+0000 7f7837e00700 1 --2- 172.21.15.28:0/1119243854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783019e3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7824000c00 tx=0x7f7824011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.436+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 shutdown_connections 2024-04-26T01:27:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.436+0000 7f7837e00700 1 --2- 172.21.15.28:0/1119243854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f783010a570 0x7f783019e3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.436+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 >> 172.21.15.28:0/1119243854 conn(0x7f783007da30 msgr2=0x7f7830107b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.436+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 shutdown_connections 2024-04-26T01:27:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:19.436+0000 7f7837e00700 1 -- 172.21.15.28:0/1119243854 wait complete. 2024-04-26T01:27:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-04-26T01:27:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:20 smithi028 systemd[1]: Stopping Ceph mon.a for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:27:21.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:20 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a[27875]: 2024-04-26T01:27:20.826+0000 7f6fc0800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:27:21.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:20 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a[27875]: 2024-04-26T01:27:20.826+0000 7f6fc0800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T01:27:21.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:20 smithi028 podman[28121]: 2024-04-26 01:27:20.961130126 +0000 UTC m=+0.493595701 container died 28d0d260348ee485489e7a590d70670e0f4cd63f95bc894f7d7e1ff61330b087 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:27:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:21 smithi028 podman[28121]: 2024-04-26 01:27:21.324528593 +0000 UTC m=+0.856994158 container cleanup 28d0d260348ee485489e7a590d70670e0f4cd63f95bc894f7d7e1ff61330b087 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:27:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:21 smithi028 bash[28121]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a 2024-04-26T01:27:21.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:21 smithi028 podman[28135]: 2024-04-26 01:27:21.508497491 +0000 UTC m=+0.531037658 container remove 28d0d260348ee485489e7a590d70670e0f4cd63f95bc894f7d7e1ff61330b087 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-26T01:27:22.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T01:27:22.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 systemd[1]: Stopped Ceph mon.a for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:27:22.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a.service: Consumed 2.010s CPU time. 2024-04-26T01:27:22.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 systemd[1]: Starting Ceph mon.a for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:27:22.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 podman[28228]: 2024-04-26 01:27:22.409312831 +0000 UTC m=+0.121837106 container create 8553833424dfc95a9afec126ca28043a2ce913c7332b1a50934eab3365cce796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True) 2024-04-26T01:27:22.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 podman[28228]: 2024-04-26 01:27:22.320829161 +0000 UTC m=+0.033353411 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:27:22.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 podman[28228]: 2024-04-26 01:27:22.67905859 +0000 UTC m=+0.391582909 container init 8553833424dfc95a9afec126ca28043a2ce913c7332b1a50934eab3365cce796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:27:22.775 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 podman[28228]: 2024-04-26 01:27:22.691330989 +0000 UTC m=+0.403855248 container start 8553833424dfc95a9afec126ca28043a2ce913c7332b1a50934eab3365cce796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: pidfile_write: ignore empty --pid-file 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: load: jerasure load: lrc 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: RocksDB version: 7.9.2 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Git sha 0 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: DB SUMMARY 2024-04-26T01:27:22.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: DB Session ID: VEQ3WKVD3VJP3YJEPSNQ 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: CURRENT file: CURRENT 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104540 ; 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.error_if_exists: 0 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.create_if_missing: 0 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.paranoid_checks: 1 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.env: 0x55b086ef9cc0 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T01:27:22.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.info_log: 0x55b099007c40 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.statistics: (nil) 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.use_fsync: 0 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_log_file_size: 0 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.allow_fallocate: 1 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T01:27:22.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.use_direct_reads: 0 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.db_log_dir: 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.wal_dir: 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T01:27:22.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.write_buffer_manager: 0x55b098f9b0e0 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.unordered_write: 0 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T01:27:22.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.row_cache: None 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.wal_filter: None 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.two_write_queues: 0 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.wal_compression: 0 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.atomic_flush: 0 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T01:27:22.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.log_readahead_size: 0 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_background_jobs: 2 2024-04-26T01:27:22.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_background_compactions: -1 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_subcompactions: 1 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_open_files: -1 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T01:27:22.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_background_flushes: -1 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Compression algorithms supported: 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: kZSTD supported: 0 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: kXpressCompression supported: 0 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: kZlibCompression supported: 1 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: kSnappyCompression supported: 1 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: kLZ4Compression supported: 1 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: kBZip2Compression supported: 0 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T01:27:22.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.merge_operator: 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_filter: None 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b099007e00) 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-04-26T01:27:22.935 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x55b098fcd1f0 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-04-26T01:27:22.936 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-04-26T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression: NoCompression 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.num_levels: 7 2024-04-26T01:27:22.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T01:27:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T01:27:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T01:27:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T01:27:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T01:27:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T01:27:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:27:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T01:27:22.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T01:27:22.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:27:22.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T01:27:22.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T01:27:22.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T01:27:22.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.inplace_update_support: 0 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.bloom_locality: 0 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.max_successive_merges: 0 2024-04-26T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.ttl: 2592000 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.enable_blob_files: false 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.min_blob_size: 0 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T01:27:22.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9400b92d-617f-4f4f-a4a5-8807e29a4b09 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094842752736, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094842754371, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99767, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97649, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14028, "raw_average_key_size": 49, "raw_value_size": 90007, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714094842, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9400b92d-617f-4f4f-a4a5-8807e29a4b09", "db_session_id": "VEQ3WKVD3VJP3YJEPSNQ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094842754532, "job": 1, "event": "recovery_finished"} 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T01:27:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b0990e2000 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: DB pointer 0x55b0990cc000 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T01:27:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 99.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 99.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T01:27:22.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 94.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 13.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 13.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x55b098fcd1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-26T01:27:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a@-1(???) e1 preinit fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a@-1(???).mds e1 new map 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a@-1(???).mds e1 print_map 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-04-26T01:27:22.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 bash[28228]: 8553833424dfc95a9afec126ca28043a2ce913c7332b1a50934eab3365cce796 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 systemd[1]: Started Ceph mon.a for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: monmap epoch 1 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: last_changed 2024-04-26T01:27:11.642338+0000 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: min_mon_release 19 (squid) 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: election_strategy: 1 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:27:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: fsmap 2024-04-26T01:27:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T01:27:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:22 smithi028 ceph-mon[28247]: mgrmap e1: no daemons active 2024-04-26T01:27:24.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.496+0000 7fd7b8000700 1 Processor -- start 2024-04-26T01:27:24.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.497+0000 7fd7b8000700 1 -- start start 2024-04-26T01:27:24.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.497+0000 7fd7b8000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b0108790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:24.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.497+0000 7fd7b8000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b0108d60 con 0x7fd7b0108370 2024-04-26T01:27:24.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.498+0000 7fd7b5600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b0108790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:24.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.498+0000 7fd7b5600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b0108790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48052/0 (socket says 172.21.15.28:48052) 2024-04-26T01:27:24.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.498+0000 7fd7b5600700 1 -- 172.21.15.28:0/3491368186 learned_addr learned my addr 172.21.15.28:0/3491368186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.499+0000 7fd7b5600700 1 -- 172.21.15.28:0/3491368186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b0109520 con 0x7fd7b0108370 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.499+0000 7fd7b5600700 1 --2- 172.21.15.28:0/3491368186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b0108790 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40098a0 tx=0x7fd7a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76dea063c86f4b15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.499+0000 7fd7afe00700 1 -- 172.21.15.28:0/3491368186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7a4004160 con 0x7fd7b0108370 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.500+0000 7fd7afe00700 1 -- 172.21.15.28:0/3491368186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd7a4004300 con 0x7fd7b0108370 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.500+0000 7fd7afe00700 1 -- 172.21.15.28:0/3491368186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7a4011af0 con 0x7fd7b0108370 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.500+0000 7fd7b8000700 1 -- 172.21.15.28:0/3491368186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 msgr2=0x7fd7b0108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.500+0000 7fd7b8000700 1 --2- 172.21.15.28:0/3491368186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b0108790 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40098a0 tx=0x7fd7a4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.501+0000 7fd7b8000700 1 -- 172.21.15.28:0/3491368186 shutdown_connections 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.501+0000 7fd7b8000700 1 --2- 172.21.15.28:0/3491368186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b0108790 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.501+0000 7fd7b8000700 1 -- 172.21.15.28:0/3491368186 >> 172.21.15.28:0/3491368186 conn(0x7fd7b01038f0 msgr2=0x7fd7b0105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.501+0000 7fd7b8000700 1 -- 172.21.15.28:0/3491368186 shutdown_connections 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.501+0000 7fd7b8000700 1 -- 172.21.15.28:0/3491368186 wait complete. 2024-04-26T01:27:24.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.502+0000 7fd7b8000700 1 Processor -- start 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.502+0000 7fd7b8000700 1 -- start start 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.502+0000 7fd7b8000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b007db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.502+0000 7fd7b8000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b010c8d0 con 0x7fd7b0108370 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.503+0000 7fd7b5600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b007db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.503+0000 7fd7b5600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b007db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48060/0 (socket says 172.21.15.28:48060) 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.503+0000 7fd7b5600700 1 -- 172.21.15.28:0/3902413149 learned_addr learned my addr 172.21.15.28:0/3902413149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.503+0000 7fd7b5600700 1 -- 172.21.15.28:0/3902413149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b007e0a0 con 0x7fd7b0108370 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.503+0000 7fd7b5600700 1 --2- 172.21.15.28:0/3902413149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b007db60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4000c00 tx=0x7fd7a4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.504+0000 7fd7ae000700 1 -- 172.21.15.28:0/3902413149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7a4003b20 con 0x7fd7b0108370 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.504+0000 7fd7ae000700 1 -- 172.21.15.28:0/3902413149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd7a401a8c0 con 0x7fd7b0108370 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.504+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b007c290 con 0x7fd7b0108370 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.504+0000 7fd7ae000700 1 -- 172.21.15.28:0/3902413149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7a4024940 con 0x7fd7b0108370 2024-04-26T01:27:24.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.504+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b007c7d0 con 0x7fd7b0108370 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.504+0000 7fd7ae000700 1 -- 172.21.15.28:0/3902413149 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd7a4022020 con 0x7fd7b0108370 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.504+0000 7fd7ae000700 1 -- 172.21.15.28:0/3902413149 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd7a40193c0 con 0x7fd7b0108370 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.505+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b00640a0 con 0x7fd7b0108370 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.508+0000 7fd7ae000700 1 -- 172.21.15.28:0/3902413149 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd7a40042c0 con 0x7fd7b0108370 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.586+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fd7b007ca80 con 0x7fd7b0108370 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.590+0000 7fd7ae000700 1 -- 172.21.15.28:0/3902413149 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fd7a402dde0 con 0x7fd7b0108370 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.592+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 msgr2=0x7fd7b007db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.592+0000 7fd7b8000700 1 --2- 172.21.15.28:0/3902413149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b007db60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4000c00 tx=0x7fd7a4011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.592+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 shutdown_connections 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.592+0000 7fd7b8000700 1 --2- 172.21.15.28:0/3902413149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b0108370 0x7fd7b007db60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.592+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 >> 172.21.15.28:0/3902413149 conn(0x7fd7b01038f0 msgr2=0x7fd7b01043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.592+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 shutdown_connections 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:23.592+0000 7fd7b8000700 1 -- 172.21.15.28:0/3902413149 wait complete. 2024-04-26T01:27:24.215 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T01:27:24.239 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:27:24.239 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-04-26T01:27:24.240 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T01:27:24.241 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T01:27:24.673 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.x 2024-04-26T01:27:24.673 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T01:27:24.949 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ead407d4-036b-11ef-bc93-c7b262605968.target.wants/ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-ead407d4-036b-11ef-bc93-c7b262605968@.service. 2024-04-26T01:27:24.988 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:24 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3902413149' entity='client.admin' 2024-04-26T01:27:25.280 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:24 smithi028 systemd[1]: Starting Ceph mgr.x for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:27:25.615 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:25 smithi028 podman[28459]: 2024-04-26 01:27:25.263689383 +0000 UTC m=+0.041193167 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:27:25.615 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:25 smithi028 podman[28459]: 2024-04-26 01:27:25.396788384 +0000 UTC m=+0.174292205 container create 2e5fee773f32d12ce66150ab282a951475ece1bbac1a17fa1b2dec77b2496710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-26T01:27:25.615 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:25 smithi028 podman[28459]: 2024-04-26 01:27:25.617080422 +0000 UTC m=+0.394584249 container init 2e5fee773f32d12ce66150ab282a951475ece1bbac1a17fa1b2dec77b2496710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T01:27:25.750 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:27:25.750 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-26T01:27:25.767 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:27:25.768 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-26T01:27:25.768 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-26T01:27:25.768 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T01:27:25.788 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:27:25.788 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-26T01:27:25.805 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:27:25.806 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-26T01:27:25.806 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-26T01:27:25.806 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T01:27:25.806 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-04-26T01:27:25.806 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-04-26T01:27:25.871 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:25 smithi028 podman[28459]: 2024-04-26 01:27:25.628663373 +0000 UTC m=+0.406167201 container start 2e5fee773f32d12ce66150ab282a951475ece1bbac1a17fa1b2dec77b2496710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True) 2024-04-26T01:27:25.872 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:25 smithi028 bash[28459]: 2e5fee773f32d12ce66150ab282a951475ece1bbac1a17fa1b2dec77b2496710 2024-04-26T01:27:25.872 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:25 smithi028 systemd[1]: Started Ceph mgr.x for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:27:26.133 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:25.875+0000 7fefcbb26200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:27:26.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:26.148+0000 7fefcbb26200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:27:26.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:26.287+0000 7fefcbb26200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:27:26.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:26 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3122653400' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:27:27.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:26.714+0000 7fefcbb26200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:27:27.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:26.837+0000 7fefcbb26200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:27.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:27:27.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:27:27.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:27:27.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:27:27.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:27:27.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:27:27.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:27.229 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:27:15.884536+0000", 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.426+0000 7fdbcf800700 1 Processor -- start 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.427+0000 7fdbcf800700 1 -- start start 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.427+0000 7fdbcf800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc807c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.427+0000 7fdbcf800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc807c8a0 con 0x7fdbc807de70 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.428+0000 7fdbcce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc807c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.428+0000 7fdbcce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc807c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48098/0 (socket says 172.21.15.28:48098) 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.428+0000 7fdbcce00700 1 -- 172.21.15.28:0/3597027403 learned_addr learned my addr 172.21.15.28:0/3597027403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.429+0000 7fdbcce00700 1 -- 172.21.15.28:0/3597027403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc807ca20 con 0x7fdbc807de70 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.429+0000 7fdbcce00700 1 --2- 172.21.15.28:0/3597027403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc807c2d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0098a0 tx=0x7fdbbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d338907b03a5a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.429+0000 7fdbc7400700 1 -- 172.21.15.28:0/3597027403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbbc004160 con 0x7fdbc807de70 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.429+0000 7fdbc7400700 1 -- 172.21.15.28:0/3597027403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdbbc004300 con 0x7fdbc807de70 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.429+0000 7fdbc7400700 1 -- 172.21.15.28:0/3597027403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbbc011af0 con 0x7fdbc807de70 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.430+0000 7fdbcf800700 1 -- 172.21.15.28:0/3597027403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 msgr2=0x7fdbc807c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.430+0000 7fdbcf800700 1 --2- 172.21.15.28:0/3597027403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc807c2d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0098a0 tx=0x7fdbbc009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.430+0000 7fdbcf800700 1 -- 172.21.15.28:0/3597027403 shutdown_connections 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.430+0000 7fdbcf800700 1 --2- 172.21.15.28:0/3597027403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc807c2d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.430+0000 7fdbcf800700 1 -- 172.21.15.28:0/3597027403 >> 172.21.15.28:0/3597027403 conn(0x7fdbc81038e0 msgr2=0x7fdbc8105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:27.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.430+0000 7fdbcf800700 1 -- 172.21.15.28:0/3597027403 shutdown_connections 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.430+0000 7fdbcf800700 1 -- 172.21.15.28:0/3597027403 wait complete. 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.432+0000 7fdbcf800700 1 Processor -- start 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.432+0000 7fdbcf800700 1 -- start start 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.432+0000 7fdbcf800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc819e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.432+0000 7fdbcf800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc810e8d0 con 0x7fdbc807de70 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.432+0000 7fdbcce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc819e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.432+0000 7fdbcce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc819e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48100/0 (socket says 172.21.15.28:48100) 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.432+0000 7fdbcce00700 1 -- 172.21.15.28:0/3122653400 learned_addr learned my addr 172.21.15.28:0/3122653400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.433+0000 7fdbcce00700 1 -- 172.21.15.28:0/3122653400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc819e8c0 con 0x7fdbc807de70 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.433+0000 7fdbcce00700 1 --2- 172.21.15.28:0/3122653400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc819e380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc011ac0 tx=0x7fdbbc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.434+0000 7fdbc5600700 1 -- 172.21.15.28:0/3122653400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbbc003b20 con 0x7fdbc807de70 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.434+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc819eb50 con 0x7fdbc807de70 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.434+0000 7fdbc5600700 1 -- 172.21.15.28:0/3122653400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdbbc01a8c0 con 0x7fdbc807de70 2024-04-26T01:27:27.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.434+0000 7fdbc5600700 1 -- 172.21.15.28:0/3122653400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbbc0248d0 con 0x7fdbc807de70 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.434+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc81a1840 con 0x7fdbc807de70 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.434+0000 7fdbc5600700 1 -- 172.21.15.28:0/3122653400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdbbc022020 con 0x7fdbc807de70 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.434+0000 7fdbc5600700 1 -- 172.21.15.28:0/3122653400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdbbc02dde0 con 0x7fdbc807de70 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.435+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc80640a0 con 0x7fdbc807de70 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.438+0000 7fdbc5600700 1 -- 172.21.15.28:0/3122653400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdbbc042e50 con 0x7fdbc807de70 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.520+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdbc81a1a10 con 0x7fdbc807de70 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.521+0000 7fdbc5600700 1 -- 172.21.15.28:0/3122653400 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fdbc81a1a10 con 0x7fdbc807de70 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.522+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 msgr2=0x7fdbc819e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.522+0000 7fdbcf800700 1 --2- 172.21.15.28:0/3122653400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc819e380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc011ac0 tx=0x7fdbbc011ba0 comp rx=0 tx=0).stop 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.522+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 shutdown_connections 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.523+0000 7fdbcf800700 1 --2- 172.21.15.28:0/3122653400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc807de70 0x7fdbc819e380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.523+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 >> 172.21.15.28:0/3122653400 conn(0x7fdbc81038e0 msgr2=0x7fdbc81043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:27.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.523+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 shutdown_connections 2024-04-26T01:27:27.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:26.523+0000 7fdbcf800700 1 -- 172.21.15.28:0/3122653400 wait complete. 2024-04-26T01:27:27.234 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-04-26T01:27:28.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:28 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:28.480+0000 7fefcbb26200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:27:28.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:28 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:28.714+0000 7fefcbb26200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:27:29.280 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:28 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:28.938+0000 7fefcbb26200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:27:29.661 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:29 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:29.339+0000 7fefcbb26200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:27:29.661 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:29 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:29.467+0000 7fefcbb26200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:27:29.913 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:29 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:29.735+0000 7fefcbb26200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:27:30.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:29 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2861051356' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:27:30.265 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:30 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:30.017+0000 7fefcbb26200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:27:30.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:30.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:27:30.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:27:30.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:27:30.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:27:30.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:30.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-26T01:27:30.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:27:30.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:30.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:27:30.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:27:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:27:15.884536+0000", 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:27:30.562 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.814+0000 7fc54c800700 1 Processor -- start 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.815+0000 7fc54c800700 1 -- start start 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.815+0000 7fc54c800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54407c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.815+0000 7fc54c800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc54407c880 con 0x7fc54407de50 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.815+0000 7fc549e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54407c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.816+0000 7fc549e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54407c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47818/0 (socket says 172.21.15.28:47818) 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.816+0000 7fc549e00700 1 -- 172.21.15.28:0/2060255182 learned_addr learned my addr 172.21.15.28:0/2060255182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.816+0000 7fc549e00700 1 -- 172.21.15.28:0/2060255182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc54407ca00 con 0x7fc54407de50 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.816+0000 7fc549e00700 1 --2- 172.21.15.28:0/2060255182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54407c2b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc5380098a0 tx=0x7fc538009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e718eff8100504df server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.817+0000 7fc548a00700 1 -- 172.21.15.28:0/2060255182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc538004160 con 0x7fc54407de50 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.817+0000 7fc548a00700 1 -- 172.21.15.28:0/2060255182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc538004300 con 0x7fc54407de50 2024-04-26T01:27:30.563 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.817+0000 7fc548a00700 1 -- 172.21.15.28:0/2060255182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc538011af0 con 0x7fc54407de50 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.818+0000 7fc54c800700 1 -- 172.21.15.28:0/2060255182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 msgr2=0x7fc54407c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.818+0000 7fc54c800700 1 --2- 172.21.15.28:0/2060255182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54407c2b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc5380098a0 tx=0x7fc538009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.818+0000 7fc54c800700 1 -- 172.21.15.28:0/2060255182 shutdown_connections 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.818+0000 7fc54c800700 1 --2- 172.21.15.28:0/2060255182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54407c2b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.818+0000 7fc54c800700 1 -- 172.21.15.28:0/2060255182 >> 172.21.15.28:0/2060255182 conn(0x7fc5441038e0 msgr2=0x7fc544105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.818+0000 7fc54c800700 1 -- 172.21.15.28:0/2060255182 shutdown_connections 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.818+0000 7fc54c800700 1 -- 172.21.15.28:0/2060255182 wait complete. 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.819+0000 7fc54c800700 1 Processor -- start 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.819+0000 7fc54c800700 1 -- start start 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.819+0000 7fc54c800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54419e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.819+0000 7fc54c800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc54410e890 con 0x7fc54407de50 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.820+0000 7fc549e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54419e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.820+0000 7fc549e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54419e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47824/0 (socket says 172.21.15.28:47824) 2024-04-26T01:27:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.820+0000 7fc549e00700 1 -- 172.21.15.28:0/2861051356 learned_addr learned my addr 172.21.15.28:0/2861051356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.820+0000 7fc549e00700 1 -- 172.21.15.28:0/2861051356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc54419e8e0 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.820+0000 7fc549e00700 1 --2- 172.21.15.28:0/2861051356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54419e3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc538011ac0 tx=0x7fc538011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.820+0000 7fc53ea00700 1 -- 172.21.15.28:0/2861051356 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc538003b20 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.821+0000 7fc53ea00700 1 -- 172.21.15.28:0/2861051356 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc53801a8c0 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.821+0000 7fc53ea00700 1 -- 172.21.15.28:0/2861051356 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5380248d0 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.821+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc54419eb70 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.821+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5441a1860 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.821+0000 7fc53ea00700 1 -- 172.21.15.28:0/2861051356 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc538022020 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.821+0000 7fc53ea00700 1 -- 172.21.15.28:0/2861051356 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc53802dde0 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.822+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5440640a0 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.825+0000 7fc53ea00700 1 -- 172.21.15.28:0/2861051356 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc538042e50 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.917+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc5441a1db0 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.918+0000 7fc53ea00700 1 -- 172.21.15.28:0/2861051356 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc538031020 con 0x7fc54407de50 2024-04-26T01:27:30.565 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.919+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 msgr2=0x7fc54419e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.919+0000 7fc54c800700 1 --2- 172.21.15.28:0/2861051356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54419e3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc538011ac0 tx=0x7fc538011ba0 comp rx=0 tx=0).stop 2024-04-26T01:27:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.919+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 shutdown_connections 2024-04-26T01:27:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.919+0000 7fc54c800700 1 --2- 172.21.15.28:0/2861051356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54407de50 0x7fc54419e3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.919+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 >> 172.21.15.28:0/2861051356 conn(0x7fc5441038e0 msgr2=0x7fc544104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.920+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 shutdown_connections 2024-04-26T01:27:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:29.920+0000 7fc54c800700 1 -- 172.21.15.28:0/2861051356 wait complete. 2024-04-26T01:27:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-04-26T01:27:31.280 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:30 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:30.847+0000 7fefcbb26200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:27:31.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:31.415+0000 7fefcbb26200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:27:31.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:31.545+0000 7fefcbb26200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:27:32.280 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:31.820+0000 7fefcbb26200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:27:32.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:32.488+0000 7fefcbb26200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:27:32.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:32.619+0000 7fefcbb26200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:27:33.144 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:32.958+0000 7fefcbb26200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:27:33.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:33 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3751203381' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:27:33.439 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:33.147+0000 7fefcbb26200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:27:33.440 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:33.278+0000 7fefcbb26200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:27:33.440 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:33.443+0000 7fefcbb26200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:27:34.015 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:33.837+0000 7fefcbb26200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:27:34.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:34.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:27:34.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:27:34.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:27:34.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:27:34.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:27:34.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:27:34.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:34.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:34.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:27:34.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:27:34.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:27:34.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:27:34.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:27:34.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:27:34.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:27:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:27:15.884536+0000", 2024-04-26T01:27:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.193+0000 7fcf24000700 1 Processor -- start 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.194+0000 7fcf24000700 1 -- start start 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.194+0000 7fcf24000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.194+0000 7fcf24000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c108da0 con 0x7fcf1c1083b0 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.195+0000 7fcf21600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.195+0000 7fcf21600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47840/0 (socket says 172.21.15.28:47840) 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.195+0000 7fcf21600700 1 -- 172.21.15.28:0/634231482 learned_addr learned my addr 172.21.15.28:0/634231482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.195+0000 7fcf21600700 1 -- 172.21.15.28:0/634231482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c109560 con 0x7fcf1c1083b0 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.195+0000 7fcf21600700 1 --2- 172.21.15.28:0/634231482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c1087d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcf100098a0 tx=0x7fcf10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad69f709da0e45f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.196+0000 7fcf1be00700 1 -- 172.21.15.28:0/634231482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf10004160 con 0x7fcf1c1083b0 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.196+0000 7fcf1be00700 1 -- 172.21.15.28:0/634231482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcf10004300 con 0x7fcf1c1083b0 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.196+0000 7fcf1be00700 1 -- 172.21.15.28:0/634231482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf10011af0 con 0x7fcf1c1083b0 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.196+0000 7fcf24000700 1 -- 172.21.15.28:0/634231482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 msgr2=0x7fcf1c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.196+0000 7fcf24000700 1 --2- 172.21.15.28:0/634231482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c1087d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcf100098a0 tx=0x7fcf10009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.197+0000 7fcf24000700 1 -- 172.21.15.28:0/634231482 shutdown_connections 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.197+0000 7fcf24000700 1 --2- 172.21.15.28:0/634231482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c1087d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.197+0000 7fcf24000700 1 -- 172.21.15.28:0/634231482 >> 172.21.15.28:0/634231482 conn(0x7fcf1c1039b0 msgr2=0x7fcf1c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.197+0000 7fcf24000700 1 -- 172.21.15.28:0/634231482 shutdown_connections 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.197+0000 7fcf24000700 1 -- 172.21.15.28:0/634231482 wait complete. 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.198+0000 7fcf24000700 1 Processor -- start 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.198+0000 7fcf24000700 1 -- start start 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.198+0000 7fcf24000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.198+0000 7fcf24000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c10c910 con 0x7fcf1c1083b0 2024-04-26T01:27:34.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.199+0000 7fcf21600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.199+0000 7fcf21600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47854/0 (socket says 172.21.15.28:47854) 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.199+0000 7fcf21600700 1 -- 172.21.15.28:0/3751203381 learned_addr learned my addr 172.21.15.28:0/3751203381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.199+0000 7fcf21600700 1 -- 172.21.15.28:0/3751203381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c19e940 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.199+0000 7fcf21600700 1 --2- 172.21.15.28:0/3751203381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c19e400 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf10000c00 tx=0x7fcf10011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.200+0000 7fcf1a000700 1 -- 172.21.15.28:0/3751203381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf10003b20 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.200+0000 7fcf1a000700 1 -- 172.21.15.28:0/3751203381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcf1001a8c0 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.200+0000 7fcf1a000700 1 -- 172.21.15.28:0/3751203381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf10024940 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.200+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1c19ebd0 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.200+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1c1a18c0 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.200+0000 7fcf1a000700 1 -- 172.21.15.28:0/3751203381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcf10022020 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.200+0000 7fcf1a000700 1 -- 172.21.15.28:0/3751203381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcf1002dd90 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.201+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf1c0640a0 con 0x7fcf1c1083b0 2024-04-26T01:27:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.204+0000 7fcf1a000700 1 -- 172.21.15.28:0/3751203381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcf10042e50 con 0x7fcf1c1083b0 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.283+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcf1c1a1e00 con 0x7fcf1c1083b0 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.283+0000 7fcf1a000700 1 -- 172.21.15.28:0/3751203381 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fcf10031020 con 0x7fcf1c1083b0 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.285+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 msgr2=0x7fcf1c19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.285+0000 7fcf24000700 1 --2- 172.21.15.28:0/3751203381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c19e400 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf10000c00 tx=0x7fcf10011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.285+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 shutdown_connections 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.285+0000 7fcf24000700 1 --2- 172.21.15.28:0/3751203381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c1083b0 0x7fcf1c19e400 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.285+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 >> 172.21.15.28:0/3751203381 conn(0x7fcf1c1039b0 msgr2=0x7fcf1c1043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.285+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 shutdown_connections 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:33.285+0000 7fcf24000700 1 -- 172.21.15.28:0/3751203381 wait complete. 2024-04-26T01:27:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-04-26T01:27:34.925 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:34.642+0000 7fefcbb26200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:27:35.280 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:34.937+0000 7fefcbb26200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: Activating manager daemon x 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: mgrmap e2: x(active, starting, since 0.00495607s) 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: Manager daemon x is now available 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' 2024-04-26T01:27:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:27:35.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' 2024-04-26T01:27:35.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:34 smithi028 ceph-mon[28247]: from='mgr.14100 172.21.15.28:0/67559021' entity='mgr.x' 2024-04-26T01:27:37.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:36 smithi028 ceph-mon[28247]: mgrmap e3: x(active, since 1.02088s) 2024-04-26T01:27:38.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:37 smithi028 ceph-mon[28247]: mgrmap e4: x(active, since 2s) 2024-04-26T01:27:38.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:37 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1499378652' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:27:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:27:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:27:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:27:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:27:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:27:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:27:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:27:38.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:38.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:27:38.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:27:38.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:38.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:27:38.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:38.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:27:38.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:27:38.110 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:27:38.111 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:27:15.884536+0000", 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:27:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.856+0000 7f2d9c800700 1 Processor -- start 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.857+0000 7f2d9c800700 1 -- start start 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.857+0000 7f2d9c800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9410a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.857+0000 7f2d9c800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9410af90 con 0x7f2d9410a5a0 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.857+0000 7f2d99e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9410a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.857+0000 7f2d99e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9410a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47916/0 (socket says 172.21.15.28:47916) 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.857+0000 7f2d99e00700 1 -- 172.21.15.28:0/1348830134 learned_addr learned my addr 172.21.15.28:0/1348830134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.858+0000 7f2d99e00700 1 -- 172.21.15.28:0/1348830134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9410b750 con 0x7f2d9410a5a0 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.858+0000 7f2d99e00700 1 --2- 172.21.15.28:0/1348830134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9410a9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2d880098a0 tx=0x7f2d88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e98821762a80254 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d98a00700 1 -- 172.21.15.28:0/1348830134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d88004160 con 0x7f2d9410a5a0 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d98a00700 1 -- 172.21.15.28:0/1348830134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2d88004300 con 0x7f2d9410a5a0 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d98a00700 1 -- 172.21.15.28:0/1348830134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d88011af0 con 0x7f2d9410a5a0 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d9c800700 1 -- 172.21.15.28:0/1348830134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 msgr2=0x7f2d9410a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d9c800700 1 --2- 172.21.15.28:0/1348830134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9410a9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2d880098a0 tx=0x7f2d88009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:38.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d9c800700 1 -- 172.21.15.28:0/1348830134 shutdown_connections 2024-04-26T01:27:38.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d9c800700 1 --2- 172.21.15.28:0/1348830134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9410a9c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:38.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d9c800700 1 -- 172.21.15.28:0/1348830134 >> 172.21.15.28:0/1348830134 conn(0x7f2d9407dba0 msgr2=0x7f2d941084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:38.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.859+0000 7f2d9c800700 1 -- 172.21.15.28:0/1348830134 shutdown_connections 2024-04-26T01:27:38.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.860+0000 7f2d9c800700 1 -- 172.21.15.28:0/1348830134 wait complete. 2024-04-26T01:27:38.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.861+0000 7f2d9c800700 1 Processor -- start 2024-04-26T01:27:38.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.861+0000 7f2d9c800700 1 -- start start 2024-04-26T01:27:38.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.861+0000 7f2d9c800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:38.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.861+0000 7f2d9c800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9410eb00 con 0x7f2d9410a5a0 2024-04-26T01:27:38.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.861+0000 7f2d99e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:38.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.862+0000 7f2d99e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47920/0 (socket says 172.21.15.28:47920) 2024-04-26T01:27:38.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.862+0000 7f2d99e00700 1 -- 172.21.15.28:0/1499378652 learned_addr learned my addr 172.21.15.28:0/1499378652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:38.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.862+0000 7f2d99e00700 1 -- 172.21.15.28:0/1499378652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9419e920 con 0x7f2d9410a5a0 2024-04-26T01:27:38.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.862+0000 7f2d99e00700 1 --2- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9419e3e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2d88000c00 tx=0x7f2d88011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:38.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.862+0000 7f2d8ea00700 1 -- 172.21.15.28:0/1499378652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d88003b20 con 0x7f2d9410a5a0 2024-04-26T01:27:38.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.863+0000 7f2d8ea00700 1 -- 172.21.15.28:0/1499378652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2d8801a8c0 con 0x7f2d9410a5a0 2024-04-26T01:27:38.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.863+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9419ebb0 con 0x7f2d9410a5a0 2024-04-26T01:27:38.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.863+0000 7f2d8ea00700 1 -- 172.21.15.28:0/1499378652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d88024940 con 0x7f2d9410a5a0 2024-04-26T01:27:38.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.863+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d941a18a0 con 0x7f2d9410a5a0 2024-04-26T01:27:38.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.864+0000 7f2d8ea00700 1 -- 172.21.15.28:0/1499378652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7f2d88022020 con 0x7f2d9410a5a0 2024-04-26T01:27:38.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.864+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d940640a0 con 0x7f2d9410a5a0 2024-04-26T01:27:38.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.864+0000 7f2d8ea00700 1 --2- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f2d7c0395a0 0x7f2d7c03ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:38.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.864+0000 7f2d8ea00700 1 -- 172.21.15.28:0/1499378652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2d8804d280 con 0x7f2d9410a5a0 2024-04-26T01:27:38.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.865+0000 7f2d99400700 1 --2- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f2d7c0395a0 0x7f2d7c03ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:38.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.865+0000 7f2d99400700 1 --2- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f2d7c0395a0 0x7f2d7c03ba60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2d84006fd0 tx=0x7f2d84006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:38.122 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.871+0000 7f2d8ea00700 1 -- 172.21.15.28:0/1499378652 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d8802ad60 con 0x7f2d9410a5a0 2024-04-26T01:27:38.122 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:36.955+0000 7f2d8ea00700 1 -- 172.21.15.28:0/1499378652 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f2d8801aa70 con 0x7f2d9410a5a0 2024-04-26T01:27:38.122 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.190+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2d941a1a70 con 0x7f2d9410a5a0 2024-04-26T01:27:38.122 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.192+0000 7f2d8ea00700 1 -- 172.21.15.28:0/1499378652 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2d941a1a70 con 0x7f2d9410a5a0 2024-04-26T01:27:38.122 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.194+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f2d7c0395a0 msgr2=0x7f2d7c03ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:38.123 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.194+0000 7f2d9c800700 1 --2- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f2d7c0395a0 0x7f2d7c03ba60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2d84006fd0 tx=0x7f2d84006e40 comp rx=0 tx=0).stop 2024-04-26T01:27:38.123 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.194+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 msgr2=0x7f2d9419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:38.123 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.194+0000 7f2d9c800700 1 --2- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9419e3e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2d88000c00 tx=0x7f2d88011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:38.123 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.195+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 shutdown_connections 2024-04-26T01:27:38.123 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.195+0000 7f2d9c800700 1 --2- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f2d7c0395a0 0x7f2d7c03ba60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:38.124 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.195+0000 7f2d9c800700 1 --2- 172.21.15.28:0/1499378652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9410a5a0 0x7f2d9419e3e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:38.124 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.195+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 >> 172.21.15.28:0/1499378652 conn(0x7f2d9407dba0 msgr2=0x7f2d94107b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:38.124 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.195+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 shutdown_connections 2024-04-26T01:27:38.124 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:37.195+0000 7f2d9c800700 1 -- 172.21.15.28:0/1499378652 wait complete. 2024-04-26T01:27:38.124 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-04-26T01:27:39.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:39 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2173346604' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T01:27:39.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:39.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T01:27:39.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:39.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T01:27:39.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T01:27:39.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.743+0000 7f8ede200700 1 Processor -- start 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.743+0000 7f8ede200700 1 -- start start 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.744+0000 7f8ede200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed8107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.744+0000 7f8ede200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed8107c30 con 0x7f8ed8107240 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.744+0000 7f8ed7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed8107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.745+0000 7f8ed7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed8107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47932/0 (socket says 172.21.15.28:47932) 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.745+0000 7f8ed7400700 1 -- 172.21.15.28:0/3148153342 learned_addr learned my addr 172.21.15.28:0/3148153342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:39.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.745+0000 7f8ed7400700 1 -- 172.21.15.28:0/3148153342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed8108440 con 0x7f8ed8107240 2024-04-26T01:27:39.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.745+0000 7f8ed7400700 1 --2- 172.21.15.28:0/3148153342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed8107660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80098a0 tx=0x7f8ec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc8d6a9f5e873fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:39.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.746+0000 7f8ed6000700 1 -- 172.21.15.28:0/3148153342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ec8004160 con 0x7f8ed8107240 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.746+0000 7f8ed6000700 1 -- 172.21.15.28:0/3148153342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8ec8004300 con 0x7f8ed8107240 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.746+0000 7f8ed6000700 1 -- 172.21.15.28:0/3148153342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ec8011af0 con 0x7f8ed8107240 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.746+0000 7f8ede200700 1 -- 172.21.15.28:0/3148153342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 msgr2=0x7f8ed8107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.746+0000 7f8ede200700 1 --2- 172.21.15.28:0/3148153342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed8107660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80098a0 tx=0x7f8ec8009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.746+0000 7f8ede200700 1 -- 172.21.15.28:0/3148153342 shutdown_connections 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.746+0000 7f8ede200700 1 --2- 172.21.15.28:0/3148153342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed8107660 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.746+0000 7f8ede200700 1 -- 172.21.15.28:0/3148153342 >> 172.21.15.28:0/3148153342 conn(0x7f8ed8102820 msgr2=0x7f8ed8104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.747+0000 7f8ede200700 1 -- 172.21.15.28:0/3148153342 shutdown_connections 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.747+0000 7f8ede200700 1 -- 172.21.15.28:0/3148153342 wait complete. 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.748+0000 7f8ede200700 1 Processor -- start 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.748+0000 7f8ede200700 1 -- start start 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.748+0000 7f8ede200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed819e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:39.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.748+0000 7f8ede200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed807afc0 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.748+0000 7f8ed7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed819e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.748+0000 7f8ed7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed819e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47934/0 (socket says 172.21.15.28:47934) 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.748+0000 7f8ed7400700 1 -- 172.21.15.28:0/2173346604 learned_addr learned my addr 172.21.15.28:0/2173346604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.749+0000 7f8ed7400700 1 -- 172.21.15.28:0/2173346604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed819e990 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.749+0000 7f8ed7400700 1 --2- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed819e450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8000c00 tx=0x7f8ec8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.749+0000 7f8ecfe00700 1 -- 172.21.15.28:0/2173346604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ec8003b20 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.749+0000 7f8ecfe00700 1 -- 172.21.15.28:0/2173346604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8ec801a8c0 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.749+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed819ec20 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.749+0000 7f8ecfe00700 1 -- 172.21.15.28:0/2173346604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ec8024940 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.750+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed81a1910 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.751+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed80640a0 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.751+0000 7f8ecfe00700 1 -- 172.21.15.28:0/2173346604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f8ec8022020 con 0x7f8ed8107240 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.751+0000 7f8ecfe00700 1 --2- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f8ebc039670 0x7f8ebc03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:39.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.751+0000 7f8ecfe00700 1 -- 172.21.15.28:0/2173346604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8ec804d490 con 0x7f8ed8107240 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.752+0000 7f8ed6a00700 1 --2- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f8ebc039670 0x7f8ebc03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.753+0000 7f8ed6a00700 1 --2- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f8ebc039670 0x7f8ebc03bb30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4006fd0 tx=0x7f8ec4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.757+0000 7f8ecfe00700 1 -- 172.21.15.28:0/2173346604 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ec804d1f0 con 0x7f8ed8107240 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.988+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f8ed8076600 con 0x7f8ed8107240 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.991+0000 7f8ecfe00700 1 -- 172.21.15.28:0/2173346604 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f8ec802ad60 con 0x7f8ed8107240 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.993+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f8ebc039670 msgr2=0x7f8ebc03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 --2- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f8ebc039670 0x7f8ebc03bb30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4006fd0 tx=0x7f8ec4006e40 comp rx=0 tx=0).stop 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 msgr2=0x7f8ed819e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:39.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 --2- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed819e450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8000c00 tx=0x7f8ec8011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:39.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 shutdown_connections 2024-04-26T01:27:39.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 --2- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f8ebc039670 0x7f8ebc03bb30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:39.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 --2- 172.21.15.28:0/2173346604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ed8107240 0x7f8ed819e450 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:39.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 >> 172.21.15.28:0/2173346604 conn(0x7f8ed8102820 msgr2=0x7f8ed810b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:39.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 shutdown_connections 2024-04-26T01:27:39.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:38.994+0000 7f8ede200700 1 -- 172.21.15.28:0/2173346604 wait complete. 2024-04-26T01:27:39.599 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-04-26T01:27:40.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:40 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2648441491' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T01:27:40.900 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: ignoring --setuser ceph since I am not root 2024-04-26T01:27:40.900 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: ignoring --setgroup ceph since I am not root 2024-04-26T01:27:41.152 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:40.903+0000 7f7446217200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:27:41.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.232+0000 7f388d600700 1 Processor -- start 2024-04-26T01:27:41.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.232+0000 7f388d600700 1 -- start start 2024-04-26T01:27:41.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.232+0000 7f388d600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388810a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:41.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.232+0000 7f388d600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f388810ad80 con 0x7f388810a390 2024-04-26T01:27:41.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.233+0000 7f3887e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388810a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:41.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.233+0000 7f3887e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388810a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40620/0 (socket says 172.21.15.28:40620) 2024-04-26T01:27:41.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.233+0000 7f3887e00700 1 -- 172.21.15.28:0/444734124 learned_addr learned my addr 172.21.15.28:0/444734124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:41.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.234+0000 7f3887e00700 1 -- 172.21.15.28:0/444734124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388810b590 con 0x7f388810a390 2024-04-26T01:27:41.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.234+0000 7f3887e00700 1 --2- 172.21.15.28:0/444734124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388810a7b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f38780098a0 tx=0x7f3878009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0f9e17ce6cbaac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:41.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.234+0000 7f3886a00700 1 -- 172.21.15.28:0/444734124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3878004160 con 0x7f388810a390 2024-04-26T01:27:41.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.234+0000 7f3886a00700 1 -- 172.21.15.28:0/444734124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3878004300 con 0x7f388810a390 2024-04-26T01:27:41.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.234+0000 7f3886a00700 1 -- 172.21.15.28:0/444734124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3878011af0 con 0x7f388810a390 2024-04-26T01:27:41.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.235+0000 7f388d600700 1 -- 172.21.15.28:0/444734124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 msgr2=0x7f388810a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:41.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.235+0000 7f388d600700 1 --2- 172.21.15.28:0/444734124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388810a7b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f38780098a0 tx=0x7f3878009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:41.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.235+0000 7f388d600700 1 -- 172.21.15.28:0/444734124 shutdown_connections 2024-04-26T01:27:41.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.235+0000 7f388d600700 1 --2- 172.21.15.28:0/444734124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388810a7b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:41.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.235+0000 7f388d600700 1 -- 172.21.15.28:0/444734124 >> 172.21.15.28:0/444734124 conn(0x7f388807d920 msgr2=0x7f38881084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:41.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.235+0000 7f388d600700 1 -- 172.21.15.28:0/444734124 shutdown_connections 2024-04-26T01:27:41.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.235+0000 7f388d600700 1 -- 172.21.15.28:0/444734124 wait complete. 2024-04-26T01:27:41.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.236+0000 7f388d600700 1 Processor -- start 2024-04-26T01:27:41.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.237+0000 7f388d600700 1 -- start start 2024-04-26T01:27:41.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.237+0000 7f388d600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388819e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:41.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.237+0000 7f388d600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f388810e940 con 0x7f388810a390 2024-04-26T01:27:41.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.237+0000 7f3887e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388819e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:41.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.237+0000 7f3887e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388819e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40636/0 (socket says 172.21.15.28:40636) 2024-04-26T01:27:41.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.237+0000 7f3887e00700 1 -- 172.21.15.28:0/2648441491 learned_addr learned my addr 172.21.15.28:0/2648441491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:41.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.238+0000 7f3887e00700 1 -- 172.21.15.28:0/2648441491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388819e7a0 con 0x7f388810a390 2024-04-26T01:27:41.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.238+0000 7f3887e00700 1 --2- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388819e260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f38780117b0 tx=0x7f3878011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:41.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.238+0000 7f3884c00700 1 -- 172.21.15.28:0/2648441491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3878003af0 con 0x7f388810a390 2024-04-26T01:27:41.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.238+0000 7f3884c00700 1 -- 172.21.15.28:0/2648441491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3878020070 con 0x7f388810a390 2024-04-26T01:27:41.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.238+0000 7f3884c00700 1 -- 172.21.15.28:0/2648441491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3878019630 con 0x7f388810a390 2024-04-26T01:27:41.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.238+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388819ea30 con 0x7f388810a390 2024-04-26T01:27:41.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.239+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38881a1720 con 0x7f388810a390 2024-04-26T01:27:41.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.240+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3888069b50 con 0x7f388810a390 2024-04-26T01:27:41.314 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.240+0000 7f3884c00700 1 -- 172.21.15.28:0/2648441491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f387801a9c0 con 0x7f388810a390 2024-04-26T01:27:41.314 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.240+0000 7f3884c00700 1 --2- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f386c039670 0x7f386c03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:41.314 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.241+0000 7f3884c00700 1 -- 172.21.15.28:0/2648441491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f387804dae0 con 0x7f388810a390 2024-04-26T01:27:41.314 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.241+0000 7f3887400700 1 --2- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f386c039670 0x7f386c03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:41.314 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.242+0000 7f3887400700 1 --2- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f386c039670 0x7f386c03bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3874006fd0 tx=0x7f3874006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:41.314 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.246+0000 7f3884c00700 1 -- 172.21.15.28:0/2648441491 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f387800e260 con 0x7f388810a390 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.547+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f38881a18f0 con 0x7f388810a390 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.601+0000 7f3884c00700 1 -- 172.21.15.28:0/2648441491 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f38881a18f0 con 0x7f388810a390 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.604+0000 7f3884c00700 1 -- 172.21.15.28:0/2648441491 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f387801a430 con 0x7f388810a390 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.605+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f386c039670 msgr2=0x7f386c03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.605+0000 7f388d600700 1 --2- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f386c039670 0x7f386c03bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3874006fd0 tx=0x7f3874006e40 comp rx=0 tx=0).stop 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.605+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 msgr2=0x7f388819e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.605+0000 7f388d600700 1 --2- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388819e260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f38780117b0 tx=0x7f3878011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.605+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 shutdown_connections 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.605+0000 7f388d600700 1 --2- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f386c039670 0x7f386c03bb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.605+0000 7f388d600700 1 --2- 172.21.15.28:0/2648441491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388810a390 0x7f388819e260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.605+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 >> 172.21.15.28:0/2648441491 conn(0x7f388807d920 msgr2=0x7f3888107b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.607+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 shutdown_connections 2024-04-26T01:27:41.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:40.607+0000 7f388d600700 1 -- 172.21.15.28:0/2648441491 wait complete. 2024-04-26T01:27:41.448 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:41 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:41.156+0000 7f7446217200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:27:41.448 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:41 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:41.290+0000 7f7446217200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:27:41.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:41 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2648441491' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T01:27:41.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:41 smithi028 ceph-mon[28247]: mgrmap e5: x(active, since 5s) 2024-04-26T01:27:42.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:41 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:41.716+0000 7f7446217200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:27:42.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:41 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:41.833+0000 7f7446217200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:27:42.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:42 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/27408490' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T01:27:42.894 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:27:42.895 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T01:27:42.895 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:27:42.895 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T01:27:42.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T01:27:42.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:27:42.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.886+0000 7f5642800700 1 Processor -- start 2024-04-26T01:27:42.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.886+0000 7f5642800700 1 -- start start 2024-04-26T01:27:42.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.887+0000 7f5642800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c07c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:42.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.887+0000 7f5642800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f563c07c810 con 0x7f563c07de20 2024-04-26T01:27:42.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.887+0000 7f5641400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c07c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.887+0000 7f5641400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c07c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40660/0 (socket says 172.21.15.28:40660) 2024-04-26T01:27:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.887+0000 7f5641400700 1 -- 172.21.15.28:0/864008776 learned_addr learned my addr 172.21.15.28:0/864008776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.888+0000 7f5641400700 1 -- 172.21.15.28:0/864008776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563c07c990 con 0x7f563c07de20 2024-04-26T01:27:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.888+0000 7f5641400700 1 --2- 172.21.15.28:0/864008776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c07c2d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f56300098a0 tx=0x7f5630009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7f4904b3296dda server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.888+0000 7f563be00700 1 -- 172.21.15.28:0/864008776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5630004160 con 0x7f563c07de20 2024-04-26T01:27:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.888+0000 7f563be00700 1 -- 172.21.15.28:0/864008776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5630004300 con 0x7f563c07de20 2024-04-26T01:27:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.888+0000 7f563be00700 1 -- 172.21.15.28:0/864008776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5630011af0 con 0x7f563c07de20 2024-04-26T01:27:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.889+0000 7f5642800700 1 -- 172.21.15.28:0/864008776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 msgr2=0x7f563c07c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.889+0000 7f5642800700 1 --2- 172.21.15.28:0/864008776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c07c2d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f56300098a0 tx=0x7f5630009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.889+0000 7f5642800700 1 -- 172.21.15.28:0/864008776 shutdown_connections 2024-04-26T01:27:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.889+0000 7f5642800700 1 --2- 172.21.15.28:0/864008776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c07c2d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.889+0000 7f5642800700 1 -- 172.21.15.28:0/864008776 >> 172.21.15.28:0/864008776 conn(0x7f563c103740 msgr2=0x7f563c105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.889+0000 7f5642800700 1 -- 172.21.15.28:0/864008776 shutdown_connections 2024-04-26T01:27:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.889+0000 7f5642800700 1 -- 172.21.15.28:0/864008776 wait complete. 2024-04-26T01:27:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.890+0000 7f5642800700 1 Processor -- start 2024-04-26T01:27:42.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.890+0000 7f5642800700 1 -- start start 2024-04-26T01:27:42.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.891+0000 7f5642800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c19e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:42.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.891+0000 7f5642800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f563c10e6f0 con 0x7f563c07de20 2024-04-26T01:27:42.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.891+0000 7f5641400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c19e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:42.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.891+0000 7f5641400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c19e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40664/0 (socket says 172.21.15.28:40664) 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.891+0000 7f5641400700 1 -- 172.21.15.28:0/27408490 learned_addr learned my addr 172.21.15.28:0/27408490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.892+0000 7f5641400700 1 -- 172.21.15.28:0/27408490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563c19e720 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.892+0000 7f5641400700 1 --2- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c19e1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5630011ac0 tx=0x7f5630011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.892+0000 7f563a000700 1 -- 172.21.15.28:0/27408490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5630003b20 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.892+0000 7f563a000700 1 -- 172.21.15.28:0/27408490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f563001a8c0 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.892+0000 7f563a000700 1 -- 172.21.15.28:0/27408490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56300248d0 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.892+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563c19e9b0 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.892+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f563c1a16a0 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.894+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f563c069b50 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.894+0000 7f563a000700 1 -- 172.21.15.28:0/27408490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f5630022020 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.894+0000 7f563a000700 1 --2- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f56240396c0 0x7f562403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.894+0000 7f563a000700 1 -- 172.21.15.28:0/27408490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f563004d4c0 con 0x7f563c07de20 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.894+0000 7f5640a00700 1 -- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f56240396c0 msgr2=0x7f562403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1017286588 2024-04-26T01:27:42.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.895+0000 7f5640a00700 1 --2- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f56240396c0 0x7f562403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:41.900+0000 7f563a000700 1 -- 172.21.15.28:0/27408490 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f563004d220 con 0x7f563c07de20 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.096+0000 7f5640a00700 1 -- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f56240396c0 msgr2=0x7f562403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1017286588 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.096+0000 7f5640a00700 1 --2- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f56240396c0 0x7f562403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.188+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f563c1a1870 con 0x7f563c07de20 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.188+0000 7f563a000700 1 -- 172.21.15.28:0/27408490 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f563c1a1870 con 0x7f563c07de20 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f56240396c0 msgr2=0x7f562403bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 --2- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f56240396c0 0x7f562403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 msgr2=0x7f563c19e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 --2- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c19e1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5630011ac0 tx=0x7f5630011ba0 comp rx=0 tx=0).stop 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 shutdown_connections 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 --2- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f56240396c0 0x7f562403bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 --2- 172.21.15.28:0/27408490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f563c07de20 0x7f563c19e1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 >> 172.21.15.28:0/27408490 conn(0x7f563c103740 msgr2=0x7f563c104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:42.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 shutdown_connections 2024-04-26T01:27:42.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:42.192+0000 7f5642800700 1 -- 172.21.15.28:0/27408490 wait complete. 2024-04-26T01:27:42.906 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-26T01:27:42.906 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-04-26T01:27:43.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:43 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:43.430+0000 7f7446217200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:27:43.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:43 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:43.664+0000 7f7446217200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:27:44.270 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:43 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:43.882+0000 7f7446217200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:27:44.530 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:44 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:44.273+0000 7f7446217200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:27:44.530 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:44 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:44.398+0000 7f7446217200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:27:44.913 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:44 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:44.655+0000 7f7446217200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:27:44.913 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:44 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:44.916+0000 7f7446217200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:27:46.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:45 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:45.722+0000 7f7446217200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:27:46.689 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:46 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:46.289+0000 7f7446217200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:27:46.690 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:46 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:46.418+0000 7f7446217200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:27:46.690 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:46 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:46.693+0000 7f7446217200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:27:47.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:47 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:47.362+0000 7f7446217200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:27:47.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:47 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:47.489+0000 7f7446217200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:27:48.128 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:47 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:47.820+0000 7f7446217200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:27:48.128 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:48.002+0000 7f7446217200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:27:48.128 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:48.132+0000 7f7446217200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:27:48.530 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:48.289+0000 7f7446217200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:27:49.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:48.667+0000 7f7446217200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:27:49.747 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:49.458+0000 7f7446217200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:27:50.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:27:49.752+0000 7f7446217200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:27:50.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: Active manager daemon x restarted 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: Activating manager daemon x 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: mgrmap e6: x(active, starting, since 0.00634977s) 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: Manager daemon x is now available 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:27:50.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:49 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:27:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:50 smithi028 ceph-mon[28247]: Found migration_current of "None". Setting to last migration. 2024-04-26T01:27:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:50 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:27:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:50 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:27:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:50 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:27:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:50 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:27:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:50 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:27:51.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:50 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:27:51.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:50 smithi028 ceph-mon[28247]: mgrmap e7: x(active, since 1.01492s) 2024-04-26T01:27:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:27:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T01:27:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T01:27:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:27:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.455+0000 7f59d3400700 1 Processor -- start 2024-04-26T01:27:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.455+0000 7f59d3400700 1 -- start start 2024-04-26T01:27:51.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.456+0000 7f59d3400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc07c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:51.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.456+0000 7f59d3400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc07c880 con 0x7f59cc07de50 2024-04-26T01:27:51.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.456+0000 7f59d0a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc07c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:51.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.456+0000 7f59d0a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc07c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40668/0 (socket says 172.21.15.28:40668) 2024-04-26T01:27:51.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.456+0000 7f59d0a00700 1 -- 172.21.15.28:0/1176996458 learned_addr learned my addr 172.21.15.28:0/1176996458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:51.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.457+0000 7f59d0a00700 1 -- 172.21.15.28:0/1176996458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc07ca00 con 0x7f59cc07de50 2024-04-26T01:27:51.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.457+0000 7f59d0a00700 1 --2- 172.21.15.28:0/1176996458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc07c2b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009e80 tx=0x7f59c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff762ef7523dc3b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:51.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.457+0000 7f59cb400700 1 -- 172.21.15.28:0/1176996458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c0017070 con 0x7f59cc07de50 2024-04-26T01:27:51.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.457+0000 7f59cb400700 1 -- 172.21.15.28:0/1176996458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f59c000b8c0 con 0x7f59cc07de50 2024-04-26T01:27:51.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.457+0000 7f59cb400700 1 -- 172.21.15.28:0/1176996458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c0004490 con 0x7f59cc07de50 2024-04-26T01:27:51.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.458+0000 7f59d3400700 1 -- 172.21.15.28:0/1176996458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 msgr2=0x7f59cc07c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:51.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.458+0000 7f59d3400700 1 --2- 172.21.15.28:0/1176996458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc07c2b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009e80 tx=0x7f59c000b330 comp rx=0 tx=0).stop 2024-04-26T01:27:51.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.458+0000 7f59d3400700 1 -- 172.21.15.28:0/1176996458 shutdown_connections 2024-04-26T01:27:51.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.458+0000 7f59d3400700 1 --2- 172.21.15.28:0/1176996458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc07c2b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:51.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.458+0000 7f59d3400700 1 -- 172.21.15.28:0/1176996458 >> 172.21.15.28:0/1176996458 conn(0x7f59cc1038e0 msgr2=0x7f59cc105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:51.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.458+0000 7f59d3400700 1 -- 172.21.15.28:0/1176996458 shutdown_connections 2024-04-26T01:27:51.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.458+0000 7f59d3400700 1 -- 172.21.15.28:0/1176996458 wait complete. 2024-04-26T01:27:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.460+0000 7f59d3400700 1 Processor -- start 2024-04-26T01:27:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.460+0000 7f59d3400700 1 -- start start 2024-04-26T01:27:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.460+0000 7f59d3400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc19e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.460+0000 7f59d3400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc10e890 con 0x7f59cc07de50 2024-04-26T01:27:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.461+0000 7f59d0a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc19e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.461+0000 7f59d0a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc19e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40674/0 (socket says 172.21.15.28:40674) 2024-04-26T01:27:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.461+0000 7f59d0a00700 1 -- 172.21.15.28:0/1294260334 learned_addr learned my addr 172.21.15.28:0/1294260334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.461+0000 7f59d0a00700 1 -- 172.21.15.28:0/1294260334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc19e8b0 con 0x7f59cc07de50 2024-04-26T01:27:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.461+0000 7f59d0a00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc19e370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f59c000b7b0 tx=0x7f59c000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.462+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c0020020 con 0x7f59cc07de50 2024-04-26T01:27:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.462+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f59c0003d10 con 0x7f59cc07de50 2024-04-26T01:27:51.424 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.462+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59cc19eb40 con 0x7f59cc07de50 2024-04-26T01:27:51.424 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.462+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c0017040 con 0x7f59cc07de50 2024-04-26T01:27:51.424 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.462+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59cc1a1830 con 0x7f59cc07de50 2024-04-26T01:27:51.424 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.464+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f59c0023020 con 0x7f59cc07de50 2024-04-26T01:27:51.424 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.464+0000 7f59c9600700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:51.424 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.464+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 --> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f59b403c290 con 0x7f59b40396c0 2024-04-26T01:27:51.424 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.464+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f59c000f050 con 0x7f59cc07de50 2024-04-26T01:27:51.424 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.464+0000 7f59cbe00700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 msgr2=0x7f59b403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1017286588 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.464+0000 7f59cbe00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.665+0000 7f59cbe00700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 msgr2=0x7f59b403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1017286588 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:43.665+0000 7f59cbe00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:44.066+0000 7f59cbe00700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 msgr2=0x7f59b403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1017286588 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:44.066+0000 7f59cbe00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:44.867+0000 7f59cbe00700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 msgr2=0x7f59b403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1017286588 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:44.867+0000 7f59cbe00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:46.469+0000 7f59cbe00700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 msgr2=0x7f59b403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1017286588 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:46.469+0000 7f59cbe00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T01:27:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:49.673+0000 7f59cbe00700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 msgr2=0x7f59b403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1017286588 2024-04-26T01:27:51.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:49.673+0000 7f59cbe00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T01:27:51.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:49.761+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f59c001c5a0 con 0x7f59cc07de50 2024-04-26T01:27:51.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:49.761+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 msgr2=0x7f59b403bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:27:51.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:49.761+0000 7f59c9600700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:51.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.764+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f59c004e970 con 0x7f59cc07de50 2024-04-26T01:27:51.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.764+0000 7f59c9600700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f59b403cfb0 0x7f59b403f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:51.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.764+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f59c00020b0 con 0x7f59b403cfb0 2024-04-26T01:27:51.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.767+0000 7f59cbe00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f59b403cfb0 0x7f59b403f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:51.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.769+0000 7f59cbe00700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f59b403cfb0 0x7f59b403f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f59b8003a10 tx=0x7f59b80092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:51.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.771+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mgr.14120 v2:172.21.15.28:6800/2991743583 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f59c00020b0 con 0x7f59b403cfb0 2024-04-26T01:27:51.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.783+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f59cc069b50 con 0x7f59b403cfb0 2024-04-26T01:27:51.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.784+0000 7f59c9600700 1 -- 172.21.15.28:0/1294260334 <== mgr.14120 v2:172.21.15.28:6800/2991743583 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f59cc069b50 con 0x7f59b403cfb0 2024-04-26T01:27:51.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f59b403cfb0 msgr2=0x7f59b403f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:51.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f59b403cfb0 0x7f59b403f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f59b8003a10 tx=0x7f59b80092b0 comp rx=0 tx=0).stop 2024-04-26T01:27:51.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 msgr2=0x7f59cc19e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:51.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc19e370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f59c000b7b0 tx=0x7f59c000be90 comp rx=0 tx=0).stop 2024-04-26T01:27:51.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 shutdown_connections 2024-04-26T01:27:51.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f59b403cfb0 0x7f59b403f3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:51.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59cc07de50 0x7f59cc19e370 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:51.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 --2- 172.21.15.28:0/1294260334 >> [v2:172.21.15.28:6800/1017286588,v1:172.21.15.28:6801/1017286588] conn(0x7f59b40396c0 0x7f59b403bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:51.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 >> 172.21.15.28:0/1294260334 conn(0x7f59cc1038e0 msgr2=0x7f59cc104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:51.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 shutdown_connections 2024-04-26T01:27:51.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:50.785+0000 7f59d3400700 1 -- 172.21.15.28:0/1294260334 wait complete. 2024-04-26T01:27:51.432 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-04-26T01:27:51.432 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-04-26T01:27:52.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:51 smithi028 ceph-mon[28247]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T01:27:52.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:51 smithi028 ceph-mon[28247]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T01:27:52.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:51 smithi028 ceph-mon[28247]: [26/Apr/2024:01:27:50] ENGINE Bus STARTING 2024-04-26T01:27:52.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:51 smithi028 ceph-mon[28247]: [26/Apr/2024:01:27:51] ENGINE Serving on https://172.21.15.28:7150 2024-04-26T01:27:52.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:51 smithi028 ceph-mon[28247]: [26/Apr/2024:01:27:51] ENGINE Serving on http://172.21.15.28:8765 2024-04-26T01:27:52.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:51 smithi028 ceph-mon[28247]: [26/Apr/2024:01:27:51] ENGINE Bus STARTED 2024-04-26T01:27:52.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:51 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:27:53.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:52 smithi028 ceph-mon[28247]: mgrmap e8: x(active, since 2s) 2024-04-26T01:27:53.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:52 smithi028 ceph-mon[28247]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:27:53.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:52 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:27:53.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:52 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:27:53.042 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.067+0000 7f4ba4e00700 1 Processor -- start 2024-04-26T01:27:53.042 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.067+0000 7f4ba4e00700 1 -- start start 2024-04-26T01:27:53.042 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.068+0000 7f4ba4e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba0108720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:53.042 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.068+0000 7f4ba4e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba0108cf0 con 0x7f4ba0108300 2024-04-26T01:27:53.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.068+0000 7f4b9de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba0108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:53.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.068+0000 7f4b9de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba0108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53496/0 (socket says 172.21.15.28:53496) 2024-04-26T01:27:53.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.068+0000 7f4b9de00700 1 -- 172.21.15.28:0/2460901471 learned_addr learned my addr 172.21.15.28:0/2460901471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:53.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.069+0000 7f4b9de00700 1 -- 172.21.15.28:0/2460901471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba0109500 con 0x7f4ba0108300 2024-04-26T01:27:53.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.069+0000 7f4b9de00700 1 --2- 172.21.15.28:0/2460901471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba0108720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4b900098a0 tx=0x7f4b90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6196981045aca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.070+0000 7f4b9d400700 1 -- 172.21.15.28:0/2460901471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b90004160 con 0x7f4ba0108300 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.070+0000 7f4b9d400700 1 -- 172.21.15.28:0/2460901471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4b90004300 con 0x7f4ba0108300 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.070+0000 7f4b9d400700 1 -- 172.21.15.28:0/2460901471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b90011af0 con 0x7f4ba0108300 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.070+0000 7f4ba4e00700 1 -- 172.21.15.28:0/2460901471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 msgr2=0x7f4ba0108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.070+0000 7f4ba4e00700 1 --2- 172.21.15.28:0/2460901471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba0108720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4b900098a0 tx=0x7f4b90009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.070+0000 7f4ba4e00700 1 -- 172.21.15.28:0/2460901471 shutdown_connections 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.070+0000 7f4ba4e00700 1 --2- 172.21.15.28:0/2460901471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba0108720 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.070+0000 7f4ba4e00700 1 -- 172.21.15.28:0/2460901471 >> 172.21.15.28:0/2460901471 conn(0x7f4ba01038e0 msgr2=0x7f4ba0105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.071+0000 7f4ba4e00700 1 -- 172.21.15.28:0/2460901471 shutdown_connections 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.071+0000 7f4ba4e00700 1 -- 172.21.15.28:0/2460901471 wait complete. 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.072+0000 7f4ba4e00700 1 Processor -- start 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.072+0000 7f4ba4e00700 1 -- start start 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.072+0000 7f4ba4e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:53.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.072+0000 7f4ba4e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba010c8b0 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.072+0000 7f4b9de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.072+0000 7f4b9de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53508/0 (socket says 172.21.15.28:53508) 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.072+0000 7f4b9de00700 1 -- 172.21.15.28:0/1114206443 learned_addr learned my addr 172.21.15.28:0/1114206443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.073+0000 7f4b9de00700 1 -- 172.21.15.28:0/1114206443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba019e930 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.073+0000 7f4b9de00700 1 --2- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba019e3f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b90000c00 tx=0x7f4b90011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.073+0000 7f4b96a00700 1 -- 172.21.15.28:0/1114206443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b90003b20 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.073+0000 7f4b96a00700 1 -- 172.21.15.28:0/1114206443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4b9001a8c0 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.073+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba019ebc0 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.073+0000 7f4b96a00700 1 -- 172.21.15.28:0/1114206443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b90024940 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.074+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba01a18b0 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.075+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba00640a0 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.075+0000 7f4b96a00700 1 -- 172.21.15.28:0/1114206443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4b90022020 con 0x7f4ba0108300 2024-04-26T01:27:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.075+0000 7f4b96a00700 1 --2- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4b840396c0 0x7f4b8403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.075+0000 7f4b96a00700 1 -- 172.21.15.28:0/1114206443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4b9004d8a0 con 0x7f4ba0108300 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.076+0000 7f4b97e00700 1 --2- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4b840396c0 0x7f4b8403bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.076+0000 7f4b97e00700 1 --2- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4b840396c0 0x7f4b8403bb80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b88006fd0 tx=0x7f4b88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.082+0000 7f4b96a00700 1 -- 172.21.15.28:0/1114206443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b9000be50 con 0x7f4ba0108300 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.323+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f4ba01049f0 con 0x7f4b840396c0 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.333+0000 7f4b96a00700 1 -- 172.21.15.28:0/1114206443 <== mgr.14120 v2:172.21.15.28:6800/2991743583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4ba01049f0 con 0x7f4b840396c0 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.335+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4b840396c0 msgr2=0x7f4b8403bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.335+0000 7f4ba4e00700 1 --2- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4b840396c0 0x7f4b8403bb80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b88006fd0 tx=0x7f4b88006e40 comp rx=0 tx=0).stop 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.335+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 msgr2=0x7f4ba019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.335+0000 7f4ba4e00700 1 --2- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba019e3f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b90000c00 tx=0x7f4b90011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.336+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 shutdown_connections 2024-04-26T01:27:53.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.336+0000 7f4ba4e00700 1 --2- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4b840396c0 0x7f4b8403bb80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:53.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.336+0000 7f4ba4e00700 1 --2- 172.21.15.28:0/1114206443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba0108300 0x7f4ba019e3f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:53.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.336+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 >> 172.21.15.28:0/1114206443 conn(0x7f4ba01038e0 msgr2=0x7f4ba01042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:53.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.336+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 shutdown_connections 2024-04-26T01:27:53.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:52.336+0000 7f4ba4e00700 1 -- 172.21.15.28:0/1114206443 wait complete. 2024-04-26T01:27:54.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T01:27:54.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.667+0000 7f218ca00700 1 Processor -- start 2024-04-26T01:27:54.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.668+0000 7f218ca00700 1 -- start start 2024-04-26T01:27:54.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.668+0000 7f218ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f2188108720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:54.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.668+0000 7f218ca00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2188108cf0 con 0x7f2188108300 2024-04-26T01:27:54.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.669+0000 7f2185e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f2188108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:54.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.669+0000 7f2185e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f2188108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53520/0 (socket says 172.21.15.28:53520) 2024-04-26T01:27:54.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.669+0000 7f2185e00700 1 -- 172.21.15.28:0/3513826178 learned_addr learned my addr 172.21.15.28:0/3513826178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:54.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.670+0000 7f2185e00700 1 -- 172.21.15.28:0/3513826178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2188109500 con 0x7f2188108300 2024-04-26T01:27:54.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.670+0000 7f2185e00700 1 --2- 172.21.15.28:0/3513826178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f2188108720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21780098a0 tx=0x7f2178009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6618bb942b56a74 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:54.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.670+0000 7f2185400700 1 -- 172.21.15.28:0/3513826178 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2178004160 con 0x7f2188108300 2024-04-26T01:27:54.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.670+0000 7f2185400700 1 -- 172.21.15.28:0/3513826178 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2178004300 con 0x7f2188108300 2024-04-26T01:27:54.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.670+0000 7f2185400700 1 -- 172.21.15.28:0/3513826178 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2178011af0 con 0x7f2188108300 2024-04-26T01:27:54.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.671+0000 7f218ca00700 1 -- 172.21.15.28:0/3513826178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 msgr2=0x7f2188108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:54.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.671+0000 7f218ca00700 1 --2- 172.21.15.28:0/3513826178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f2188108720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21780098a0 tx=0x7f2178009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:54.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.671+0000 7f218ca00700 1 -- 172.21.15.28:0/3513826178 shutdown_connections 2024-04-26T01:27:54.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.671+0000 7f218ca00700 1 --2- 172.21.15.28:0/3513826178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f2188108720 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:54.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.671+0000 7f218ca00700 1 -- 172.21.15.28:0/3513826178 >> 172.21.15.28:0/3513826178 conn(0x7f21881038e0 msgr2=0x7f2188105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:54.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.671+0000 7f218ca00700 1 -- 172.21.15.28:0/3513826178 shutdown_connections 2024-04-26T01:27:54.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.671+0000 7f218ca00700 1 -- 172.21.15.28:0/3513826178 wait complete. 2024-04-26T01:27:54.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.673+0000 7f218ca00700 1 Processor -- start 2024-04-26T01:27:54.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.673+0000 7f218ca00700 1 -- start start 2024-04-26T01:27:54.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.673+0000 7f218ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f218807db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:54.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.673+0000 7f218ca00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f218810c8b0 con 0x7f2188108300 2024-04-26T01:27:54.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.674+0000 7f2185e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f218807db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:54.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.674+0000 7f2185e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f218807db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53534/0 (socket says 172.21.15.28:53534) 2024-04-26T01:27:54.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.674+0000 7f2185e00700 1 -- 172.21.15.28:0/1689119725 learned_addr learned my addr 172.21.15.28:0/1689119725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:54.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.674+0000 7f2185e00700 1 -- 172.21.15.28:0/1689119725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218807e0a0 con 0x7f2188108300 2024-04-26T01:27:54.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.674+0000 7f2185e00700 1 --2- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f218807db60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2178000c00 tx=0x7f2178011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.675+0000 7f217ea00700 1 -- 172.21.15.28:0/1689119725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2178003b20 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.675+0000 7f217ea00700 1 -- 172.21.15.28:0/1689119725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f217801a8c0 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.675+0000 7f217ea00700 1 -- 172.21.15.28:0/1689119725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2178024940 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.675+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218807c290 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.675+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218807c7d0 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.676+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2188069b50 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.676+0000 7f217ea00700 1 -- 172.21.15.28:0/1689119725 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2178022020 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.677+0000 7f217ea00700 1 --2- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f216c0396c0 0x7f216c03bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.677+0000 7f217ea00700 1 -- 172.21.15.28:0/1689119725 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f217804d860 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.677+0000 7f217fe00700 1 --2- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f216c0396c0 0x7f216c03bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.678+0000 7f217fe00700 1 --2- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f216c0396c0 0x7f216c03bb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2170006fd0 tx=0x7f2170006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.683+0000 7f217ea00700 1 -- 172.21.15.28:0/1689119725 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f217800f2e0 con 0x7f2188108300 2024-04-26T01:27:54.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.917+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f2188104af0 con 0x7f216c0396c0 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.919+0000 7f217ea00700 1 -- 172.21.15.28:0/1689119725 <== mgr.14120 v2:172.21.15.28:6800/2991743583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f2188104af0 con 0x7f216c0396c0 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f216c0396c0 msgr2=0x7f216c03bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 --2- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f216c0396c0 0x7f216c03bb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2170006fd0 tx=0x7f2170006e40 comp rx=0 tx=0).stop 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 msgr2=0x7f218807db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 --2- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f218807db60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2178000c00 tx=0x7f2178011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 shutdown_connections 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 --2- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f216c0396c0 0x7f216c03bb80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 --2- 172.21.15.28:0/1689119725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2188108300 0x7f218807db60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 >> 172.21.15.28:0/1689119725 conn(0x7f21881038e0 msgr2=0x7f21881043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 shutdown_connections 2024-04-26T01:27:54.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:53.922+0000 7f218ca00700 1 -- 172.21.15.28:0/1689119725 wait complete. 2024-04-26T01:27:54.590 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-04-26T01:27:54.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:54 smithi028 ceph-mon[28247]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:27:56.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-mon[28247]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:27:56.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-mon[28247]: Generating ssh key... 2024-04-26T01:27:56.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: Generating public/private rsa key pair. 2024-04-26T01:27:56.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: Your identification has been saved in /tmp/tmpjb42hcwd/key. 2024-04-26T01:27:56.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: Your public key has been saved in /tmp/tmpjb42hcwd/key.pub. 2024-04-26T01:27:56.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: The key fingerprint is: 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: SHA256:oO8zdzONkapYuHdoF70+gcRtLNWFbUvGQa1v1vaTuvM ceph-ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: The key's randomart image is: 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: +---[RSA 3072]----+ 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | . *+o | 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | . o * .| 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | .. + + o | 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | . .+ + o | 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | . .S= . ..| 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | o o = *| 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | . o. o * +o| 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | =* = B .. o.| 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: | oooB o.+ o=E.| 2024-04-26T01:27:56.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:27:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: +----[SHA256]-----+ 2024-04-26T01:27:56.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.204+0000 7fd45c200700 1 Processor -- start 2024-04-26T01:27:56.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.204+0000 7fd45c200700 1 -- start start 2024-04-26T01:27:56.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.205+0000 7fd45c200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45410a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:56.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.205+0000 7fd45c200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd45410af90 con 0x7fd45410a5a0 2024-04-26T01:27:56.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.205+0000 7fd459800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45410a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:56.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.205+0000 7fd459800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45410a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53540/0 (socket says 172.21.15.28:53540) 2024-04-26T01:27:56.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.205+0000 7fd459800700 1 -- 172.21.15.28:0/3855699407 learned_addr learned my addr 172.21.15.28:0/3855699407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:56.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.206+0000 7fd459800700 1 -- 172.21.15.28:0/3855699407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45410b750 con 0x7fd45410a5a0 2024-04-26T01:27:56.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.206+0000 7fd459800700 1 --2- 172.21.15.28:0/3855699407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45410a9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd4480098a0 tx=0x7fd448009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcebc5d142812c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:56.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.206+0000 7fd453e00700 1 -- 172.21.15.28:0/3855699407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd448004160 con 0x7fd45410a5a0 2024-04-26T01:27:56.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.206+0000 7fd453e00700 1 -- 172.21.15.28:0/3855699407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd448004300 con 0x7fd45410a5a0 2024-04-26T01:27:56.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.206+0000 7fd453e00700 1 -- 172.21.15.28:0/3855699407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd448011af0 con 0x7fd45410a5a0 2024-04-26T01:27:56.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.207+0000 7fd45c200700 1 -- 172.21.15.28:0/3855699407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 msgr2=0x7fd45410a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.207+0000 7fd45c200700 1 --2- 172.21.15.28:0/3855699407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45410a9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd4480098a0 tx=0x7fd448009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.207+0000 7fd45c200700 1 -- 172.21.15.28:0/3855699407 shutdown_connections 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.207+0000 7fd45c200700 1 --2- 172.21.15.28:0/3855699407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45410a9c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.207+0000 7fd45c200700 1 -- 172.21.15.28:0/3855699407 >> 172.21.15.28:0/3855699407 conn(0x7fd45407dba0 msgr2=0x7fd4541084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.207+0000 7fd45c200700 1 -- 172.21.15.28:0/3855699407 shutdown_connections 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.207+0000 7fd45c200700 1 -- 172.21.15.28:0/3855699407 wait complete. 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.209+0000 7fd45c200700 1 Processor -- start 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.209+0000 7fd45c200700 1 -- start start 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.210+0000 7fd45c200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.210+0000 7fd45c200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd45410eb00 con 0x7fd45410a5a0 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.210+0000 7fd459800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.210+0000 7fd459800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53550/0 (socket says 172.21.15.28:53550) 2024-04-26T01:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.210+0000 7fd459800700 1 -- 172.21.15.28:0/3535122484 learned_addr learned my addr 172.21.15.28:0/3535122484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.210+0000 7fd459800700 1 -- 172.21.15.28:0/3535122484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45419e920 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.211+0000 7fd459800700 1 --2- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45419e3e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd448000c00 tx=0x7fd448011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.211+0000 7fd452000700 1 -- 172.21.15.28:0/3535122484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd448003b20 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.211+0000 7fd452000700 1 -- 172.21.15.28:0/3535122484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd44801a8c0 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.211+0000 7fd452000700 1 -- 172.21.15.28:0/3535122484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd448024940 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.211+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd45419ebb0 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.211+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4541a18a0 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.212+0000 7fd452000700 1 -- 172.21.15.28:0/3535122484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd448022020 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.212+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4540640a0 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.213+0000 7fd452000700 1 --2- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd43c039720 0x7fd43c03bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.213+0000 7fd452000700 1 -- 172.21.15.28:0/3535122484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd44804d850 con 0x7fd45410a5a0 2024-04-26T01:27:56.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.213+0000 7fd458e00700 1 --2- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd43c039720 0x7fd43c03bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:56.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.214+0000 7fd458e00700 1 --2- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd43c039720 0x7fd43c03bbe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd440006fd0 tx=0x7fd440006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:56.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.219+0000 7fd452000700 1 -- 172.21.15.28:0/3535122484 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd44802d3f0 con 0x7fd45410a5a0 2024-04-26T01:27:56.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.449+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fd4541082a0 con 0x7fd43c039720 2024-04-26T01:27:56.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.800+0000 7fd452000700 1 -- 172.21.15.28:0/3535122484 <== mgr.14120 v2:172.21.15.28:6800/2991743583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd4541082a0 con 0x7fd43c039720 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd43c039720 msgr2=0x7fd43c03bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 --2- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd43c039720 0x7fd43c03bbe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd440006fd0 tx=0x7fd440006e40 comp rx=0 tx=0).stop 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 msgr2=0x7fd45419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 --2- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45419e3e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd448000c00 tx=0x7fd448011b70 comp rx=0 tx=0).stop 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 shutdown_connections 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 --2- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd43c039720 0x7fd43c03bbe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 --2- 172.21.15.28:0/3535122484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45410a5a0 0x7fd45419e3e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 >> 172.21.15.28:0/3535122484 conn(0x7fd45407dba0 msgr2=0x7fd454107b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 shutdown_connections 2024-04-26T01:27:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:55.804+0000 7fd45c200700 1 -- 172.21.15.28:0/3535122484 wait complete. 2024-04-26T01:27:57.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:56 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:27:57.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:56 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:27:57.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:57 smithi028 ceph-mon[28247]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDobPXclO1by8lhIOg+eOXbX1tAObEniuejqmVGwUhsvdOKdx5S2v61zSc1/BtBHewGH7wuOIBubobaPrRwiecntmQ7ID/2hSlSesAkgiGh10KO61hXxVlXArb1iW5YGKAN+b4R4+WHdFP6WtTe6CtnvsDYo5CpQ/h0JIRSPT8se3GajVEmk9Q+PUpKofCRvxOkSC8YEW4c0USfC5ROLGUkGQa1FhFwbXsVEk+YPLriExti++Bqy4+STEDonYhyRXVIQQ+AClooBkfp3Voc0SoMwpg54Y819PA43WtHanROHTEsV0fXlO/CtbzTWT90yKxhKpeN+w+qeG28bpif/T762UtauP/tzLeUPthu5ydk+dK5OV/zZAAnAlZOOU0vZ4jFG0IkPYCBX8MGvABELXnV14M+0pZUhCbXzbD2PEOpWvKvL7TDd/hvVWZ5svDm1gDcArrMRdleMCk/40wEUYCs4ec6PodvuijX1D4FveN0AhaVYNaX0o28ut0Mvj6smjk= ceph-ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.135+0000 7f2292e00700 1 Processor -- start 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.135+0000 7f2292e00700 1 -- start start 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.136+0000 7f2292e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c10f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.136+0000 7f2292e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f228c10f910 con 0x7f228c10ef20 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.136+0000 7f2291a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c10f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.136+0000 7f2291a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c10f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53564/0 (socket says 172.21.15.28:53564) 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.136+0000 7f2291a00700 1 -- 172.21.15.28:0/3000261929 learned_addr learned my addr 172.21.15.28:0/3000261929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.137+0000 7f2291a00700 1 -- 172.21.15.28:0/3000261929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228c10fa90 con 0x7f228c10ef20 2024-04-26T01:27:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.137+0000 7f2291a00700 1 --2- 172.21.15.28:0/3000261929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c10f340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f22800098a0 tx=0x7f2280009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2800f4fe0ddc589 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.137+0000 7f228be00700 1 -- 172.21.15.28:0/3000261929 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2280004160 con 0x7f228c10ef20 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.137+0000 7f228be00700 1 -- 172.21.15.28:0/3000261929 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2280004300 con 0x7f228c10ef20 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.138+0000 7f228be00700 1 -- 172.21.15.28:0/3000261929 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2280011af0 con 0x7f228c10ef20 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.138+0000 7f2292e00700 1 -- 172.21.15.28:0/3000261929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 msgr2=0x7f228c10f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.138+0000 7f2292e00700 1 --2- 172.21.15.28:0/3000261929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c10f340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f22800098a0 tx=0x7f2280009bb0 comp rx=0 tx=0).stop 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.138+0000 7f2292e00700 1 -- 172.21.15.28:0/3000261929 shutdown_connections 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.138+0000 7f2292e00700 1 --2- 172.21.15.28:0/3000261929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c10f340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.138+0000 7f2292e00700 1 -- 172.21.15.28:0/3000261929 >> 172.21.15.28:0/3000261929 conn(0x7f228c07d970 msgr2=0x7f228c07fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.138+0000 7f2292e00700 1 -- 172.21.15.28:0/3000261929 shutdown_connections 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.139+0000 7f2292e00700 1 -- 172.21.15.28:0/3000261929 wait complete. 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.139+0000 7f2292e00700 1 Processor -- start 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.140+0000 7f2292e00700 1 -- start start 2024-04-26T01:27:58.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.140+0000 7f2292e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c1a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.140+0000 7f2292e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f228c112aa0 con 0x7f228c10ef20 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.140+0000 7f2291a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c1a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.140+0000 7f2291a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c1a25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53580/0 (socket says 172.21.15.28:53580) 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.140+0000 7f2291a00700 1 -- 172.21.15.28:0/1093317474 learned_addr learned my addr 172.21.15.28:0/1093317474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.141+0000 7f2291a00700 1 -- 172.21.15.28:0/1093317474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228c1a2b20 con 0x7f228c10ef20 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.141+0000 7f2291a00700 1 --2- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c1a25e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2280011ac0 tx=0x7f2280011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.141+0000 7f228a000700 1 -- 172.21.15.28:0/1093317474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2280003970 con 0x7f228c10ef20 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.141+0000 7f228a000700 1 -- 172.21.15.28:0/1093317474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2280003b10 con 0x7f228c10ef20 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.141+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228c1a2e10 con 0x7f228c10ef20 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.141+0000 7f228a000700 1 -- 172.21.15.28:0/1093317474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2280024610 con 0x7f228c10ef20 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.141+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f228c1a5aa0 con 0x7f228c10ef20 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.142+0000 7f228a000700 1 -- 172.21.15.28:0/1093317474 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f228001a9e0 con 0x7f228c10ef20 2024-04-26T01:27:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.142+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f228c069b50 con 0x7f228c10ef20 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.143+0000 7f228a000700 1 --2- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f22740396c0 0x7f227403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.143+0000 7f228a000700 1 -- 172.21.15.28:0/1093317474 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f228004d930 con 0x7f228c10ef20 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.144+0000 7f2291000700 1 --2- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f22740396c0 0x7f227403bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.144+0000 7f2291000700 1 --2- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f22740396c0 0x7f227403bb80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2278006fd0 tx=0x7f2278006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.149+0000 7f228a000700 1 -- 172.21.15.28:0/1093317474 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f228000b3e0 con 0x7f228c10ef20 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.380+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f228c07fb40 con 0x7f22740396c0 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.382+0000 7f228a000700 1 -- 172.21.15.28:0/1093317474 <== mgr.14120 v2:172.21.15.28:6800/2991743583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f228c07fb40 con 0x7f22740396c0 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f22740396c0 msgr2=0x7f227403bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 --2- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f22740396c0 0x7f227403bb80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2278006fd0 tx=0x7f2278006e40 comp rx=0 tx=0).stop 2024-04-26T01:27:58.012 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 msgr2=0x7f228c1a25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 --2- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c1a25e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2280011ac0 tx=0x7f2280011ba0 comp rx=0 tx=0).stop 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 shutdown_connections 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 --2- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f22740396c0 0x7f227403bb80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 --2- 172.21.15.28:0/1093317474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f228c10ef20 0x7f228c1a25e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 >> 172.21.15.28:0/1093317474 conn(0x7f228c07d970 msgr2=0x7f228c07f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.385+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 shutdown_connections 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:57.386+0000 7f2292e00700 1 -- 172.21.15.28:0/1093317474 wait complete. 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T01:27:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T01:27:58.014 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-04-26T01:27:59.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:27:59 smithi028 ceph-mon[28247]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:28:00.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:00 smithi028 ceph-mon[28247]: Deploying cephadm binary to smithi028 2024-04-26T01:28:05.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-04-26T01:28:05.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.679+0000 7fd351e00700 1 Processor -- start 2024-04-26T01:28:05.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.679+0000 7fd351e00700 1 -- start start 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.680+0000 7fd351e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c10b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.680+0000 7fd351e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c07bde0 con 0x7fd34c108e70 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.680+0000 7fd34aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c10b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.680+0000 7fd34aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c10b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53596/0 (socket says 172.21.15.28:53596) 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.680+0000 7fd34aa00700 1 -- 172.21.15.28:0/3718828163 learned_addr learned my addr 172.21.15.28:0/3718828163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.681+0000 7fd34aa00700 1 -- 172.21.15.28:0/3718828163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c10b7a0 con 0x7fd34c108e70 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.681+0000 7fd34aa00700 1 --2- 172.21.15.28:0/3718828163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c10b260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd33c0098a0 tx=0x7fd33c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0f25bb32b35c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.681+0000 7fd349600700 1 -- 172.21.15.28:0/3718828163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd33c004160 con 0x7fd34c108e70 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.681+0000 7fd349600700 1 -- 172.21.15.28:0/3718828163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd33c004300 con 0x7fd34c108e70 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.681+0000 7fd349600700 1 -- 172.21.15.28:0/3718828163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd33c011af0 con 0x7fd34c108e70 2024-04-26T01:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.682+0000 7fd351e00700 1 -- 172.21.15.28:0/3718828163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 msgr2=0x7fd34c10b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.682+0000 7fd351e00700 1 --2- 172.21.15.28:0/3718828163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c10b260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd33c0098a0 tx=0x7fd33c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.682+0000 7fd351e00700 1 -- 172.21.15.28:0/3718828163 shutdown_connections 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.682+0000 7fd351e00700 1 --2- 172.21.15.28:0/3718828163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c10b260 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.682+0000 7fd351e00700 1 -- 172.21.15.28:0/3718828163 >> 172.21.15.28:0/3718828163 conn(0x7fd34c102820 msgr2=0x7fd34c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.683+0000 7fd351e00700 1 -- 172.21.15.28:0/3718828163 shutdown_connections 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.683+0000 7fd351e00700 1 -- 172.21.15.28:0/3718828163 wait complete. 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.684+0000 7fd351e00700 1 Processor -- start 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.685+0000 7fd351e00700 1 -- start start 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.685+0000 7fd351e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.685+0000 7fd351e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c10c6b0 con 0x7fd34c108e70 2024-04-26T01:28:05.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.685+0000 7fd34aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.685+0000 7fd34aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c19e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53600/0 (socket says 172.21.15.28:53600) 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.685+0000 7fd34aa00700 1 -- 172.21.15.28:0/2677345107 learned_addr learned my addr 172.21.15.28:0/2677345107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.686+0000 7fd34aa00700 1 -- 172.21.15.28:0/2677345107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c19e970 con 0x7fd34c108e70 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.686+0000 7fd34aa00700 1 --2- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c19e430 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd33c011ac0 tx=0x7fd33c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.686+0000 7fd350a00700 1 -- 172.21.15.28:0/2677345107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd33c003b20 con 0x7fd34c108e70 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.686+0000 7fd350a00700 1 -- 172.21.15.28:0/2677345107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd33c01a8c0 con 0x7fd34c108e70 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.686+0000 7fd350a00700 1 -- 172.21.15.28:0/2677345107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd33c0248d0 con 0x7fd34c108e70 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.686+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd34c19ec00 con 0x7fd34c108e70 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.687+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd34c1a18f0 con 0x7fd34c108e70 2024-04-26T01:28:05.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.688+0000 7fd350a00700 1 -- 172.21.15.28:0/2677345107 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd33c022020 con 0x7fd34c108e70 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.688+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd34c0640a0 con 0x7fd34c108e70 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.688+0000 7fd350a00700 1 --2- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd3300396c0 0x7fd33003bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.688+0000 7fd350a00700 1 -- 172.21.15.28:0/2677345107 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd33c04d810 con 0x7fd34c108e70 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.689+0000 7fd34a000700 1 --2- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd3300396c0 0x7fd33003bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.689+0000 7fd34a000700 1 --2- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd3300396c0 0x7fd33003bb80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd338006fd0 tx=0x7fd338006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.694+0000 7fd350a00700 1 -- 172.21.15.28:0/2677345107 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd33c01ab20 con 0x7fd34c108e70 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:27:58.929+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7fd34c103950 con 0x7fd3300396c0 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.103+0000 7fd350a00700 1 -- 172.21.15.28:0/2677345107 <== mgr.14120 v2:172.21.15.28:6800/2991743583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fd34c103950 con 0x7fd3300396c0 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd3300396c0 msgr2=0x7fd33003bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 --2- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd3300396c0 0x7fd33003bb80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd338006fd0 tx=0x7fd338006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 msgr2=0x7fd34c19e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 --2- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c19e430 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd33c011ac0 tx=0x7fd33c011ba0 comp rx=0 tx=0).stop 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 shutdown_connections 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 --2- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fd3300396c0 0x7fd33003bb80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 --2- 172.21.15.28:0/2677345107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34c108e70 0x7fd34c19e430 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 >> 172.21.15.28:0/2677345107 conn(0x7fd34c102820 msgr2=0x7fd34c103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 shutdown_connections 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:05.107+0000 7fd351e00700 1 -- 172.21.15.28:0/2677345107 wait complete. 2024-04-26T01:28:05.750 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-04-26T01:28:06.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:06 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:28:06.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:06 smithi028 ceph-mon[28247]: Added host smithi028 2024-04-26T01:28:06.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:06 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.427+0000 7fe7dc000700 1 Processor -- start 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.427+0000 7fe7dc000700 1 -- start start 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.428+0000 7fe7dc000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d4109010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.428+0000 7fe7dc000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d407bea0 con 0x7fe7d4106be0 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.428+0000 7fe7d9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d4109010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.429+0000 7fe7d9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d4109010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47808/0 (socket says 172.21.15.28:47808) 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.429+0000 7fe7d9600700 1 -- 172.21.15.28:0/2045593401 learned_addr learned my addr 172.21.15.28:0/2045593401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.429+0000 7fe7d9600700 1 -- 172.21.15.28:0/2045593401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d4109550 con 0x7fe7d4106be0 2024-04-26T01:28:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.429+0000 7fe7d9600700 1 --2- 172.21.15.28:0/2045593401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d4109010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8009e80 tx=0x7fe7c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae10b10b5b6ecf90 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.430+0000 7fe7d3e00700 1 -- 172.21.15.28:0/2045593401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7c8017070 con 0x7fe7d4106be0 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.430+0000 7fe7d3e00700 1 -- 172.21.15.28:0/2045593401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe7c800b8c0 con 0x7fe7d4106be0 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.430+0000 7fe7d3e00700 1 -- 172.21.15.28:0/2045593401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7c8004490 con 0x7fe7d4106be0 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.430+0000 7fe7dc000700 1 -- 172.21.15.28:0/2045593401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 msgr2=0x7fe7d4109010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.430+0000 7fe7dc000700 1 --2- 172.21.15.28:0/2045593401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d4109010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8009e80 tx=0x7fe7c800b330 comp rx=0 tx=0).stop 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.431+0000 7fe7dc000700 1 -- 172.21.15.28:0/2045593401 shutdown_connections 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.431+0000 7fe7dc000700 1 --2- 172.21.15.28:0/2045593401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d4109010 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.431+0000 7fe7dc000700 1 -- 172.21.15.28:0/2045593401 >> 172.21.15.28:0/2045593401 conn(0x7fe7d4102820 msgr2=0x7fe7d4104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.431+0000 7fe7dc000700 1 -- 172.21.15.28:0/2045593401 shutdown_connections 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.431+0000 7fe7dc000700 1 -- 172.21.15.28:0/2045593401 wait complete. 2024-04-26T01:28:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.433+0000 7fe7dc000700 1 Processor -- start 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.433+0000 7fe7dc000700 1 -- start start 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.433+0000 7fe7dc000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d41a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.433+0000 7fe7dc000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d410a580 con 0x7fe7d4106be0 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.433+0000 7fe7d9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d41a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.434+0000 7fe7d9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d41a27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47810/0 (socket says 172.21.15.28:47810) 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.434+0000 7fe7d9600700 1 -- 172.21.15.28:0/3287755085 learned_addr learned my addr 172.21.15.28:0/3287755085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.434+0000 7fe7d9600700 1 -- 172.21.15.28:0/3287755085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d41a2ce0 con 0x7fe7d4106be0 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.434+0000 7fe7d9600700 1 --2- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d41a27a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800b7b0 tx=0x7fe7c800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.435+0000 7fe7d2000700 1 -- 172.21.15.28:0/3287755085 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7c8020020 con 0x7fe7d4106be0 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.435+0000 7fe7d2000700 1 -- 172.21.15.28:0/3287755085 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe7c8003d10 con 0x7fe7d4106be0 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.435+0000 7fe7d2000700 1 -- 172.21.15.28:0/3287755085 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7c8017040 con 0x7fe7d4106be0 2024-04-26T01:28:07.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.435+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d41a2f70 con 0x7fe7d4106be0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.435+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d41a5c60 con 0x7fe7d4106be0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.436+0000 7fe7d2000700 1 -- 172.21.15.28:0/3287755085 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe7c8023020 con 0x7fe7d4106be0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.436+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d40640a0 con 0x7fe7d4106be0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.436+0000 7fe7d2000700 1 --2- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fe7bc0396c0 0x7fe7bc03bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.436+0000 7fe7d2000700 1 -- 172.21.15.28:0/3287755085 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe7c800f050 con 0x7fe7d4106be0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.437+0000 7fe7d8c00700 1 --2- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fe7bc0396c0 0x7fe7bc03bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.438+0000 7fe7d8c00700 1 --2- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fe7bc0396c0 0x7fe7bc03bb80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0006fd0 tx=0x7fe7c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.443+0000 7fe7d2000700 1 -- 172.21.15.28:0/3287755085 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7c802a020 con 0x7fe7d4106be0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.699+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe7d402e330 con 0x7fe7bc0396c0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.710+0000 7fe7d2000700 1 -- 172.21.15.28:0/3287755085 <== mgr.14120 v2:172.21.15.28:6800/2991743583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe7d402e330 con 0x7fe7bc0396c0 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.713+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fe7bc0396c0 msgr2=0x7fe7bc03bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:07.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.713+0000 7fe7dc000700 1 --2- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fe7bc0396c0 0x7fe7bc03bb80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0006fd0 tx=0x7fe7c0006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.713+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 msgr2=0x7fe7d41a27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.713+0000 7fe7dc000700 1 --2- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d41a27a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800b7b0 tx=0x7fe7c800be90 comp rx=0 tx=0).stop 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.713+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 shutdown_connections 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.713+0000 7fe7dc000700 1 --2- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fe7bc0396c0 0x7fe7bc03bb80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.713+0000 7fe7dc000700 1 --2- 172.21.15.28:0/3287755085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4106be0 0x7fe7d41a27a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.713+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 >> 172.21.15.28:0/3287755085 conn(0x7fe7d4102820 msgr2=0x7fe7d4104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.714+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 shutdown_connections 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:06.714+0000 7fe7dc000700 1 -- 172.21.15.28:0/3287755085 wait complete. 2024-04-26T01:28:07.549 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-04-26T01:28:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:07 smithi028 ceph-mon[28247]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:28:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:07 smithi028 ceph-mon[28247]: Saving service mon spec with placement count:5 2024-04-26T01:28:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:07 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:28:09.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T01:28:09.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.619+0000 7f58bbe00700 1 Processor -- start 2024-04-26T01:28:09.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.619+0000 7f58bbe00700 1 -- start start 2024-04-26T01:28:09.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.619+0000 7f58bbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc10a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.619+0000 7f58bbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc10ad80 con 0x7f58bc10a390 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.620+0000 7f58baa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc10a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.620+0000 7f58baa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc10a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47822/0 (socket says 172.21.15.28:47822) 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.620+0000 7f58baa00700 1 -- 172.21.15.28:0/53038203 learned_addr learned my addr 172.21.15.28:0/53038203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.620+0000 7f58baa00700 1 -- 172.21.15.28:0/53038203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58bc10b590 con 0x7f58bc10a390 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.621+0000 7f58baa00700 1 --2- 172.21.15.28:0/53038203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc10a7b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0098a0 tx=0x7f58ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9adccffe4245c576 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.621+0000 7f58ba000700 1 -- 172.21.15.28:0/53038203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58ac004160 con 0x7f58bc10a390 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.621+0000 7f58ba000700 1 -- 172.21.15.28:0/53038203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f58ac004300 con 0x7f58bc10a390 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.621+0000 7f58ba000700 1 -- 172.21.15.28:0/53038203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58ac011af0 con 0x7f58bc10a390 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.622+0000 7f58bbe00700 1 -- 172.21.15.28:0/53038203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 msgr2=0x7f58bc10a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.622+0000 7f58bbe00700 1 --2- 172.21.15.28:0/53038203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc10a7b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0098a0 tx=0x7f58ac009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.622+0000 7f58bbe00700 1 -- 172.21.15.28:0/53038203 shutdown_connections 2024-04-26T01:28:09.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.622+0000 7f58bbe00700 1 --2- 172.21.15.28:0/53038203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc10a7b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.622+0000 7f58bbe00700 1 -- 172.21.15.28:0/53038203 >> 172.21.15.28:0/53038203 conn(0x7f58bc07d7c0 msgr2=0x7f58bc07dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.622+0000 7f58bbe00700 1 -- 172.21.15.28:0/53038203 shutdown_connections 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.622+0000 7f58bbe00700 1 -- 172.21.15.28:0/53038203 wait complete. 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.623+0000 7f58bbe00700 1 Processor -- start 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.623+0000 7f58bbe00700 1 -- start start 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.624+0000 7f58bbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc19e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.624+0000 7f58bbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc10e940 con 0x7f58bc10a390 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.624+0000 7f58baa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc19e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.624+0000 7f58baa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc19e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47826/0 (socket says 172.21.15.28:47826) 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.624+0000 7f58baa00700 1 -- 172.21.15.28:0/1430251349 learned_addr learned my addr 172.21.15.28:0/1430251349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.624+0000 7f58baa00700 1 -- 172.21.15.28:0/1430251349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58bc19e820 con 0x7f58bc10a390 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.625+0000 7f58baa00700 1 --2- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc19e2e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0117b0 tx=0x7f58ac011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:09.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.625+0000 7f58b3e00700 1 -- 172.21.15.28:0/1430251349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58ac003af0 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.625+0000 7f58b3e00700 1 -- 172.21.15.28:0/1430251349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f58ac020070 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.625+0000 7f58b3e00700 1 -- 172.21.15.28:0/1430251349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58ac019630 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.625+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58bc19eab0 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.625+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58bc1a17a0 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.626+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58bc069b50 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.626+0000 7f58b3e00700 1 -- 172.21.15.28:0/1430251349 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f58ac01a9c0 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.627+0000 7f58b3e00700 1 --2- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f58a00396c0 0x7f58a003bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.627+0000 7f58b3e00700 1 -- 172.21.15.28:0/1430251349 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f58ac04dbb0 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.627+0000 7f58b2000700 1 --2- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f58a00396c0 0x7f58a003bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.628+0000 7f58b2000700 1 --2- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f58a00396c0 0x7f58a003bb80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f58a4006fd0 tx=0x7f58a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.633+0000 7f58b3e00700 1 -- 172.21.15.28:0/1430251349 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58ac00e260 con 0x7f58bc10a390 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.880+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f58bc10cec0 con 0x7f58a00396c0 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.887+0000 7f58b3e00700 1 -- 172.21.15.28:0/1430251349 <== mgr.14120 v2:172.21.15.28:6800/2991743583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f58bc10cec0 con 0x7f58a00396c0 2024-04-26T01:28:09.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.890+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f58a00396c0 msgr2=0x7f58a003bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.890+0000 7f58bbe00700 1 --2- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f58a00396c0 0x7f58a003bb80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f58a4006fd0 tx=0x7f58a4006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.890+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 msgr2=0x7f58bc19e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.890+0000 7f58bbe00700 1 --2- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc19e2e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0117b0 tx=0x7f58ac011b70 comp rx=0 tx=0).stop 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.890+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 shutdown_connections 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.890+0000 7f58bbe00700 1 --2- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f58a00396c0 0x7f58a003bb80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.890+0000 7f58bbe00700 1 --2- 172.21.15.28:0/1430251349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58bc10a390 0x7f58bc19e2e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.890+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 >> 172.21.15.28:0/1430251349 conn(0x7f58bc07d7c0 msgr2=0x7f58bc10c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.891+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 shutdown_connections 2024-04-26T01:28:09.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:08.891+0000 7f58bbe00700 1 -- 172.21.15.28:0/1430251349 wait complete. 2024-04-26T01:28:10.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:09 smithi028 ceph-mon[28247]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:28:10.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:09 smithi028 ceph-mon[28247]: Saving service mgr spec with placement count:2 2024-04-26T01:28:10.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:09 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:28:10.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:09 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:28:11.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:10 smithi028 ceph-mon[28247]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:11.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:10 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:28:11.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:10 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2184542528' entity='client.admin' 2024-04-26T01:28:11.372 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.404+0000 7f6bd9000700 1 Processor -- start 2024-04-26T01:28:11.372 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.404+0000 7f6bd9000700 1 -- start start 2024-04-26T01:28:11.372 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.404+0000 7f6bd9000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd4108720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:11.373 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.404+0000 7f6bd9000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd4108cf0 con 0x7f6bd4108300 2024-04-26T01:28:11.373 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.405+0000 7f6bd1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd4108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:11.373 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.405+0000 7f6bd1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd4108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60572/0 (socket says 172.21.15.28:60572) 2024-04-26T01:28:11.373 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.405+0000 7f6bd1e00700 1 -- 172.21.15.28:0/4198860880 learned_addr learned my addr 172.21.15.28:0/4198860880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:11.373 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.406+0000 7f6bd1e00700 1 -- 172.21.15.28:0/4198860880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd4109500 con 0x7f6bd4108300 2024-04-26T01:28:11.373 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.406+0000 7f6bd1e00700 1 --2- 172.21.15.28:0/4198860880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd4108720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40098a0 tx=0x7f6bc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b58b785ed905517 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.406+0000 7f6bd0a00700 1 -- 172.21.15.28:0/4198860880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bc4004160 con 0x7f6bd4108300 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.406+0000 7f6bd0a00700 1 -- 172.21.15.28:0/4198860880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bc4004300 con 0x7f6bd4108300 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.406+0000 7f6bd0a00700 1 -- 172.21.15.28:0/4198860880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bc4011a20 con 0x7f6bd4108300 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.407+0000 7f6bd9000700 1 -- 172.21.15.28:0/4198860880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 msgr2=0x7f6bd4108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.407+0000 7f6bd9000700 1 --2- 172.21.15.28:0/4198860880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd4108720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40098a0 tx=0x7f6bc4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.407+0000 7f6bd9000700 1 -- 172.21.15.28:0/4198860880 shutdown_connections 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.407+0000 7f6bd9000700 1 --2- 172.21.15.28:0/4198860880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd4108720 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.407+0000 7f6bd9000700 1 -- 172.21.15.28:0/4198860880 >> 172.21.15.28:0/4198860880 conn(0x7f6bd41038e0 msgr2=0x7f6bd4105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.407+0000 7f6bd9000700 1 -- 172.21.15.28:0/4198860880 shutdown_connections 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.407+0000 7f6bd9000700 1 -- 172.21.15.28:0/4198860880 wait complete. 2024-04-26T01:28:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.408+0000 7f6bd9000700 1 Processor -- start 2024-04-26T01:28:11.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.409+0000 7f6bd9000700 1 -- start start 2024-04-26T01:28:11.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.409+0000 7f6bd9000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:11.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.409+0000 7f6bd9000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd410c8b0 con 0x7f6bd4108300 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.409+0000 7f6bd1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.409+0000 7f6bd1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60584/0 (socket says 172.21.15.28:60584) 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.409+0000 7f6bd1e00700 1 -- 172.21.15.28:0/2184542528 learned_addr learned my addr 172.21.15.28:0/2184542528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.410+0000 7f6bd1e00700 1 -- 172.21.15.28:0/2184542528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd407e0a0 con 0x7f6bd4108300 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.410+0000 7f6bd1e00700 1 --2- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd407db60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009520 tx=0x7f6bc4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.410+0000 7f6bcaa00700 1 -- 172.21.15.28:0/2184542528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bc40039e0 con 0x7f6bd4108300 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.410+0000 7f6bcaa00700 1 -- 172.21.15.28:0/2184542528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bc401b8f0 con 0x7f6bd4108300 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.410+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd407c290 con 0x7f6bd4108300 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.410+0000 7f6bcaa00700 1 -- 172.21.15.28:0/2184542528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bc4019770 con 0x7f6bd4108300 2024-04-26T01:28:11.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.410+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd407c7d0 con 0x7f6bd4108300 2024-04-26T01:28:11.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.411+0000 7f6bcaa00700 1 -- 172.21.15.28:0/2184542528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6bc4019a70 con 0x7f6bd4108300 2024-04-26T01:28:11.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.412+0000 7f6bcaa00700 1 --2- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f6bb8039670 0x7f6bb803bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:11.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.412+0000 7f6bcaa00700 1 -- 172.21.15.28:0/2184542528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6bc404df90 con 0x7f6bd4108300 2024-04-26T01:28:11.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.412+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd40640a0 con 0x7f6bd4108300 2024-04-26T01:28:11.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.412+0000 7f6bd1400700 1 --2- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f6bb8039670 0x7f6bb803bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:11.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.413+0000 7f6bd1400700 1 --2- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f6bb8039670 0x7f6bb803bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0006fd0 tx=0x7f6bc0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:11.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.418+0000 7f6bcaa00700 1 -- 172.21.15.28:0/2184542528 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bc401f030 con 0x7f6bd4108300 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.660+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f6bd407ca50 con 0x7f6bd4108300 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.663+0000 7f6bcaa00700 1 -- 172.21.15.28:0/2184542528 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f6bc40172b0 con 0x7f6bd4108300 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f6bb8039670 msgr2=0x7f6bb803bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 --2- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f6bb8039670 0x7f6bb803bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0006fd0 tx=0x7f6bc0006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 msgr2=0x7f6bd407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 --2- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd407db60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009520 tx=0x7f6bc4011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 shutdown_connections 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 --2- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f6bb8039670 0x7f6bb803bb30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 --2- 172.21.15.28:0/2184542528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd4108300 0x7f6bd407db60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 >> 172.21.15.28:0/2184542528 conn(0x7f6bd41038e0 msgr2=0x7f6bd4104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 shutdown_connections 2024-04-26T01:28:11.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:10.666+0000 7f6bd9000700 1 -- 172.21.15.28:0/2184542528 wait complete. 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.085+0000 7f4a65600700 1 Processor -- start 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.086+0000 7f4a65600700 1 -- start start 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.086+0000 7f4a65600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601074a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.086+0000 7f4a65600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a60107a70 con 0x7f4a60107080 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.087+0000 7f4a5fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.087+0000 7f4a5fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60596/0 (socket says 172.21.15.28:60596) 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.087+0000 7f4a5fe00700 1 -- 172.21.15.28:0/277229266 learned_addr learned my addr 172.21.15.28:0/277229266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.087+0000 7f4a5fe00700 1 -- 172.21.15.28:0/277229266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a60108280 con 0x7f4a60107080 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.088+0000 7f4a5fe00700 1 --2- 172.21.15.28:0/277229266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601074a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4a500098a0 tx=0x7f4a50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af95e7d87703828 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:13.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.088+0000 7f4a5f400700 1 -- 172.21.15.28:0/277229266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a50004160 con 0x7f4a60107080 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.088+0000 7f4a5f400700 1 -- 172.21.15.28:0/277229266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a50004300 con 0x7f4a60107080 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.088+0000 7f4a5f400700 1 -- 172.21.15.28:0/277229266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a50011a20 con 0x7f4a60107080 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.089+0000 7f4a65600700 1 -- 172.21.15.28:0/277229266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 msgr2=0x7f4a601074a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.089+0000 7f4a65600700 1 --2- 172.21.15.28:0/277229266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601074a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4a500098a0 tx=0x7f4a50009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.089+0000 7f4a65600700 1 -- 172.21.15.28:0/277229266 shutdown_connections 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.089+0000 7f4a65600700 1 --2- 172.21.15.28:0/277229266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601074a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.089+0000 7f4a65600700 1 -- 172.21.15.28:0/277229266 >> 172.21.15.28:0/277229266 conn(0x7f4a601026c0 msgr2=0x7f4a60104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.089+0000 7f4a65600700 1 -- 172.21.15.28:0/277229266 shutdown_connections 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.089+0000 7f4a65600700 1 -- 172.21.15.28:0/277229266 wait complete. 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.090+0000 7f4a65600700 1 Processor -- start 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.091+0000 7f4a65600700 1 -- start start 2024-04-26T01:28:13.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.091+0000 7f4a65600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601921f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.091+0000 7f4a65600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6010b630 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.091+0000 7f4a5fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601921f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.092+0000 7f4a5fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601921f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60606/0 (socket says 172.21.15.28:60606) 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.092+0000 7f4a5fe00700 1 -- 172.21.15.28:0/1269177031 learned_addr learned my addr 172.21.15.28:0/1269177031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.092+0000 7f4a5fe00700 1 -- 172.21.15.28:0/1269177031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a60192730 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.092+0000 7f4a5fe00700 1 --2- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601921f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009520 tx=0x7f4a50011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.093+0000 7f4a5d600700 1 -- 172.21.15.28:0/1269177031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a500039e0 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.093+0000 7f4a5d600700 1 -- 172.21.15.28:0/1269177031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a5001b8f0 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.093+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a60190920 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.093+0000 7f4a5d600700 1 -- 172.21.15.28:0/1269177031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a50019770 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.093+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a60190e60 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.094+0000 7f4a5d600700 1 -- 172.21.15.28:0/1269177031 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4a50019910 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.094+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3c005360 con 0x7f4a60107080 2024-04-26T01:28:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.095+0000 7f4a5d600700 1 --2- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4a440396c0 0x7f4a4403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.095+0000 7f4a5d600700 1 -- 172.21.15.28:0/1269177031 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4a5004dfc0 con 0x7f4a60107080 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.095+0000 7f4a57e00700 1 --2- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4a440396c0 0x7f4a4403bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.096+0000 7f4a57e00700 1 --2- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4a440396c0 0x7f4a4403bb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a48006fd0 tx=0x7f4a48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.101+0000 7f4a5d600700 1 -- 172.21.15.28:0/1269177031 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a5001f030 con 0x7f4a60107080 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.335+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f4a3c005650 con 0x7f4a60107080 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.340+0000 7f4a5d600700 1 -- 172.21.15.28:0/1269177031 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f4a50016df0 con 0x7f4a60107080 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.343+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4a440396c0 msgr2=0x7f4a4403bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.343+0000 7f4a65600700 1 --2- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4a440396c0 0x7f4a4403bb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a48006fd0 tx=0x7f4a48006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.343+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 msgr2=0x7f4a601921f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.343+0000 7f4a65600700 1 --2- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601921f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009520 tx=0x7f4a50011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.344+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 shutdown_connections 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.344+0000 7f4a65600700 1 --2- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f4a440396c0 0x7f4a4403bb80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:13.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.344+0000 7f4a65600700 1 --2- 172.21.15.28:0/1269177031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60107080 0x7f4a601921f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:13.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.344+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 >> 172.21.15.28:0/1269177031 conn(0x7f4a601026c0 msgr2=0x7f4a60103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:13.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.344+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 shutdown_connections 2024-04-26T01:28:13.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:12.344+0000 7f4a65600700 1 -- 172.21.15.28:0/1269177031 wait complete. 2024-04-26T01:28:13.333 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-04-26T01:28:13.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:13 smithi028 ceph-mon[28247]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:13.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:13 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1269177031' entity='client.admin' 2024-04-26T01:28:13.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:13 smithi028 ceph-mon[28247]: from='mgr.14120 172.21.15.28:0/1290246631' entity='mgr.x' 2024-04-26T01:28:14.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:14 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3491814826' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T01:28:14.756 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:14 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: ignoring --setuser ceph since I am not root 2024-04-26T01:28:14.756 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:14 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: ignoring --setgroup ceph since I am not root 2024-04-26T01:28:14.756 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:14 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:14.655+0000 7f2096802200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:28:15.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:14 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:14.909+0000 7f2096802200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.908+0000 7f88b6800700 1 Processor -- start 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.908+0000 7f88b6800700 1 -- start start 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.909+0000 7f88b6800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b0108790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.909+0000 7f88b6800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88b0108d60 con 0x7f88b0108370 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.909+0000 7f88af400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b0108790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.909+0000 7f88af400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b0108790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60610/0 (socket says 172.21.15.28:60610) 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.909+0000 7f88af400700 1 -- 172.21.15.28:0/3533544323 learned_addr learned my addr 172.21.15.28:0/3533544323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.910+0000 7f88af400700 1 -- 172.21.15.28:0/3533544323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b0109520 con 0x7f88b0108370 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.910+0000 7f88af400700 1 --2- 172.21.15.28:0/3533544323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b0108790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f88a00098a0 tx=0x7f88a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04b6352c9b61a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:15.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.911+0000 7f88ae000700 1 -- 172.21.15.28:0/3533544323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88a0004160 con 0x7f88b0108370 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.911+0000 7f88ae000700 1 -- 172.21.15.28:0/3533544323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a0004300 con 0x7f88b0108370 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.911+0000 7f88ae000700 1 -- 172.21.15.28:0/3533544323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88a0011a20 con 0x7f88b0108370 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.912+0000 7f88b6800700 1 -- 172.21.15.28:0/3533544323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 msgr2=0x7f88b0108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.912+0000 7f88b6800700 1 --2- 172.21.15.28:0/3533544323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b0108790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f88a00098a0 tx=0x7f88a0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.912+0000 7f88b6800700 1 -- 172.21.15.28:0/3533544323 shutdown_connections 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.912+0000 7f88b6800700 1 --2- 172.21.15.28:0/3533544323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b0108790 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.912+0000 7f88b6800700 1 -- 172.21.15.28:0/3533544323 >> 172.21.15.28:0/3533544323 conn(0x7f88b01038f0 msgr2=0x7f88b0105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.912+0000 7f88b6800700 1 -- 172.21.15.28:0/3533544323 shutdown_connections 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.912+0000 7f88b6800700 1 -- 172.21.15.28:0/3533544323 wait complete. 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.913+0000 7f88b6800700 1 Processor -- start 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.913+0000 7f88b6800700 1 -- start start 2024-04-26T01:28:15.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.914+0000 7f88b6800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b007db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.914+0000 7f88b6800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88b010c8d0 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.914+0000 7f88af400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b007db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.914+0000 7f88af400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b007db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60616/0 (socket says 172.21.15.28:60616) 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.914+0000 7f88af400700 1 -- 172.21.15.28:0/3491814826 learned_addr learned my addr 172.21.15.28:0/3491814826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.915+0000 7f88af400700 1 -- 172.21.15.28:0/3491814826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b007e0a0 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.915+0000 7f88af400700 1 --2- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b007db60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009520 tx=0x7f88a0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.915+0000 7f88a7e00700 1 -- 172.21.15.28:0/3491814826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88a00039e0 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.915+0000 7f88a7e00700 1 -- 172.21.15.28:0/3491814826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a001b8f0 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.915+0000 7f88a7e00700 1 -- 172.21.15.28:0/3491814826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88a0019770 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.915+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b007c290 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.916+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b007c7d0 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.917+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b00640a0 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.917+0000 7f88a7e00700 1 -- 172.21.15.28:0/3491814826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f88a001b440 con 0x7f88b0108370 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.918+0000 7f88a7e00700 1 --2- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f88940396c0 0x7f889403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:15.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.918+0000 7f88a7e00700 1 -- 172.21.15.28:0/3491814826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f88a004df90 con 0x7f88b0108370 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.918+0000 7f88aea00700 1 --2- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f88940396c0 0x7f889403bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.919+0000 7f88aea00700 1 --2- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f88940396c0 0x7f889403bb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f889c006fd0 tx=0x7f889c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:13.924+0000 7f88a7e00700 1 -- 172.21.15.28:0/3491814826 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88a001f030 con 0x7f88b0108370 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.213+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f88b007cf20 con 0x7f88b0108370 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.347+0000 7f88a7e00700 1 -- 172.21.15.28:0/3491814826 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f88a000b3e0 con 0x7f88b0108370 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.350+0000 7f88a7e00700 1 -- 172.21.15.28:0/3491814826 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f88a0019c90 con 0x7f88b0108370 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.351+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f88940396c0 msgr2=0x7f889403bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.351+0000 7f88b6800700 1 --2- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f88940396c0 0x7f889403bb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f889c006fd0 tx=0x7f889c006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.351+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 msgr2=0x7f88b007db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.351+0000 7f88b6800700 1 --2- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b007db60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009520 tx=0x7f88a0011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.352+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 shutdown_connections 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.352+0000 7f88b6800700 1 --2- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7f88940396c0 0x7f889403bb80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.352+0000 7f88b6800700 1 --2- 172.21.15.28:0/3491814826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b0108370 0x7f88b007db60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.352+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 >> 172.21.15.28:0/3491814826 conn(0x7f88b01038f0 msgr2=0x7f88b0104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:15.109 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.352+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 shutdown_connections 2024-04-26T01:28:15.109 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:14.352+0000 7f88b6800700 1 -- 172.21.15.28:0/3491814826 wait complete. 2024-04-26T01:28:15.342 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:15 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:15.056+0000 7f2096802200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:28:15.596 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:15 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:15.481+0000 7f2096802200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:28:15.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:15 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3491814826' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T01:28:15.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:15 smithi028 ceph-mon[28247]: mgrmap e9: x(active, since 24s) 2024-04-26T01:28:15.894 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:15 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:15.600+0000 7f2096802200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:28:16.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:16 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2952210760' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.688+0000 7fde39200700 1 Processor -- start 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.689+0000 7fde39200700 1 -- start start 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.689+0000 7fde39200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3407c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.689+0000 7fde39200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde3407c900 con 0x7fde3407ded0 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.690+0000 7fde31e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3407c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.690+0000 7fde31e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3407c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60644/0 (socket says 172.21.15.28:60644) 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.690+0000 7fde31e00700 1 -- 172.21.15.28:0/821257816 learned_addr learned my addr 172.21.15.28:0/821257816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.690+0000 7fde31e00700 1 -- 172.21.15.28:0/821257816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3407ca80 con 0x7fde3407ded0 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.690+0000 7fde31e00700 1 --2- 172.21.15.28:0/821257816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3407c330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde240098a0 tx=0x7fde24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=691add42a0feb0c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.691+0000 7fde30a00700 1 -- 172.21.15.28:0/821257816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde24004160 con 0x7fde3407ded0 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.691+0000 7fde30a00700 1 -- 172.21.15.28:0/821257816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde24004300 con 0x7fde3407ded0 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.691+0000 7fde30a00700 1 -- 172.21.15.28:0/821257816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde24011a20 con 0x7fde3407ded0 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.691+0000 7fde39200700 1 -- 172.21.15.28:0/821257816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 msgr2=0x7fde3407c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.691+0000 7fde39200700 1 --2- 172.21.15.28:0/821257816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3407c330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde240098a0 tx=0x7fde24009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.692+0000 7fde39200700 1 -- 172.21.15.28:0/821257816 shutdown_connections 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.692+0000 7fde39200700 1 --2- 172.21.15.28:0/821257816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3407c330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.692+0000 7fde39200700 1 -- 172.21.15.28:0/821257816 >> 172.21.15.28:0/821257816 conn(0x7fde341038e0 msgr2=0x7fde34105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:16.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.692+0000 7fde39200700 1 -- 172.21.15.28:0/821257816 shutdown_connections 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.692+0000 7fde39200700 1 -- 172.21.15.28:0/821257816 wait complete. 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.693+0000 7fde39200700 1 Processor -- start 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.693+0000 7fde39200700 1 -- start start 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.693+0000 7fde39200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3419e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.693+0000 7fde39200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde3410e930 con 0x7fde3407ded0 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.694+0000 7fde31e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3419e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.694+0000 7fde31e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3419e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60648/0 (socket says 172.21.15.28:60648) 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.694+0000 7fde31e00700 1 -- 172.21.15.28:0/2952210760 learned_addr learned my addr 172.21.15.28:0/2952210760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.694+0000 7fde31e00700 1 -- 172.21.15.28:0/2952210760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3419e8e0 con 0x7fde3407ded0 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.694+0000 7fde31e00700 1 --2- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3419e3a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde24009130 tx=0x7fde24011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.695+0000 7fde2aa00700 1 -- 172.21.15.28:0/2952210760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde240039e0 con 0x7fde3407ded0 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.695+0000 7fde2aa00700 1 -- 172.21.15.28:0/2952210760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde2401a8e0 con 0x7fde3407ded0 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.695+0000 7fde2aa00700 1 -- 172.21.15.28:0/2952210760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde24019760 con 0x7fde3407ded0 2024-04-26T01:28:16.645 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.695+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3419eb70 con 0x7fde3407ded0 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.695+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde341a1860 con 0x7fde3407ded0 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.696+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde340640a0 con 0x7fde3407ded0 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.696+0000 7fde2aa00700 1 -- 172.21.15.28:0/2952210760 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fde24022020 con 0x7fde3407ded0 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.697+0000 7fde2aa00700 1 --2- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fde180396d0 0x7fde1803bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.697+0000 7fde2aa00700 1 -- 172.21.15.28:0/2952210760 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fde2404da20 con 0x7fde3407ded0 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.697+0000 7fde31400700 1 -- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fde180396d0 msgr2=0x7fde1803bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2991743583 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.697+0000 7fde31400700 1 --2- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fde180396d0 0x7fde1803bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.702+0000 7fde2aa00700 1 -- 172.21.15.28:0/2952210760 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde2402b350 con 0x7fde3407ded0 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.898+0000 7fde31400700 1 -- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fde180396d0 msgr2=0x7fde1803bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2991743583 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.898+0000 7fde31400700 1 --2- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fde180396d0 0x7fde1803bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.996+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fde341a1e10 con 0x7fde3407ded0 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.996+0000 7fde2aa00700 1 -- 172.21.15.28:0/2952210760 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fde2401aa90 con 0x7fde3407ded0 2024-04-26T01:28:16.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.999+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fde180396d0 msgr2=0x7fde1803bb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.999+0000 7fde39200700 1 --2- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fde180396d0 0x7fde1803bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.999+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 msgr2=0x7fde3419e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.999+0000 7fde39200700 1 --2- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3419e3a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde24009130 tx=0x7fde24011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.999+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 shutdown_connections 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.999+0000 7fde39200700 1 --2- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7fde180396d0 0x7fde1803bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.999+0000 7fde39200700 1 --2- 172.21.15.28:0/2952210760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3407ded0 0x7fde3419e3a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:15.999+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 >> 172.21.15.28:0/2952210760 conn(0x7fde341038e0 msgr2=0x7fde34104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:16.000+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 shutdown_connections 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:16.000+0000 7fde39200700 1 -- 172.21.15.28:0/2952210760 wait complete. 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-26T01:28:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-04-26T01:28:17.530 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:17 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:17.234+0000 7f2096802200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:28:17.530 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:17 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:17.471+0000 7f2096802200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:28:18.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:17 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:17.691+0000 7f2096802200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:28:18.479 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:18.088+0000 7f2096802200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:28:18.479 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:18.221+0000 7f2096802200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:28:18.741 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:18.483+0000 7f2096802200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:28:19.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:18.745+0000 7f2096802200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:28:20.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:19 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:19.556+0000 7f2096802200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:28:20.527 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:20 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:20.125+0000 7f2096802200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:28:20.528 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:20 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:20.255+0000 7f2096802200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:28:20.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:20 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:20.531+0000 7f2096802200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:28:21.530 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:21 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:21.202+0000 7f2096802200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:28:21.530 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:21 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:21.329+0000 7f2096802200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:28:21.974 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:21 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:21.661+0000 7f2096802200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:28:21.974 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:21 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:21.846+0000 7f2096802200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:28:22.280 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:21 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:21.977+0000 7f2096802200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:28:22.280 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:22 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:22.135+0000 7f2096802200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:28:22.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:22 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:22.514+0000 7f2096802200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:28:23.593 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:23.303+0000 7f2096802200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:28:24.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: Active manager daemon x restarted 2024-04-26T01:28:24.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: Activating manager daemon x 2024-04-26T01:28:24.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T01:28:24.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: mgrmap e10: x(active, starting, since 0.00834526s) 2024-04-26T01:28:24.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:28:24.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:28:24.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:28:24.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:28:24.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:28:24.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-mon[28247]: Manager daemon x is now available 2024-04-26T01:28:24.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:28:23 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:28:23.598+0000 7f2096802200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:28:24.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:28:24.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:28:24.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:28:24.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:24 smithi028 ceph-mon[28247]: mgrmap e11: x(active, since 1.01777s) 2024-04-26T01:28:25.473 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:28:25.473 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T01:28:25.473 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.262+0000 7fec01000700 1 Processor -- start 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.262+0000 7fec01000700 1 -- start start 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.263+0000 7fec01000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc07c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.263+0000 7fec01000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febfc07c810 con 0x7febfc07de20 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.263+0000 7febfb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc07c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.263+0000 7febfb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc07c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60652/0 (socket says 172.21.15.28:60652) 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.263+0000 7febfb400700 1 -- 172.21.15.28:0/4110094663 learned_addr learned my addr 172.21.15.28:0/4110094663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.264+0000 7febfb400700 1 -- 172.21.15.28:0/4110094663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc07c990 con 0x7febfc07de20 2024-04-26T01:28:25.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.264+0000 7febfb400700 1 --2- 172.21.15.28:0/4110094663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc07c2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7febec0098a0 tx=0x7febec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e10d8c14fca0ced server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:25.477 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.264+0000 7febfa000700 1 -- 172.21.15.28:0/4110094663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febec004160 con 0x7febfc07de20 2024-04-26T01:28:25.477 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.265+0000 7febfa000700 1 -- 172.21.15.28:0/4110094663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febec004300 con 0x7febfc07de20 2024-04-26T01:28:25.477 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.265+0000 7febfa000700 1 -- 172.21.15.28:0/4110094663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febec011a20 con 0x7febfc07de20 2024-04-26T01:28:25.477 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.265+0000 7fec01000700 1 -- 172.21.15.28:0/4110094663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 msgr2=0x7febfc07c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:25.477 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.265+0000 7fec01000700 1 --2- 172.21.15.28:0/4110094663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc07c2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7febec0098a0 tx=0x7febec009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.265+0000 7fec01000700 1 -- 172.21.15.28:0/4110094663 shutdown_connections 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.265+0000 7fec01000700 1 --2- 172.21.15.28:0/4110094663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc07c2d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.265+0000 7fec01000700 1 -- 172.21.15.28:0/4110094663 >> 172.21.15.28:0/4110094663 conn(0x7febfc103740 msgr2=0x7febfc105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.266+0000 7fec01000700 1 -- 172.21.15.28:0/4110094663 shutdown_connections 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.266+0000 7fec01000700 1 -- 172.21.15.28:0/4110094663 wait complete. 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.267+0000 7fec01000700 1 Processor -- start 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.267+0000 7fec01000700 1 -- start start 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.267+0000 7fec01000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.267+0000 7fec01000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febfc10e6e0 con 0x7febfc07de20 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.267+0000 7febfb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.267+0000 7febfb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc19e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60666/0 (socket says 172.21.15.28:60666) 2024-04-26T01:28:25.478 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.267+0000 7febfb400700 1 -- 172.21.15.28:0/2192446910 learned_addr learned my addr 172.21.15.28:0/2192446910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.268+0000 7febfb400700 1 -- 172.21.15.28:0/2192446910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc19e6c0 con 0x7febfc07de20 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.268+0000 7febfb400700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc19e180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7febec000c00 tx=0x7febec011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.268+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febec003af0 con 0x7febfc07de20 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.268+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febec01b8f0 con 0x7febfc07de20 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.268+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febec0238e0 con 0x7febfc07de20 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.268+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febfc19e950 con 0x7febfc07de20 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.269+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febfc1a1640 con 0x7febfc07de20 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.270+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7febec023b90 con 0x7febfc07de20 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.270+0000 7febf3e00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.270+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 --> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7febe003c2f0 con 0x7febe0039720 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.270+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7febec04e030 con 0x7febfc07de20 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.270+0000 7febfaa00700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 msgr2=0x7febe003bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2991743583 2024-04-26T01:28:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.270+0000 7febfaa00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:28:25.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.471+0000 7febfaa00700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 msgr2=0x7febe003bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2991743583 2024-04-26T01:28:25.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.471+0000 7febfaa00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:28:25.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.872+0000 7febfaa00700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 msgr2=0x7febe003bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2991743583 2024-04-26T01:28:25.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:17.872+0000 7febfaa00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T01:28:25.483 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:18.673+0000 7febfaa00700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 msgr2=0x7febe003bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2991743583 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:18.673+0000 7febfaa00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:20.275+0000 7febfaa00700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 msgr2=0x7febe003bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2991743583 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:20.275+0000 7febfaa00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:23.479+0000 7febfaa00700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 msgr2=0x7febe003bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2991743583 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:23.479+0000 7febfaa00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:23.606+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7febec02c3f0 con 0x7febfc07de20 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:23.606+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 msgr2=0x7febe003bbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:23.606+0000 7febf3e00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.611+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7febec00bce0 con 0x7febfc07de20 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.611+0000 7febf3e00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7febe003cfc0 0x7febe003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.611+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7febe003c2f0 con 0x7febe003cfc0 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.612+0000 7febfaa00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7febe003cfc0 0x7febe003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.614+0000 7febfaa00700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7febe003cfc0 0x7febe003f3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7febe8003a10 tx=0x7febe80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.615+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7febe003c2f0 con 0x7febe003cfc0 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.628+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7febfc069b50 con 0x7febe003cfc0 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.628+0000 7febf3e00700 1 -- 172.21.15.28:0/2192446910 <== mgr.14152 v2:172.21.15.28:6800/3431203710 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7febfc069b50 con 0x7febe003cfc0 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7febe003cfc0 msgr2=0x7febe003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7febe003cfc0 0x7febe003f3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7febe8003a10 tx=0x7febe80092b0 comp rx=0 tx=0).stop 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 msgr2=0x7febfc19e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc19e180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7febec000c00 tx=0x7febec011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 shutdown_connections 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7febe003cfc0 0x7febe003f3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc07de20 0x7febfc19e180 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 --2- 172.21.15.28:0/2192446910 >> [v2:172.21.15.28:6800/2991743583,v1:172.21.15.28:6801/2991743583] conn(0x7febe0039720 0x7febe003bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 >> 172.21.15.28:0/2192446910 conn(0x7febfc103740 msgr2=0x7febfc1041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 shutdown_connections 2024-04-26T01:28:25.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:24.629+0000 7fec01000700 1 -- 172.21.15.28:0/2192446910 wait complete. 2024-04-26T01:28:25.486 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-04-26T01:28:25.486 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-04-26T01:28:25.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:25 smithi028 ceph-mon[28247]: [26/Apr/2024:01:28:24] ENGINE Bus STARTING 2024-04-26T01:28:25.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:25 smithi028 ceph-mon[28247]: [26/Apr/2024:01:28:24] ENGINE Serving on https://172.21.15.28:7150 2024-04-26T01:28:25.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:25 smithi028 ceph-mon[28247]: [26/Apr/2024:01:28:24] ENGINE Serving on http://172.21.15.28:8765 2024-04-26T01:28:25.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:25 smithi028 ceph-mon[28247]: [26/Apr/2024:01:28:24] ENGINE Bus STARTED 2024-04-26T01:28:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:26 smithi028 ceph-mon[28247]: mgrmap e12: x(active, since 2s) 2024-04-26T01:28:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:26 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.536+0000 7f265c800700 1 Processor -- start 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.537+0000 7f265c800700 1 -- start start 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.537+0000 7f265c800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f2654108760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.537+0000 7f265c800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2654108d30 con 0x7f2654108340 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.538+0000 7f2659e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f2654108760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.538+0000 7f2659e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f2654108760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32950/0 (socket says 172.21.15.28:32950) 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.538+0000 7f2659e00700 1 -- 172.21.15.28:0/3199124623 learned_addr learned my addr 172.21.15.28:0/3199124623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.538+0000 7f2659e00700 1 -- 172.21.15.28:0/3199124623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2654109540 con 0x7f2654108340 2024-04-26T01:28:27.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.539+0000 7f2659e00700 1 --2- 172.21.15.28:0/3199124623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f2654108760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f26480098a0 tx=0x7f2648009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166b520f6603ffb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.539+0000 7f2658a00700 1 -- 172.21.15.28:0/3199124623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2648004160 con 0x7f2654108340 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.539+0000 7f2658a00700 1 -- 172.21.15.28:0/3199124623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2648004300 con 0x7f2654108340 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.539+0000 7f2658a00700 1 -- 172.21.15.28:0/3199124623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2648011a20 con 0x7f2654108340 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.540+0000 7f265c800700 1 -- 172.21.15.28:0/3199124623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 msgr2=0x7f2654108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.540+0000 7f265c800700 1 --2- 172.21.15.28:0/3199124623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f2654108760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f26480098a0 tx=0x7f2648009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.540+0000 7f265c800700 1 -- 172.21.15.28:0/3199124623 shutdown_connections 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.540+0000 7f265c800700 1 --2- 172.21.15.28:0/3199124623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f2654108760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.540+0000 7f265c800700 1 -- 172.21.15.28:0/3199124623 >> 172.21.15.28:0/3199124623 conn(0x7f2654103940 msgr2=0x7f2654105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.540+0000 7f265c800700 1 -- 172.21.15.28:0/3199124623 shutdown_connections 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.540+0000 7f265c800700 1 -- 172.21.15.28:0/3199124623 wait complete. 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.542+0000 7f265c800700 1 Processor -- start 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.542+0000 7f265c800700 1 -- start start 2024-04-26T01:28:27.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.543+0000 7f265c800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f265419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.543+0000 7f265c800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f265410c8f0 con 0x7f2654108340 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.543+0000 7f2659e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f265419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.543+0000 7f2659e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f265419e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32958/0 (socket says 172.21.15.28:32958) 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.543+0000 7f2659e00700 1 -- 172.21.15.28:0/3181621404 learned_addr learned my addr 172.21.15.28:0/3181621404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.544+0000 7f2659e00700 1 -- 172.21.15.28:0/3181621404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265419e910 con 0x7f2654108340 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.544+0000 7f2659e00700 1 --2- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f265419e3d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2648009520 tx=0x7f2648011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.544+0000 7f264ea00700 1 -- 172.21.15.28:0/3181621404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26480039e0 con 0x7f2654108340 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.544+0000 7f264ea00700 1 -- 172.21.15.28:0/3181621404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f264801b8f0 con 0x7f2654108340 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.544+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265419eba0 con 0x7f2654108340 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.545+0000 7f264ea00700 1 -- 172.21.15.28:0/3181621404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2648019770 con 0x7f2654108340 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.545+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26541a1890 con 0x7f2654108340 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.546+0000 7f264ea00700 1 -- 172.21.15.28:0/3181621404 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f264801b440 con 0x7f2654108340 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.546+0000 7f264ea00700 1 --2- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f263c039710 0x7f263c03bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.546+0000 7f264ea00700 1 -- 172.21.15.28:0/3181621404 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f26480516b0 con 0x7f2654108340 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.546+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26540640a0 con 0x7f2654108340 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.547+0000 7f2659400700 1 --2- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f263c039710 0x7f263c03bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.548+0000 7f2659400700 1 --2- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f263c039710 0x7f263c03bbd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2644006fd0 tx=0x7f2644006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.553+0000 7f264ea00700 1 -- 172.21.15.28:0/3181621404 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f264801f030 con 0x7f2654108340 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:26.791+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f26541049b0 con 0x7f263c039710 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.017+0000 7f264ea00700 1 -- 172.21.15.28:0/3181621404 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f26541049b0 con 0x7f263c039710 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.020+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f263c039710 msgr2=0x7f263c03bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.020+0000 7f265c800700 1 --2- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f263c039710 0x7f263c03bbd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2644006fd0 tx=0x7f2644006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.020+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 msgr2=0x7f265419e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.020+0000 7f265c800700 1 --2- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f265419e3d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2648009520 tx=0x7f2648011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.020+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 shutdown_connections 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.020+0000 7f265c800700 1 --2- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f263c039710 0x7f263c03bbd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.020+0000 7f265c800700 1 --2- 172.21.15.28:0/3181621404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2654108340 0x7f265419e3d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:27.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.020+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 >> 172.21.15.28:0/3181621404 conn(0x7f2654103940 msgr2=0x7f26541042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:27.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.021+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 shutdown_connections 2024-04-26T01:28:27.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:27.021+0000 7f265c800700 1 -- 172.21.15.28:0/3181621404 wait complete. 2024-04-26T01:28:27.715 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-04-26T01:28:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:28 smithi028 ceph-mon[28247]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:28:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:29.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:29 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:29.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:29 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:29.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$G4umd2kg90dtOevNFtNv9uo1/XZVe6CNhTqICw8AO370ytluZ14oy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714094908, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T01:28:29.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.330+0000 7f4905a00700 1 Processor -- start 2024-04-26T01:28:29.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.330+0000 7f4905a00700 1 -- start start 2024-04-26T01:28:29.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.330+0000 7f4905a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f490007c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:29.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.330+0000 7f4905a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f490007c900 con 0x7f490007ded0 2024-04-26T01:28:29.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.331+0000 7f48fea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f490007c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.331+0000 7f48fea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f490007c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32970/0 (socket says 172.21.15.28:32970) 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.331+0000 7f48fea00700 1 -- 172.21.15.28:0/1675536786 learned_addr learned my addr 172.21.15.28:0/1675536786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.331+0000 7f48fea00700 1 -- 172.21.15.28:0/1675536786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490007ca80 con 0x7f490007ded0 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.332+0000 7f48fea00700 1 --2- 172.21.15.28:0/1675536786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f490007c330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009e80 tx=0x7f48f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12e1da13081aca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.332+0000 7f48fe000700 1 -- 172.21.15.28:0/1675536786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f0017070 con 0x7f490007ded0 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.332+0000 7f48fe000700 1 -- 172.21.15.28:0/1675536786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48f000b8c0 con 0x7f490007ded0 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.332+0000 7f48fe000700 1 -- 172.21.15.28:0/1675536786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f0004490 con 0x7f490007ded0 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.333+0000 7f4905a00700 1 -- 172.21.15.28:0/1675536786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 msgr2=0x7f490007c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.333+0000 7f4905a00700 1 --2- 172.21.15.28:0/1675536786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f490007c330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009e80 tx=0x7f48f000b330 comp rx=0 tx=0).stop 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.333+0000 7f4905a00700 1 -- 172.21.15.28:0/1675536786 shutdown_connections 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.333+0000 7f4905a00700 1 --2- 172.21.15.28:0/1675536786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f490007c330 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.333+0000 7f4905a00700 1 -- 172.21.15.28:0/1675536786 >> 172.21.15.28:0/1675536786 conn(0x7f49001038e0 msgr2=0x7f4900105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.333+0000 7f4905a00700 1 -- 172.21.15.28:0/1675536786 shutdown_connections 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.333+0000 7f4905a00700 1 -- 172.21.15.28:0/1675536786 wait complete. 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.334+0000 7f4905a00700 1 Processor -- start 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.334+0000 7f4905a00700 1 -- start start 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.335+0000 7f4905a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f49001a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.335+0000 7f4905a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f490010a560 con 0x7f490007ded0 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.335+0000 7f48fea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f49001a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.335+0000 7f48fea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f49001a27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32978/0 (socket says 172.21.15.28:32978) 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.335+0000 7f48fea00700 1 -- 172.21.15.28:0/1601857852 learned_addr learned my addr 172.21.15.28:0/1601857852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.336+0000 7f48fea00700 1 -- 172.21.15.28:0/1601857852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49001a2d30 con 0x7f490007ded0 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.336+0000 7f48fea00700 1 --2- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f49001a27f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009fd0 tx=0x7f48f000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.336+0000 7f48f7e00700 1 -- 172.21.15.28:0/1601857852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f0020020 con 0x7f490007ded0 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.336+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49001a2fc0 con 0x7f490007ded0 2024-04-26T01:28:29.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.336+0000 7f48f7e00700 1 -- 172.21.15.28:0/1601857852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48f0003b50 con 0x7f490007ded0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.336+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49001a5cb0 con 0x7f490007ded0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.337+0000 7f48f7e00700 1 -- 172.21.15.28:0/1601857852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f0017040 con 0x7f490007ded0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.338+0000 7f48f7e00700 1 -- 172.21.15.28:0/1601857852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f48f0023020 con 0x7f490007ded0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.338+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49000640a0 con 0x7f490007ded0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.338+0000 7f48f7e00700 1 --2- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48e4039720 0x7f48e403bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.338+0000 7f48f7e00700 1 -- 172.21.15.28:0/1601857852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f48f000f050 con 0x7f490007ded0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.339+0000 7f48f6000700 1 --2- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48e4039720 0x7f48e403bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.339+0000 7f48f6000700 1 --2- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48e4039720 0x7f48e403bbe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f48e8006fd0 tx=0x7f48e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.349+0000 7f48f7e00700 1 -- 172.21.15.28:0/1601857852 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48f002a020 con 0x7f490007ded0 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.606+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f490002e1d0 con 0x7f48e4039720 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.875+0000 7f48f7e00700 1 -- 172.21.15.28:0/1601857852 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f490002e1d0 con 0x7f48e4039720 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.879+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48e4039720 msgr2=0x7f48e403bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.879+0000 7f4905a00700 1 --2- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48e4039720 0x7f48e403bbe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f48e8006fd0 tx=0x7f48e8006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.879+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 msgr2=0x7f49001a27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.879+0000 7f4905a00700 1 --2- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f49001a27f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009fd0 tx=0x7f48f000bdc0 comp rx=0 tx=0).stop 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.880+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 shutdown_connections 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.880+0000 7f4905a00700 1 --2- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48e4039720 0x7f48e403bbe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.880+0000 7f4905a00700 1 --2- 172.21.15.28:0/1601857852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490007ded0 0x7f49001a27f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.880+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 >> 172.21.15.28:0/1601857852 conn(0x7f49001038e0 msgr2=0x7f4900105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.880+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 shutdown_connections 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:28.880+0000 7f4905a00700 1 -- 172.21.15.28:0/1601857852 wait complete. 2024-04-26T01:28:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-04-26T01:28:30.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:30 smithi028 ceph-mon[28247]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:28:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:31 smithi028 ceph-mon[28247]: mgrmap e13: x(active, since 6s) 2024-04-26T01:28:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:31 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2483453029' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T01:28:31.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T01:28:31.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.444+0000 7f3964600700 1 Processor -- start 2024-04-26T01:28:31.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.444+0000 7f3964600700 1 -- start start 2024-04-26T01:28:31.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.445+0000 7f3964600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:31.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.445+0000 7f3964600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f395c107c30 con 0x7f395c107240 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.445+0000 7f3961c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.445+0000 7f3961c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50380/0 (socket says 172.21.15.28:50380) 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.445+0000 7f3961c00700 1 -- 172.21.15.28:0/2606708739 learned_addr learned my addr 172.21.15.28:0/2606708739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.446+0000 7f3961c00700 1 -- 172.21.15.28:0/2606708739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395c108440 con 0x7f395c107240 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.446+0000 7f3961c00700 1 --2- 172.21.15.28:0/2606708739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c107660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39500098a0 tx=0x7f3950009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce84d62aac010c1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.446+0000 7f395be00700 1 -- 172.21.15.28:0/2606708739 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3950004160 con 0x7f395c107240 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.447+0000 7f395be00700 1 -- 172.21.15.28:0/2606708739 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3950004300 con 0x7f395c107240 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.447+0000 7f395be00700 1 -- 172.21.15.28:0/2606708739 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3950011a20 con 0x7f395c107240 2024-04-26T01:28:31.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.447+0000 7f3964600700 1 -- 172.21.15.28:0/2606708739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 msgr2=0x7f395c107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:31.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.447+0000 7f3964600700 1 --2- 172.21.15.28:0/2606708739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c107660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39500098a0 tx=0x7f3950009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.447+0000 7f3964600700 1 -- 172.21.15.28:0/2606708739 shutdown_connections 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.447+0000 7f3964600700 1 --2- 172.21.15.28:0/2606708739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c107660 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.447+0000 7f3964600700 1 -- 172.21.15.28:0/2606708739 >> 172.21.15.28:0/2606708739 conn(0x7f395c102820 msgr2=0x7f395c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.448+0000 7f3964600700 1 -- 172.21.15.28:0/2606708739 shutdown_connections 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.448+0000 7f3964600700 1 -- 172.21.15.28:0/2606708739 wait complete. 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.449+0000 7f3964600700 1 Processor -- start 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.449+0000 7f3964600700 1 -- start start 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.449+0000 7f3964600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.449+0000 7f3964600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f395c10b7f0 con 0x7f395c107240 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.449+0000 7f3961c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:31.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.449+0000 7f3961c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c19e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50388/0 (socket says 172.21.15.28:50388) 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.449+0000 7f3961c00700 1 -- 172.21.15.28:0/2483453029 learned_addr learned my addr 172.21.15.28:0/2483453029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.450+0000 7f3961c00700 1 -- 172.21.15.28:0/2483453029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395c19e990 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.450+0000 7f3961c00700 1 --2- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c19e450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f39500116e0 tx=0x7f3950011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.450+0000 7f395a000700 1 -- 172.21.15.28:0/2483453029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39500039e0 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.450+0000 7f395a000700 1 -- 172.21.15.28:0/2483453029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f395001b8f0 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.450+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395c19ec20 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.450+0000 7f395a000700 1 -- 172.21.15.28:0/2483453029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3950023760 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.450+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395c1a1870 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.452+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395c0640a0 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.452+0000 7f395a000700 1 -- 172.21.15.28:0/2483453029 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f395001b440 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.453+0000 7f395a000700 1 --2- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3944039770 0x7f394403bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.453+0000 7f395a000700 1 -- 172.21.15.28:0/2483453029 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f395004e170 con 0x7f395c107240 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.453+0000 7f3961200700 1 --2- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3944039770 0x7f394403bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:31.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.454+0000 7f3961200700 1 --2- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3944039770 0x7f394403bc30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3948006fd0 tx=0x7f3948006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.459+0000 7f395a000700 1 -- 172.21.15.28:0/2483453029 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f395001f030 con 0x7f395c107240 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.694+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f395c02e210 con 0x7f395c107240 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.695+0000 7f395a000700 1 -- 172.21.15.28:0/2483453029 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f395002ce10 con 0x7f395c107240 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.699+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3944039770 msgr2=0x7f394403bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.699+0000 7f3964600700 1 --2- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3944039770 0x7f394403bc30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3948006fd0 tx=0x7f3948006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.699+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 msgr2=0x7f395c19e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.699+0000 7f3964600700 1 --2- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c19e450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f39500116e0 tx=0x7f3950011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.699+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 shutdown_connections 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.699+0000 7f3964600700 1 --2- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3944039770 0x7f394403bc30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.699+0000 7f3964600700 1 --2- 172.21.15.28:0/2483453029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f395c107240 0x7f395c19e450 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.699+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 >> 172.21.15.28:0/2483453029 conn(0x7f395c102820 msgr2=0x7f395c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.700+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 shutdown_connections 2024-04-26T01:28:31.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:30.700+0000 7f3964600700 1 -- 172.21.15.28:0/2483453029 wait complete. 2024-04-26T01:28:31.910 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:28:31.910 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-26T01:28:31.936 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:28:31.936 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-26T01:28:31.936 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-26T01:28:31.937 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T01:28:31.943 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-04-26T01:28:31.943 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:31.944 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-04-26T01:28:31.944 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-04-26T01:28:31.944 INFO:teuthology.orchestra.run.smithi028.stdout: Password: jrq44s3gz4 2024-04-26T01:28:31.944 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:31.944 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config directory 2024-04-26T01:28:31.947 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-04-26T01:28:33.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.611+0000 7fb732800700 1 Processor -- start 2024-04-26T01:28:33.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.612+0000 7fb732800700 1 -- start start 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.612+0000 7fb732800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c10b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.612+0000 7fb732800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb72c07bde0 con 0x7fb72c108e60 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.613+0000 7fb72b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c10b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.613+0000 7fb72b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c10b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50404/0 (socket says 172.21.15.28:50404) 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.613+0000 7fb72b400700 1 -- 172.21.15.28:0/3982243596 learned_addr learned my addr 172.21.15.28:0/3982243596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.613+0000 7fb72b400700 1 -- 172.21.15.28:0/3982243596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb72c10b790 con 0x7fb72c108e60 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.613+0000 7fb72b400700 1 --2- 172.21.15.28:0/3982243596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c10b250 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0098a0 tx=0x7fb71c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2637d7839fa56a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.614+0000 7fb72a000700 1 -- 172.21.15.28:0/3982243596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb71c004160 con 0x7fb72c108e60 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.614+0000 7fb72a000700 1 -- 172.21.15.28:0/3982243596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb71c004300 con 0x7fb72c108e60 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.614+0000 7fb72a000700 1 -- 172.21.15.28:0/3982243596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb71c011a20 con 0x7fb72c108e60 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.614+0000 7fb732800700 1 -- 172.21.15.28:0/3982243596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 msgr2=0x7fb72c10b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.614+0000 7fb732800700 1 --2- 172.21.15.28:0/3982243596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c10b250 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0098a0 tx=0x7fb71c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:33.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.615+0000 7fb732800700 1 -- 172.21.15.28:0/3982243596 shutdown_connections 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.615+0000 7fb732800700 1 --2- 172.21.15.28:0/3982243596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c10b250 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.615+0000 7fb732800700 1 -- 172.21.15.28:0/3982243596 >> 172.21.15.28:0/3982243596 conn(0x7fb72c1027d0 msgr2=0x7fb72c104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.615+0000 7fb732800700 1 -- 172.21.15.28:0/3982243596 shutdown_connections 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.615+0000 7fb732800700 1 -- 172.21.15.28:0/3982243596 wait complete. 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.616+0000 7fb732800700 1 Processor -- start 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.616+0000 7fb732800700 1 -- start start 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.616+0000 7fb732800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c19e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.616+0000 7fb732800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb72c10c6e0 con 0x7fb72c108e60 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.617+0000 7fb72b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c19e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.617+0000 7fb72b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c19e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50420/0 (socket says 172.21.15.28:50420) 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.617+0000 7fb72b400700 1 -- 172.21.15.28:0/4167720079 learned_addr learned my addr 172.21.15.28:0/4167720079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.617+0000 7fb72b400700 1 -- 172.21.15.28:0/4167720079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb72c19e8e0 con 0x7fb72c108e60 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.617+0000 7fb72b400700 1 --2- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c19e3a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb71c000c00 tx=0x7fb71c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.618+0000 7fb723e00700 1 -- 172.21.15.28:0/4167720079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb71c003aa0 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.618+0000 7fb723e00700 1 -- 172.21.15.28:0/4167720079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb71c01b8f0 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.618+0000 7fb723e00700 1 -- 172.21.15.28:0/4167720079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb71c023870 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.618+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb72c19eb70 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.618+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb72c1a1860 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.619+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb72c0640a0 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.619+0000 7fb723e00700 1 -- 172.21.15.28:0/4167720079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb71c01b440 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.620+0000 7fb723e00700 1 --2- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7100397c0 0x7fb71003bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.620+0000 7fb723e00700 1 -- 172.21.15.28:0/4167720079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb71c04e130 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.620+0000 7fb72aa00700 1 --2- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7100397c0 0x7fb71003bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.620+0000 7fb72aa00700 1 --2- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7100397c0 0x7fb71003bc80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb718006fd0 tx=0x7fb718006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.625+0000 7fb723e00700 1 -- 172.21.15.28:0/4167720079 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb71c01f030 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.855+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fb72c1a1a30 con 0x7fb72c108e60 2024-04-26T01:28:33.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.856+0000 7fb723e00700 1 -- 172.21.15.28:0/4167720079 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fb72c1a1a30 con 0x7fb72c108e60 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.859+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7100397c0 msgr2=0x7fb71003bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.859+0000 7fb732800700 1 --2- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7100397c0 0x7fb71003bc80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb718006fd0 tx=0x7fb718006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.859+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 msgr2=0x7fb72c19e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.859+0000 7fb732800700 1 --2- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c19e3a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb71c000c00 tx=0x7fb71c011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.859+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 shutdown_connections 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.859+0000 7fb732800700 1 --2- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7100397c0 0x7fb71003bc80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.859+0000 7fb732800700 1 --2- 172.21.15.28:0/4167720079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb72c108e60 0x7fb72c19e3a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.859+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 >> 172.21.15.28:0/4167720079 conn(0x7fb72c1027d0 msgr2=0x7fb72c103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.860+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 shutdown_connections 2024-04-26T01:28:33.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:32.860+0000 7fb732800700 1 -- 172.21.15.28:0/4167720079 wait complete. 2024-04-26T01:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.191+0000 7fe4c5200700 1 Processor -- start 2024-04-26T01:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.192+0000 7fe4c5200700 1 -- start start 2024-04-26T01:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.192+0000 7fe4c5200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c0110390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.192+0000 7fe4c5200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c0110960 con 0x7fe4c010ff70 2024-04-26T01:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.193+0000 7fe4bde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c0110390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.193+0000 7fe4bde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c0110390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50426/0 (socket says 172.21.15.28:50426) 2024-04-26T01:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.193+0000 7fe4bde00700 1 -- 172.21.15.28:0/3699692290 learned_addr learned my addr 172.21.15.28:0/3699692290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.193+0000 7fe4bde00700 1 -- 172.21.15.28:0/3699692290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c0110ae0 con 0x7fe4c010ff70 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.194+0000 7fe4bde00700 1 --2- 172.21.15.28:0/3699692290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c0110390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe4b00098a0 tx=0x7fe4b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dd7fc2a06a3c6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.194+0000 7fe4bca00700 1 -- 172.21.15.28:0/3699692290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4b0004160 con 0x7fe4c010ff70 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.194+0000 7fe4bca00700 1 -- 172.21.15.28:0/3699692290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4b0004300 con 0x7fe4c010ff70 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.194+0000 7fe4bca00700 1 -- 172.21.15.28:0/3699692290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4b0011a20 con 0x7fe4c010ff70 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.195+0000 7fe4c5200700 1 -- 172.21.15.28:0/3699692290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 msgr2=0x7fe4c0110390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.195+0000 7fe4c5200700 1 --2- 172.21.15.28:0/3699692290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c0110390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe4b00098a0 tx=0x7fe4b0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.195+0000 7fe4c5200700 1 -- 172.21.15.28:0/3699692290 shutdown_connections 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.195+0000 7fe4c5200700 1 --2- 172.21.15.28:0/3699692290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c0110390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.195+0000 7fe4c5200700 1 -- 172.21.15.28:0/3699692290 >> 172.21.15.28:0/3699692290 conn(0x7fe4c007da40 msgr2=0x7fe4c007fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.195+0000 7fe4c5200700 1 -- 172.21.15.28:0/3699692290 shutdown_connections 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.195+0000 7fe4c5200700 1 -- 172.21.15.28:0/3699692290 wait complete. 2024-04-26T01:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.196+0000 7fe4c5200700 1 Processor -- start 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.196+0000 7fe4c5200700 1 -- start start 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.197+0000 7fe4c5200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c01a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.197+0000 7fe4c5200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c010fb50 con 0x7fe4c010ff70 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.197+0000 7fe4bde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c01a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.197+0000 7fe4bde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c01a2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50440/0 (socket says 172.21.15.28:50440) 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.197+0000 7fe4bde00700 1 -- 172.21.15.28:0/1638718423 learned_addr learned my addr 172.21.15.28:0/1638718423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.197+0000 7fe4bde00700 1 -- 172.21.15.28:0/1638718423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c01a2cb0 con 0x7fe4c010ff70 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.198+0000 7fe4bde00700 1 --2- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c01a2770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009520 tx=0x7fe4b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.198+0000 7fe4b6a00700 1 -- 172.21.15.28:0/1638718423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4b00039e0 con 0x7fe4c010ff70 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.198+0000 7fe4b6a00700 1 -- 172.21.15.28:0/1638718423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4b001b8f0 con 0x7fe4c010ff70 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.198+0000 7fe4b6a00700 1 -- 172.21.15.28:0/1638718423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4b0019770 con 0x7fe4c010ff70 2024-04-26T01:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.198+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c01a2f40 con 0x7fe4c010ff70 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.198+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c01a5c30 con 0x7fe4c010ff70 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.199+0000 7fe4b6a00700 1 -- 172.21.15.28:0/1638718423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe4b001b440 con 0x7fe4c010ff70 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.199+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c00640a0 con 0x7fe4c010ff70 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.200+0000 7fe4b6a00700 1 --2- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe4a40397b0 0x7fe4a403bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.200+0000 7fe4b6a00700 1 -- 172.21.15.28:0/1638718423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe4b004e100 con 0x7fe4c010ff70 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.200+0000 7fe4bd400700 1 --2- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe4a40397b0 0x7fe4a403bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.201+0000 7fe4bd400700 1 --2- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe4a40397b0 0x7fe4a403bc70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac006fd0 tx=0x7fe4ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.206+0000 7fe4b6a00700 1 -- 172.21.15.28:0/1638718423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe4b001f030 con 0x7fe4c010ff70 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.564+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fe4c01a6220 con 0x7fe4c010ff70 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.570+0000 7fe4b6a00700 1 -- 172.21.15.28:0/1638718423 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fe4b0010380 con 0x7fe4c010ff70 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe4a40397b0 msgr2=0x7fe4a403bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 --2- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe4a40397b0 0x7fe4a403bc70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac006fd0 tx=0x7fe4ac006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 msgr2=0x7fe4c01a2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 --2- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c01a2770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009520 tx=0x7fe4b0011aa0 comp rx=0 tx=0).stop 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 shutdown_connections 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 --2- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe4a40397b0 0x7fe4a403bc70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 --2- 172.21.15.28:0/1638718423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c010ff70 0x7fe4c01a2770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 >> 172.21.15.28:0/1638718423 conn(0x7fe4c007da40 msgr2=0x7fe4c007f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.573+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 shutdown_connections 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-26T01:28:34.574+0000 7fe4c5200700 1 -- 172.21.15.28:0/1638718423 wait complete. 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T01:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T01:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-04-26T01:28:35.334 INFO:tasks.cephadm:Fetching config... 2024-04-26T01:28:35.335 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:28:35.335 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T01:28:35.377 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T01:28:35.377 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:28:35.377 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T01:28:35.456 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T01:28:35.457 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:28:35.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T01:28:35.569 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T01:28:35.569 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:28:35.569 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T01:28:35.652 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T01:28:35.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDobPXclO1by8lhIOg+eOXbX1tAObEniuejqmVGwUhsvdOKdx5S2v61zSc1/BtBHewGH7wuOIBubobaPrRwiecntmQ7ID/2hSlSesAkgiGh10KO61hXxVlXArb1iW5YGKAN+b4R4+WHdFP6WtTe6CtnvsDYo5CpQ/h0JIRSPT8se3GajVEmk9Q+PUpKofCRvxOkSC8YEW4c0USfC5ROLGUkGQa1FhFwbXsVEk+YPLriExti++Bqy4+STEDonYhyRXVIQQ+AClooBkfp3Voc0SoMwpg54Y819PA43WtHanROHTEsV0fXlO/CtbzTWT90yKxhKpeN+w+qeG28bpif/T762UtauP/tzLeUPthu5ydk+dK5OV/zZAAnAlZOOU0vZ4jFG0IkPYCBX8MGvABELXnV14M+0pZUhCbXzbD2PEOpWvKvL7TDd/hvVWZ5svDm1gDcArrMRdleMCk/40wEUYCs4ec6PodvuijX1D4FveN0AhaVYNaX0o28ut0Mvj6smjk= ceph-ead407d4-036b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDobPXclO1by8lhIOg+eOXbX1tAObEniuejqmVGwUhsvdOKdx5S2v61zSc1/BtBHewGH7wuOIBubobaPrRwiecntmQ7ID/2hSlSesAkgiGh10KO61hXxVlXArb1iW5YGKAN+b4R4+WHdFP6WtTe6CtnvsDYo5CpQ/h0JIRSPT8se3GajVEmk9Q+PUpKofCRvxOkSC8YEW4c0USfC5ROLGUkGQa1FhFwbXsVEk+YPLriExti++Bqy4+STEDonYhyRXVIQQ+AClooBkfp3Voc0SoMwpg54Y819PA43WtHanROHTEsV0fXlO/CtbzTWT90yKxhKpeN+w+qeG28bpif/T762UtauP/tzLeUPthu5ydk+dK5OV/zZAAnAlZOOU0vZ4jFG0IkPYCBX8MGvABELXnV14M+0pZUhCbXzbD2PEOpWvKvL7TDd/hvVWZ5svDm1gDcArrMRdleMCk/40wEUYCs4ec6PodvuijX1D4FveN0AhaVYNaX0o28ut0Mvj6smjk= ceph-ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:28:35.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:35 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1638718423' entity='client.admin' 2024-04-26T01:28:35.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDobPXclO1by8lhIOg+eOXbX1tAObEniuejqmVGwUhsvdOKdx5S2v61zSc1/BtBHewGH7wuOIBubobaPrRwiecntmQ7ID/2hSlSesAkgiGh10KO61hXxVlXArb1iW5YGKAN+b4R4+WHdFP6WtTe6CtnvsDYo5CpQ/h0JIRSPT8se3GajVEmk9Q+PUpKofCRvxOkSC8YEW4c0USfC5ROLGUkGQa1FhFwbXsVEk+YPLriExti++Bqy4+STEDonYhyRXVIQQ+AClooBkfp3Voc0SoMwpg54Y819PA43WtHanROHTEsV0fXlO/CtbzTWT90yKxhKpeN+w+qeG28bpif/T762UtauP/tzLeUPthu5ydk+dK5OV/zZAAnAlZOOU0vZ4jFG0IkPYCBX8MGvABELXnV14M+0pZUhCbXzbD2PEOpWvKvL7TDd/hvVWZ5svDm1gDcArrMRdleMCk/40wEUYCs4ec6PodvuijX1D4FveN0AhaVYNaX0o28ut0Mvj6smjk= ceph-ead407d4-036b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:28:35.849 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDobPXclO1by8lhIOg+eOXbX1tAObEniuejqmVGwUhsvdOKdx5S2v61zSc1/BtBHewGH7wuOIBubobaPrRwiecntmQ7ID/2hSlSesAkgiGh10KO61hXxVlXArb1iW5YGKAN+b4R4+WHdFP6WtTe6CtnvsDYo5CpQ/h0JIRSPT8se3GajVEmk9Q+PUpKofCRvxOkSC8YEW4c0USfC5ROLGUkGQa1FhFwbXsVEk+YPLriExti++Bqy4+STEDonYhyRXVIQQ+AClooBkfp3Voc0SoMwpg54Y819PA43WtHanROHTEsV0fXlO/CtbzTWT90yKxhKpeN+w+qeG28bpif/T762UtauP/tzLeUPthu5ydk+dK5OV/zZAAnAlZOOU0vZ4jFG0IkPYCBX8MGvABELXnV14M+0pZUhCbXzbD2PEOpWvKvL7TDd/hvVWZ5svDm1gDcArrMRdleMCk/40wEUYCs4ec6PodvuijX1D4FveN0AhaVYNaX0o28ut0Mvj6smjk= ceph-ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:28:35.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDobPXclO1by8lhIOg+eOXbX1tAObEniuejqmVGwUhsvdOKdx5S2v61zSc1/BtBHewGH7wuOIBubobaPrRwiecntmQ7ID/2hSlSesAkgiGh10KO61hXxVlXArb1iW5YGKAN+b4R4+WHdFP6WtTe6CtnvsDYo5CpQ/h0JIRSPT8se3GajVEmk9Q+PUpKofCRvxOkSC8YEW4c0USfC5ROLGUkGQa1FhFwbXsVEk+YPLriExti++Bqy4+STEDonYhyRXVIQQ+AClooBkfp3Voc0SoMwpg54Y819PA43WtHanROHTEsV0fXlO/CtbzTWT90yKxhKpeN+w+qeG28bpif/T762UtauP/tzLeUPthu5ydk+dK5OV/zZAAnAlZOOU0vZ4jFG0IkPYCBX8MGvABELXnV14M+0pZUhCbXzbD2PEOpWvKvL7TDd/hvVWZ5svDm1gDcArrMRdleMCk/40wEUYCs4ec6PodvuijX1D4FveN0AhaVYNaX0o28ut0Mvj6smjk= ceph-ead407d4-036b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:28:35.904 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDobPXclO1by8lhIOg+eOXbX1tAObEniuejqmVGwUhsvdOKdx5S2v61zSc1/BtBHewGH7wuOIBubobaPrRwiecntmQ7ID/2hSlSesAkgiGh10KO61hXxVlXArb1iW5YGKAN+b4R4+WHdFP6WtTe6CtnvsDYo5CpQ/h0JIRSPT8se3GajVEmk9Q+PUpKofCRvxOkSC8YEW4c0USfC5ROLGUkGQa1FhFwbXsVEk+YPLriExti++Bqy4+STEDonYhyRXVIQQ+AClooBkfp3Voc0SoMwpg54Y819PA43WtHanROHTEsV0fXlO/CtbzTWT90yKxhKpeN+w+qeG28bpif/T762UtauP/tzLeUPthu5ydk+dK5OV/zZAAnAlZOOU0vZ4jFG0IkPYCBX8MGvABELXnV14M+0pZUhCbXzbD2PEOpWvKvL7TDd/hvVWZ5svDm1gDcArrMRdleMCk/40wEUYCs4ec6PodvuijX1D4FveN0AhaVYNaX0o28ut0Mvj6smjk= ceph-ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:28:35.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T01:28:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:28:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.032+0000 7f6acb400700 1 -- 172.21.15.28:0/838457850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 msgr2=0x7f6ac4109070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.032+0000 7f6acb400700 1 --2- 172.21.15.28:0/838457850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 0x7f6ac4109070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009910 tx=0x7f6ab8009c20 comp rx=0 tx=0).stop 2024-04-26T01:28:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.032+0000 7f6acb400700 1 -- 172.21.15.28:0/838457850 shutdown_connections 2024-04-26T01:28:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.032+0000 7f6acb400700 1 --2- 172.21.15.28:0/838457850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 0x7f6ac4109070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.032+0000 7f6acb400700 1 -- 172.21.15.28:0/838457850 >> 172.21.15.28:0/838457850 conn(0x7f6ac4077b30 msgr2=0x7f6ac4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.032+0000 7f6acb400700 1 -- 172.21.15.28:0/838457850 shutdown_connections 2024-04-26T01:28:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.033+0000 7f6acb400700 1 -- 172.21.15.28:0/838457850 wait complete. 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.034+0000 7f6acb400700 1 Processor -- start 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.034+0000 7f6acb400700 1 -- start start 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.034+0000 7f6acb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 0x7f6ac419e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.034+0000 7f6acb400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac410ea20 con 0x7f6ac4108c50 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.035+0000 7f6ac8a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 0x7f6ac419e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.035+0000 7f6ac8a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 0x7f6ac419e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58366/0 (socket says 172.21.15.28:58366) 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.035+0000 7f6ac8a00700 1 -- 172.21.15.28:0/2369478905 learned_addr learned my addr 172.21.15.28:0/2369478905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.035+0000 7f6ac8a00700 1 -- 172.21.15.28:0/2369478905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac419e670 con 0x7f6ac4108c50 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.035+0000 7f6ac8a00700 1 --2- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 0x7f6ac419e130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8005950 tx=0x7f6ab8004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.036+0000 7f6ac1600700 1 -- 172.21.15.28:0/2369478905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ab8026070 con 0x7f6ac4108c50 2024-04-26T01:28:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.036+0000 7f6ac1600700 1 -- 172.21.15.28:0/2369478905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ab800bc00 con 0x7f6ac4108c50 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.036+0000 7f6ac1600700 1 -- 172.21.15.28:0/2369478905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ab8003a10 con 0x7f6ac4108c50 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.036+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac419e900 con 0x7f6ac4108c50 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.036+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac41a15a0 con 0x7f6ac4108c50 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.037+0000 7f6ac1600700 1 -- 172.21.15.28:0/2369478905 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6ab8029070 con 0x7f6ac4108c50 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.037+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac410f1a0 con 0x7f6ac4108c50 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.038+0000 7f6ac1600700 1 --2- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6aac0396b0 0x7f6aac03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.038+0000 7f6ac1600700 1 -- 172.21.15.28:0/2369478905 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6ab801e070 con 0x7f6ac4108c50 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.038+0000 7f6ac3e00700 1 --2- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6aac0396b0 0x7f6aac03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.039+0000 7f6ac3e00700 1 --2- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6aac0396b0 0x7f6aac03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0006fd0 tx=0x7f6ab0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.044+0000 7f6ac1600700 1 -- 172.21.15.28:0/2369478905 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ab8013070 con 0x7f6ac4108c50 2024-04-26T01:28:39.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.273+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f6ac402e090 con 0x7f6ac4108c50 2024-04-26T01:28:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.277+0000 7f6ac1600700 1 -- 172.21.15.28:0/2369478905 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f6ab80300c0 con 0x7f6ac4108c50 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.280+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6aac0396b0 msgr2=0x7f6aac03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.280+0000 7f6acb400700 1 --2- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6aac0396b0 0x7f6aac03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0006fd0 tx=0x7f6ab0006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.280+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 msgr2=0x7f6ac419e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.280+0000 7f6acb400700 1 --2- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 0x7f6ac419e130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8005950 tx=0x7f6ab8004fa0 comp rx=0 tx=0).stop 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.281+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 shutdown_connections 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.281+0000 7f6acb400700 1 --2- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6aac0396b0 0x7f6aac03bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.281+0000 7f6acb400700 1 --2- 172.21.15.28:0/2369478905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ac4108c50 0x7f6ac419e130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.281+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 >> 172.21.15.28:0/2369478905 conn(0x7f6ac4077b30 msgr2=0x7f6ac4078660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.281+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 shutdown_connections 2024-04-26T01:28:39.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:39.281+0000 7f6acb400700 1 -- 172.21.15.28:0/2369478905 wait complete. 2024-04-26T01:28:39.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:28:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:28:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2369478905' entity='client.admin' 2024-04-26T01:28:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:28:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:40.035 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T01:28:40.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T01:28:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:28:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.519+0000 7f31a9200700 1 -- 172.21.15.28:0/2627415938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 msgr2=0x7f31a4105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.519+0000 7f31a9200700 1 --2- 172.21.15.28:0/2627415938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 0x7f31a4105320 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3194009910 tx=0x7f3194009c20 comp rx=0 tx=0).stop 2024-04-26T01:28:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.520+0000 7f31a9200700 1 -- 172.21.15.28:0/2627415938 shutdown_connections 2024-04-26T01:28:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.520+0000 7f31a9200700 1 --2- 172.21.15.28:0/2627415938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 0x7f31a4105320 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.520+0000 7f31a9200700 1 -- 172.21.15.28:0/2627415938 >> 172.21.15.28:0/2627415938 conn(0x7f31a40fc940 msgr2=0x7f31a40fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.520+0000 7f31a9200700 1 -- 172.21.15.28:0/2627415938 shutdown_connections 2024-04-26T01:28:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.520+0000 7f31a9200700 1 -- 172.21.15.28:0/2627415938 wait complete. 2024-04-26T01:28:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.521+0000 7f31a9200700 1 Processor -- start 2024-04-26T01:28:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.521+0000 7f31a9200700 1 -- start start 2024-04-26T01:28:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.521+0000 7f31a9200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 0x7f31a4199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.521+0000 7f31a9200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4108380 con 0x7f31a4102f30 2024-04-26T01:28:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.522+0000 7f31a1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 0x7f31a4199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.522+0000 7f31a1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 0x7f31a4199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58384/0 (socket says 172.21.15.28:58384) 2024-04-26T01:28:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.522+0000 7f31a1e00700 1 -- 172.21.15.28:0/2375201700 learned_addr learned my addr 172.21.15.28:0/2375201700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.522+0000 7f31a1e00700 1 -- 172.21.15.28:0/2375201700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a419a1b0 con 0x7f31a4102f30 2024-04-26T01:28:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.522+0000 7f31a1e00700 1 --2- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 0x7f31a4199c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3194005b40 tx=0x7f3194004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.523+0000 7f319aa00700 1 -- 172.21.15.28:0/2375201700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3194027070 con 0x7f31a4102f30 2024-04-26T01:28:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.523+0000 7f319aa00700 1 -- 172.21.15.28:0/2375201700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f319400be10 con 0x7f31a4102f30 2024-04-26T01:28:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.523+0000 7f319aa00700 1 -- 172.21.15.28:0/2375201700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3194022de0 con 0x7f31a4102f30 2024-04-26T01:28:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.523+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a419a4a0 con 0x7f31a4102f30 2024-04-26T01:28:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.523+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a419d130 con 0x7f31a4102f30 2024-04-26T01:28:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.524+0000 7f319aa00700 1 -- 172.21.15.28:0/2375201700 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f319402a070 con 0x7f31a4102f30 2024-04-26T01:28:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.524+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a4108b00 con 0x7f31a4102f30 2024-04-26T01:28:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.525+0000 7f319aa00700 1 --2- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3188039650 0x7f318803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.525+0000 7f319aa00700 1 -- 172.21.15.28:0/2375201700 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f319401e030 con 0x7f31a4102f30 2024-04-26T01:28:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.525+0000 7f319be00700 1 --2- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3188039650 0x7f318803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.526+0000 7f319be00700 1 --2- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3188039650 0x7f318803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f318c006fd0 tx=0x7f318c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.530+0000 7f319aa00700 1 -- 172.21.15.28:0/2375201700 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3194013070 con 0x7f31a4102f30 2024-04-26T01:28:42.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.766+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f31a402e170 con 0x7f3188039650 2024-04-26T01:28:42.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.772+0000 7f319aa00700 1 -- 172.21.15.28:0/2375201700 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f31a402e170 con 0x7f3188039650 2024-04-26T01:28:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.775+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3188039650 msgr2=0x7f318803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.775+0000 7f31a9200700 1 --2- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3188039650 0x7f318803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f318c006fd0 tx=0x7f318c006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.775+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 msgr2=0x7f31a4199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.775+0000 7f31a9200700 1 --2- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 0x7f31a4199c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3194005b40 tx=0x7f3194004f80 comp rx=0 tx=0).stop 2024-04-26T01:28:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.776+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 shutdown_connections 2024-04-26T01:28:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.776+0000 7f31a9200700 1 --2- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3188039650 0x7f318803bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.776+0000 7f31a9200700 1 --2- 172.21.15.28:0/2375201700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a4102f30 0x7f31a4199c70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.776+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 >> 172.21.15.28:0/2375201700 conn(0x7f31a40fc940 msgr2=0x7f31a40fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.776+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 shutdown_connections 2024-04-26T01:28:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:42.776+0000 7f31a9200700 1 -- 172.21.15.28:0/2375201700 wait complete. 2024-04-26T01:28:43.589 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi053 2024-04-26T01:28:43.589 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:28:43.589 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/etc/ceph/ceph.conf 2024-04-26T01:28:43.608 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:28:43.608 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:28:43.666 INFO:tasks.cephadm:Adding host smithi053 to orchestrator... 2024-04-26T01:28:43.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch host add smithi053 2024-04-26T01:28:44.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:43 smithi028 ceph-mon[28247]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:28:44.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:44.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:28:44.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:28:44.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:28:44.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:43 smithi028 ceph-mon[28247]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-26T01:28:44.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:43 smithi028 ceph-mon[28247]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:28:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:28:45.001 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:44 smithi028 ceph-mon[28247]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:45.001 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:44 smithi028 ceph-mon[28247]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:28:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.157+0000 7f61a2400700 1 -- 172.21.15.28:0/842895809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 msgr2=0x7f619c1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.157+0000 7f61a2400700 1 --2- 172.21.15.28:0/842895809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 0x7f619c1088c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f618c00b0a0 tx=0x7f618c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:28:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.157+0000 7f61a2400700 1 -- 172.21.15.28:0/842895809 shutdown_connections 2024-04-26T01:28:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.157+0000 7f61a2400700 1 --2- 172.21.15.28:0/842895809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 0x7f619c1088c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.157+0000 7f61a2400700 1 -- 172.21.15.28:0/842895809 >> 172.21.15.28:0/842895809 conn(0x7f619c076ea0 msgr2=0x7f619c0772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.158+0000 7f61a2400700 1 -- 172.21.15.28:0/842895809 shutdown_connections 2024-04-26T01:28:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.158+0000 7f61a2400700 1 -- 172.21.15.28:0/842895809 wait complete. 2024-04-26T01:28:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.159+0000 7f61a2400700 1 Processor -- start 2024-04-26T01:28:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.159+0000 7f61a2400700 1 -- start start 2024-04-26T01:28:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.159+0000 7f61a2400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 0x7f619c19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.159+0000 7f61a2400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f619c10c920 con 0x7f619c1084e0 2024-04-26T01:28:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.160+0000 7f619b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 0x7f619c19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.160+0000 7f619b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 0x7f619c19a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58402/0 (socket says 172.21.15.28:58402) 2024-04-26T01:28:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.160+0000 7f619b400700 1 -- 172.21.15.28:0/2296674112 learned_addr learned my addr 172.21.15.28:0/2296674112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.160+0000 7f619b400700 1 -- 172.21.15.28:0/2296674112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c19a560 con 0x7f619c1084e0 2024-04-26T01:28:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.160+0000 7f619b400700 1 --2- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 0x7f619c19a020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f618c006130 tx=0x7f618c0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.161+0000 7f6198c00700 1 -- 172.21.15.28:0/2296674112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f618c021020 con 0x7f619c1084e0 2024-04-26T01:28:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.161+0000 7f6198c00700 1 -- 172.21.15.28:0/2296674112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f618c0079b0 con 0x7f619c1084e0 2024-04-26T01:28:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.161+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619c19a7f0 con 0x7f619c1084e0 2024-04-26T01:28:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.161+0000 7f6198c00700 1 -- 172.21.15.28:0/2296674112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f618c017040 con 0x7f619c1084e0 2024-04-26T01:28:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.161+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f619c19e8f0 con 0x7f619c1084e0 2024-04-26T01:28:46.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.162+0000 7f6198c00700 1 -- 172.21.15.28:0/2296674112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f618c024020 con 0x7f619c1084e0 2024-04-26T01:28:46.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.162+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f619c103b90 con 0x7f619c1084e0 2024-04-26T01:28:46.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.163+0000 7f6198c00700 1 --2- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6180039650 0x7f618003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:46.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.163+0000 7f6198c00700 1 -- 172.21.15.28:0/2296674112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f618c00f050 con 0x7f619c1084e0 2024-04-26T01:28:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.163+0000 7f6192a00700 1 --2- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6180039650 0x7f618003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.164+0000 7f6192a00700 1 --2- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6180039650 0x7f618003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6184006fd0 tx=0x7f6184006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:46.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.169+0000 7f6198c00700 1 -- 172.21.15.28:0/2296674112 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f618c02b020 con 0x7f619c1084e0 2024-04-26T01:28:46.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:46.405+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi053", "target": ["mon-mgr", ""]}) -- 0x7f619c19b380 con 0x7f6180039650 2024-04-26T01:28:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:46 smithi028 ceph-mon[28247]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:28:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:46 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:46 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:46 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:47.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:47 smithi028 ceph-mon[28247]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:47.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:47 smithi028 ceph-mon[28247]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:28:48.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:48 smithi028 ceph-mon[28247]: Deploying cephadm binary to smithi053 2024-04-26T01:28:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:49 smithi028 ceph-mon[28247]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:50.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:50 smithi028 ceph-mon[28247]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:51.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.297+0000 7f6198c00700 1 -- 172.21.15.28:0/2296674112 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f619c19b380 con 0x7f6180039650 2024-04-26T01:28:51.294 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi053' with addr '172.21.15.53' 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.301+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6180039650 msgr2=0x7f618003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.301+0000 7f61a2400700 1 --2- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6180039650 0x7f618003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6184006fd0 tx=0x7f6184006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.301+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 msgr2=0x7f619c19a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.301+0000 7f61a2400700 1 --2- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 0x7f619c19a020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f618c006130 tx=0x7f618c0097d0 comp rx=0 tx=0).stop 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.302+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 shutdown_connections 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.302+0000 7f61a2400700 1 --2- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6180039650 0x7f618003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.302+0000 7f61a2400700 1 --2- 172.21.15.28:0/2296674112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c1084e0 0x7f619c19a020 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.302+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 >> 172.21.15.28:0/2296674112 conn(0x7f619c076ea0 msgr2=0x7f619c104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.302+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 shutdown_connections 2024-04-26T01:28:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:51.302+0000 7f61a2400700 1 -- 172.21.15.28:0/2296674112 wait complete. 2024-04-26T01:28:52.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T01:28:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:28:52.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:52 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:52.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:52 smithi028 ceph-mon[28247]: Added host smithi053 2024-04-26T01:28:52.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:52 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:28:52.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:52 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:53.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:53 smithi028 ceph-mon[28247]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:53.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:53 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.694+0000 7f7bc2a00700 1 -- 172.21.15.28:0/1596871895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 msgr2=0x7f7bbc075250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.694+0000 7f7bc2a00700 1 --2- 172.21.15.28:0/1596871895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 0x7f7bbc075250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009910 tx=0x7f7bac009c20 comp rx=0 tx=0).stop 2024-04-26T01:28:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.694+0000 7f7bc2a00700 1 -- 172.21.15.28:0/1596871895 shutdown_connections 2024-04-26T01:28:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.694+0000 7f7bc2a00700 1 --2- 172.21.15.28:0/1596871895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 0x7f7bbc075250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.694+0000 7f7bc2a00700 1 -- 172.21.15.28:0/1596871895 >> 172.21.15.28:0/1596871895 conn(0x7f7bbc0fdf80 msgr2=0x7f7bbc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.695+0000 7f7bc2a00700 1 -- 172.21.15.28:0/1596871895 shutdown_connections 2024-04-26T01:28:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.695+0000 7f7bc2a00700 1 -- 172.21.15.28:0/1596871895 wait complete. 2024-04-26T01:28:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.696+0000 7f7bc2a00700 1 Processor -- start 2024-04-26T01:28:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.696+0000 7f7bc2a00700 1 -- start start 2024-04-26T01:28:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.696+0000 7f7bc2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 0x7f7bbc19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.696+0000 7f7bc2a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10ebb0 con 0x7f7bbc074e70 2024-04-26T01:28:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.697+0000 7f7bbbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 0x7f7bbc19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.697+0000 7f7bbbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 0x7f7bbc19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60400/0 (socket says 172.21.15.28:60400) 2024-04-26T01:28:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.697+0000 7f7bbbe00700 1 -- 172.21.15.28:0/4182528619 learned_addr learned my addr 172.21.15.28:0/4182528619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.697+0000 7f7bbbe00700 1 -- 172.21.15.28:0/4182528619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc19a650 con 0x7f7bbc074e70 2024-04-26T01:28:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.697+0000 7f7bbbe00700 1 --2- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 0x7f7bbc19a110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7bac006010 tx=0x7f7bac005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.698+0000 7f7bb8c00700 1 -- 172.21.15.28:0/4182528619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bac026070 con 0x7f7bbc074e70 2024-04-26T01:28:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.698+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bbc19a8e0 con 0x7f7bbc074e70 2024-04-26T01:28:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.698+0000 7f7bb8c00700 1 -- 172.21.15.28:0/4182528619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bac00bc00 con 0x7f7bbc074e70 2024-04-26T01:28:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.698+0000 7f7bb8c00700 1 -- 172.21.15.28:0/4182528619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bac003910 con 0x7f7bbc074e70 2024-04-26T01:28:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.698+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bbc19ad40 con 0x7f7bbc074e70 2024-04-26T01:28:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.699+0000 7f7bb8c00700 1 -- 172.21.15.28:0/4182528619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f7bac029070 con 0x7f7bbc074e70 2024-04-26T01:28:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.699+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bbc076520 con 0x7f7bbc074e70 2024-04-26T01:28:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.700+0000 7f7bb8c00700 1 --2- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ba00396b0 0x7f7ba003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.700+0000 7f7bb8c00700 1 -- 172.21.15.28:0/4182528619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7bac01e070 con 0x7f7bbc074e70 2024-04-26T01:28:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.700+0000 7f7bbb400700 1 --2- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ba00396b0 0x7f7ba003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.701+0000 7f7bbb400700 1 --2- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ba00396b0 0x7f7ba003bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8006fd0 tx=0x7f7ba8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.706+0000 7f7bb8c00700 1 -- 172.21.15.28:0/4182528619 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7bac013070 con 0x7f7bbc074e70 2024-04-26T01:28:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.944+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7bbc19b4e0 con 0x7f7ba00396b0 2024-04-26T01:28:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.947+0000 7f7bb8c00700 1 -- 172.21.15.28:0/4182528619 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f7bbc19b4e0 con 0x7f7ba00396b0 2024-04-26T01:28:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:28:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}] 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.950+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ba00396b0 msgr2=0x7f7ba003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.950+0000 7f7bc2a00700 1 --2- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ba00396b0 0x7f7ba003bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8006fd0 tx=0x7f7ba8006e40 comp rx=0 tx=0).stop 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.950+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 msgr2=0x7f7bbc19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.950+0000 7f7bc2a00700 1 --2- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 0x7f7bbc19a110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7bac006010 tx=0x7f7bac005780 comp rx=0 tx=0).stop 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.950+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 shutdown_connections 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.950+0000 7f7bc2a00700 1 --2- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ba00396b0 0x7f7ba003bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.950+0000 7f7bc2a00700 1 --2- 172.21.15.28:0/4182528619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc074e70 0x7f7bbc19a110 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.951+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 >> 172.21.15.28:0/4182528619 conn(0x7f7bbc0fdf80 msgr2=0x7f7bbc1066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.951+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 shutdown_connections 2024-04-26T01:28:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:54.951+0000 7f7bc2a00700 1 -- 172.21.15.28:0/4182528619 wait complete. 2024-04-26T01:28:55.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:55 smithi028 ceph-mon[28247]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:55.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:55.709 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi169 2024-04-26T01:28:55.709 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:28:55.709 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.conf 2024-04-26T01:28:55.728 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:28:55.728 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:28:55.787 INFO:tasks.cephadm:Adding host smithi169 to orchestrator... 2024-04-26T01:28:55.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch host add smithi169 2024-04-26T01:28:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:28:56.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:56 smithi028 ceph-mon[28247]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:28:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:57 smithi028 ceph-mon[28247]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.195+0000 7fb923600700 1 -- 172.21.15.28:0/1442678631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 msgr2=0x7fb91c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:28:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.195+0000 7fb923600700 1 --2- 172.21.15.28:0/1442678631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 0x7fb91c10dd40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb91000b0a0 tx=0x7fb91000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:28:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.195+0000 7fb923600700 1 -- 172.21.15.28:0/1442678631 shutdown_connections 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.195+0000 7fb923600700 1 --2- 172.21.15.28:0/1442678631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 0x7fb91c10dd40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.195+0000 7fb923600700 1 -- 172.21.15.28:0/1442678631 >> 172.21.15.28:0/1442678631 conn(0x7fb91c0781e0 msgr2=0x7fb91c07a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.195+0000 7fb923600700 1 -- 172.21.15.28:0/1442678631 shutdown_connections 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.195+0000 7fb923600700 1 -- 172.21.15.28:0/1442678631 wait complete. 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.196+0000 7fb923600700 1 Processor -- start 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.197+0000 7fb923600700 1 -- start start 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.197+0000 7fb923600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 0x7fb91c19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.197+0000 7fb923600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c112f60 con 0x7fb91c10d960 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.197+0000 7fb920c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 0x7fb91c19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.197+0000 7fb920c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 0x7fb91c19e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60410/0 (socket says 172.21.15.28:60410) 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.197+0000 7fb920c00700 1 -- 172.21.15.28:0/3974993669 learned_addr learned my addr 172.21.15.28:0/3974993669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.198+0000 7fb920c00700 1 -- 172.21.15.28:0/3974993669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91c19e8f0 con 0x7fb91c10d960 2024-04-26T01:28:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.198+0000 7fb920c00700 1 --2- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 0x7fb91c19e3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d010 tx=0x7fb910009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.198+0000 7fb919600700 1 -- 172.21.15.28:0/3974993669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb910021020 con 0x7fb91c10d960 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.198+0000 7fb919600700 1 -- 172.21.15.28:0/3974993669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb910004000 con 0x7fb91c10d960 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.198+0000 7fb919600700 1 -- 172.21.15.28:0/3974993669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb910017040 con 0x7fb91c10d960 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.198+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91c19eb80 con 0x7fb91c10d960 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.199+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb91c1a2c80 con 0x7fb91c10d960 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.200+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb91c10a180 con 0x7fb91c10d960 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.200+0000 7fb919600700 1 -- 172.21.15.28:0/3974993669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb910024020 con 0x7fb91c10d960 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.200+0000 7fb919600700 1 --2- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb9040396f0 0x7fb90403bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.201+0000 7fb919600700 1 -- 172.21.15.28:0/3974993669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb91000f050 con 0x7fb91c10d960 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.201+0000 7fb91be00700 1 --2- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb9040396f0 0x7fb90403bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:28:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.202+0000 7fb91be00700 1 --2- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb9040396f0 0x7fb90403bbb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb908006fd0 tx=0x7fb908006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:28:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.206+0000 7fb919600700 1 -- 172.21.15.28:0/3974993669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb91002b020 con 0x7fb91c10d960 2024-04-26T01:28:58.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:58.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:58.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:58.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:58.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:28:58.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:28:58.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:28:58.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:28:58.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:58 smithi028 ceph-mon[28247]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:28:58.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:28:58.446+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}) -- 0x7fb91c0640a0 con 0x7fb9040396f0 2024-04-26T01:28:59.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:59 smithi028 ceph-mon[28247]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:28:59.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:59 smithi028 ceph-mon[28247]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:28:59.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:59 smithi028 ceph-mon[28247]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:28:59.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:59 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:59.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:59 smithi028 ceph-mon[28247]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:28:59.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:59 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:28:59.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:28:59 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:00.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:00 smithi028 ceph-mon[28247]: Deploying cephadm binary to smithi169 2024-04-26T01:29:01.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:01 smithi028 ceph-mon[28247]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:03.374 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi169' with addr '172.21.15.169' 2024-04-26T01:29:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.379+0000 7fb919600700 1 -- 172.21.15.28:0/3974993669 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb91c0640a0 con 0x7fb9040396f0 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb9040396f0 msgr2=0x7fb90403bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 --2- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb9040396f0 0x7fb90403bbb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb908006fd0 tx=0x7fb908006e40 comp rx=0 tx=0).stop 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 msgr2=0x7fb91c19e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 --2- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 0x7fb91c19e3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d010 tx=0x7fb910009790 comp rx=0 tx=0).stop 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 shutdown_connections 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 --2- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb9040396f0 0x7fb90403bbb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 --2- 172.21.15.28:0/3974993669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91c10d960 0x7fb91c19e3b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 >> 172.21.15.28:0/3974993669 conn(0x7fb91c0781e0 msgr2=0x7fb91c1104f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 shutdown_connections 2024-04-26T01:29:03.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:03.383+0000 7fb923600700 1 -- 172.21.15.28:0/3974993669 wait complete. 2024-04-26T01:29:03.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:03 smithi028 ceph-mon[28247]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:03.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:03 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:03.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:03 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:04.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T01:29:04.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:04 smithi028 ceph-mon[28247]: Added host smithi169 2024-04-26T01:29:04.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:04 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:29:05.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:05 smithi028 ceph-mon[28247]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:05.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:05 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.707+0000 7fd32e400700 1 -- 172.21.15.28:0/1793523214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 msgr2=0x7fd3280751c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.707+0000 7fd32e400700 1 --2- 172.21.15.28:0/1793523214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 0x7fd3280751c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd318009910 tx=0x7fd318009c20 comp rx=0 tx=0).stop 2024-04-26T01:29:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.707+0000 7fd32e400700 1 -- 172.21.15.28:0/1793523214 shutdown_connections 2024-04-26T01:29:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.707+0000 7fd32e400700 1 --2- 172.21.15.28:0/1793523214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 0x7fd3280751c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.707+0000 7fd32e400700 1 -- 172.21.15.28:0/1793523214 >> 172.21.15.28:0/1793523214 conn(0x7fd3280fdf80 msgr2=0x7fd3281003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.707+0000 7fd32e400700 1 -- 172.21.15.28:0/1793523214 shutdown_connections 2024-04-26T01:29:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.707+0000 7fd32e400700 1 -- 172.21.15.28:0/1793523214 wait complete. 2024-04-26T01:29:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.708+0000 7fd32e400700 1 Processor -- start 2024-04-26T01:29:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.708+0000 7fd32e400700 1 -- start start 2024-04-26T01:29:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.709+0000 7fd32e400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 0x7fd32819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.709+0000 7fd32e400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd328112fa0 con 0x7fd328074de0 2024-04-26T01:29:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.709+0000 7fd327400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 0x7fd32819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.709+0000 7fd327400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 0x7fd32819e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53188/0 (socket says 172.21.15.28:53188) 2024-04-26T01:29:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.709+0000 7fd327400700 1 -- 172.21.15.28:0/1432549802 learned_addr learned my addr 172.21.15.28:0/1432549802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:29:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.709+0000 7fd327400700 1 -- 172.21.15.28:0/1432549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd32819ea80 con 0x7fd328074de0 2024-04-26T01:29:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.709+0000 7fd327400700 1 --2- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 0x7fd32819e540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd318006010 tx=0x7fd318005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.710+0000 7fd31fe00700 1 -- 172.21.15.28:0/1432549802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd318026070 con 0x7fd328074de0 2024-04-26T01:29:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.710+0000 7fd31fe00700 1 -- 172.21.15.28:0/1432549802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd31800bc00 con 0x7fd328074de0 2024-04-26T01:29:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.710+0000 7fd31fe00700 1 -- 172.21.15.28:0/1432549802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd318003910 con 0x7fd328074de0 2024-04-26T01:29:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.710+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd32819ed10 con 0x7fd328074de0 2024-04-26T01:29:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.710+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd32819f170 con 0x7fd328074de0 2024-04-26T01:29:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.711+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd328076520 con 0x7fd328074de0 2024-04-26T01:29:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.711+0000 7fd31fe00700 1 -- 172.21.15.28:0/1432549802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd318029070 con 0x7fd328074de0 2024-04-26T01:29:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.712+0000 7fd31fe00700 1 --2- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd30c0396b0 0x7fd30c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.712+0000 7fd31fe00700 1 -- 172.21.15.28:0/1432549802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd31801e070 con 0x7fd328074de0 2024-04-26T01:29:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.712+0000 7fd326a00700 1 --2- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd30c0396b0 0x7fd30c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.713+0000 7fd326a00700 1 --2- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd30c0396b0 0x7fd30c03bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd314006fd0 tx=0x7fd314006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:06.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.718+0000 7fd31fe00700 1 -- 172.21.15.28:0/1432549802 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd318013070 con 0x7fd328074de0 2024-04-26T01:29:06.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.955+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd32819f8a0 con 0x7fd30c0396b0 2024-04-26T01:29:06.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.957+0000 7fd31fe00700 1 -- 172.21.15.28:0/1432549802 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fd32819f8a0 con 0x7fd30c0396b0 2024-04-26T01:29:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:29:06.953 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}] 2024-04-26T01:29:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.960+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd30c0396b0 msgr2=0x7fd30c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.960+0000 7fd32e400700 1 --2- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd30c0396b0 0x7fd30c03bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd314006fd0 tx=0x7fd314006e40 comp rx=0 tx=0).stop 2024-04-26T01:29:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.960+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 msgr2=0x7fd32819e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.960+0000 7fd32e400700 1 --2- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 0x7fd32819e540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd318006010 tx=0x7fd318005780 comp rx=0 tx=0).stop 2024-04-26T01:29:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.961+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 shutdown_connections 2024-04-26T01:29:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.961+0000 7fd32e400700 1 --2- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd30c0396b0 0x7fd30c03bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.961+0000 7fd32e400700 1 --2- 172.21.15.28:0/1432549802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328074de0 0x7fd32819e540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.961+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 >> 172.21.15.28:0/1432549802 conn(0x7fd3280fdf80 msgr2=0x7fd3281041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.961+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 shutdown_connections 2024-04-26T01:29:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:06.961+0000 7fd32e400700 1 -- 172.21.15.28:0/1432549802 wait complete. 2024-04-26T01:29:07.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:07 smithi028 ceph-mon[28247]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:07.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:07 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:07.692 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T01:29:07.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T01:29:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:29:08.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:08 smithi028 ceph-mon[28247]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:29:09.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:09 smithi028 ceph-mon[28247]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:10.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.180+0000 7f8bb6600700 1 -- 172.21.15.28:0/2777112133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 msgr2=0x7f8bb00751c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:10.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.180+0000 7f8bb6600700 1 --2- 172.21.15.28:0/2777112133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 0x7f8bb00751c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0009910 tx=0x7f8ba0009c20 comp rx=0 tx=0).stop 2024-04-26T01:29:10.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.180+0000 7f8bb6600700 1 -- 172.21.15.28:0/2777112133 shutdown_connections 2024-04-26T01:29:10.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.180+0000 7f8bb6600700 1 --2- 172.21.15.28:0/2777112133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 0x7f8bb00751c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:10.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.180+0000 7f8bb6600700 1 -- 172.21.15.28:0/2777112133 >> 172.21.15.28:0/2777112133 conn(0x7f8bb00fdf60 msgr2=0x7f8bb0100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:10.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.180+0000 7f8bb6600700 1 -- 172.21.15.28:0/2777112133 shutdown_connections 2024-04-26T01:29:10.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.180+0000 7f8bb6600700 1 -- 172.21.15.28:0/2777112133 wait complete. 2024-04-26T01:29:10.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.181+0000 7f8bb6600700 1 Processor -- start 2024-04-26T01:29:10.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.182+0000 7f8bb6600700 1 -- start start 2024-04-26T01:29:10.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.182+0000 7f8bb6600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 0x7f8bb019e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:10.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.182+0000 7f8bb6600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb0112fa0 con 0x7f8bb0074de0 2024-04-26T01:29:10.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.182+0000 7f8baf400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 0x7f8bb019e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:10.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.182+0000 7f8baf400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 0x7f8bb019e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42932/0 (socket says 172.21.15.28:42932) 2024-04-26T01:29:10.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.182+0000 7f8baf400700 1 -- 172.21.15.28:0/2183329858 learned_addr learned my addr 172.21.15.28:0/2183329858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:29:10.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.183+0000 7f8baf400700 1 -- 172.21.15.28:0/2183329858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb019e9f0 con 0x7f8bb0074de0 2024-04-26T01:29:10.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.183+0000 7f8baf400700 1 --2- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 0x7f8bb019e4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0006010 tx=0x7f8ba0005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:10.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.183+0000 7f8ba7e00700 1 -- 172.21.15.28:0/2183329858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ba0026070 con 0x7f8bb0074de0 2024-04-26T01:29:10.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.184+0000 7f8ba7e00700 1 -- 172.21.15.28:0/2183329858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ba000bc00 con 0x7f8bb0074de0 2024-04-26T01:29:10.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.184+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb019ec80 con 0x7f8bb0074de0 2024-04-26T01:29:10.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.184+0000 7f8ba7e00700 1 -- 172.21.15.28:0/2183329858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ba0003910 con 0x7f8bb0074de0 2024-04-26T01:29:10.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.184+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb019f0e0 con 0x7f8bb0074de0 2024-04-26T01:29:10.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.185+0000 7f8ba7e00700 1 -- 172.21.15.28:0/2183329858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8ba0029070 con 0x7f8bb0074de0 2024-04-26T01:29:10.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.185+0000 7f8ba7e00700 1 --2- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8b940396b0 0x7f8b9403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:10.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.185+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb0076520 con 0x7f8bb0074de0 2024-04-26T01:29:10.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.185+0000 7f8ba7e00700 1 -- 172.21.15.28:0/2183329858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8ba001e070 con 0x7f8bb0074de0 2024-04-26T01:29:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.187+0000 7f8baea00700 1 --2- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8b940396b0 0x7f8b9403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.188+0000 7f8baea00700 1 --2- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8b940396b0 0x7f8b9403bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c006fd0 tx=0x7f8b9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.192+0000 7f8ba7e00700 1 -- 172.21.15.28:0/2183329858 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ba0030070 con 0x7f8bb0074de0 2024-04-26T01:29:10.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:10.431+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f8bb019f810 con 0x7f8bb0074de0 2024-04-26T01:29:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:29:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:10 smithi028 ceph-mon[28247]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:29:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.323+0000 7f8ba7e00700 1 -- 172.21.15.28:0/2183329858 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f8ba0013070 con 0x7f8bb0074de0 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8b940396b0 msgr2=0x7f8b9403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 --2- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8b940396b0 0x7f8b9403bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c006fd0 tx=0x7f8b9c006e40 comp rx=0 tx=0).stop 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 msgr2=0x7f8bb019e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 --2- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 0x7f8bb019e4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0006010 tx=0x7f8ba0005780 comp rx=0 tx=0).stop 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 shutdown_connections 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 --2- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8b940396b0 0x7f8b9403bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 --2- 172.21.15.28:0/2183329858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb0074de0 0x7f8bb019e4b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 >> 172.21.15.28:0/2183329858 conn(0x7f8bb00fdf60 msgr2=0x7f8bb01041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 shutdown_connections 2024-04-26T01:29:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:11.328+0000 7f8bb6600700 1 -- 172.21.15.28:0/2183329858 wait complete. 2024-04-26T01:29:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-04-26T01:29:11.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:11 smithi028 ceph-mon[28247]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:11.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:11 smithi028 ceph-mon[28247]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:11.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:11 smithi028 ceph-mon[28247]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:29:11.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:11 smithi028 ceph-mon[28247]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:29:11.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:11 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2183329858' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T01:29:11.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:11.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:11.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:12.081 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-04-26T01:29:12.081 INFO:tasks.cephadm:Adding mon.b on smithi053 2024-04-26T01:29:12.081 INFO:tasks.cephadm:Adding mon.c on smithi169 2024-04-26T01:29:12.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c' 2024-04-26T01:29:12.553 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:12.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:12 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2183329858' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T01:29:12.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:12 smithi028 ceph-mon[28247]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:13.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:13 smithi028 ceph-mon[28247]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:13.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.952+0000 7fbf627af700 1 -- 172.21.15.169:0/1892684319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 msgr2=0x7fbf5c1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:13.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.952+0000 7fbf627af700 1 --2- 172.21.15.169:0/1892684319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 0x7fbf5c1088c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009910 tx=0x7fbf4c009c20 comp rx=0 tx=0).stop 2024-04-26T01:29:13.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.952+0000 7fbf627af700 1 -- 172.21.15.169:0/1892684319 shutdown_connections 2024-04-26T01:29:13.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.952+0000 7fbf627af700 1 --2- 172.21.15.169:0/1892684319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 0x7fbf5c1088c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:13.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.952+0000 7fbf627af700 1 -- 172.21.15.169:0/1892684319 >> 172.21.15.169:0/1892684319 conn(0x7fbf5c0fdf80 msgr2=0x7fbf5c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:13.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.952+0000 7fbf627af700 1 -- 172.21.15.169:0/1892684319 shutdown_connections 2024-04-26T01:29:13.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.952+0000 7fbf627af700 1 -- 172.21.15.169:0/1892684319 wait complete. 2024-04-26T01:29:13.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.953+0000 7fbf627af700 1 Processor -- start 2024-04-26T01:29:13.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.953+0000 7fbf627af700 1 -- start start 2024-04-26T01:29:13.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.953+0000 7fbf627af700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 0x7fbf5c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:13.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.953+0000 7fbf627af700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c10c940 con 0x7fbf5c1084e0 2024-04-26T01:29:13.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.954+0000 7fbf5bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 0x7fbf5c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:13.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.954+0000 7fbf5bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 0x7fbf5c197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.169:37034/0 (socket says 172.21.15.169:37034) 2024-04-26T01:29:13.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.954+0000 7fbf5bfff700 1 -- 172.21.15.169:0/772715099 learned_addr learned my addr 172.21.15.169:0/772715099 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:29:13.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.954+0000 7fbf5bfff700 1 -- 172.21.15.169:0/772715099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5c1983f0 con 0x7fbf5c1084e0 2024-04-26T01:29:13.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.955+0000 7fbf5bfff700 1 --2- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 0x7fbf5c197eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0056d0 tx=0x7fbf4c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:13.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.955+0000 7fbf597fa700 1 -- 172.21.15.169:0/772715099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf4c026070 con 0x7fbf5c1084e0 2024-04-26T01:29:13.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.955+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5c1986e0 con 0x7fbf5c1084e0 2024-04-26T01:29:13.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.955+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf5c19c780 con 0x7fbf5c1084e0 2024-04-26T01:29:13.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.955+0000 7fbf597fa700 1 -- 172.21.15.169:0/772715099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf4c00bba0 con 0x7fbf5c1084e0 2024-04-26T01:29:13.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.955+0000 7fbf597fa700 1 -- 172.21.15.169:0/772715099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf4c030060 con 0x7fbf5c1084e0 2024-04-26T01:29:13.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.956+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf48005330 con 0x7fbf5c1084e0 2024-04-26T01:29:13.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.957+0000 7fbf597fa700 1 -- 172.21.15.169:0/772715099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbf4c029070 con 0x7fbf5c1084e0 2024-04-26T01:29:13.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.957+0000 7fbf597fa700 1 --2- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbf44039650 0x7fbf4403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:13.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.957+0000 7fbf597fa700 1 -- 172.21.15.169:0/772715099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbf4c01e070 con 0x7fbf5c1084e0 2024-04-26T01:29:13.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.957+0000 7fbf5b7fe700 1 --2- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbf44039650 0x7fbf4403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:13.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.958+0000 7fbf5b7fe700 1 --2- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbf44039650 0x7fbf4403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbf50006fd0 tx=0x7fbf50006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:13.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:13.973+0000 7fbf597fa700 1 -- 172.21.15.169:0/772715099 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf4c013070 con 0x7fbf5c1084e0 2024-04-26T01:29:14.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.184+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c", "target": ["mon-mgr", ""]}) -- 0x7fbf48000c80 con 0x7fbf44039650 2024-04-26T01:29:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.198+0000 7fbf597fa700 1 -- 172.21.15.169:0/772715099 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbf48000c80 con 0x7fbf44039650 2024-04-26T01:29:14.194 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mon update... 2024-04-26T01:29:14.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbf44039650 msgr2=0x7fbf4403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:14.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 --2- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbf44039650 0x7fbf4403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbf50006fd0 tx=0x7fbf50006e40 comp rx=0 tx=0).stop 2024-04-26T01:29:14.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 msgr2=0x7fbf5c197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:14.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 --2- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 0x7fbf5c197eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0056d0 tx=0x7fbf4c0057b0 comp rx=0 tx=0).stop 2024-04-26T01:29:14.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 shutdown_connections 2024-04-26T01:29:14.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 --2- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbf44039650 0x7fbf4403bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:14.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 --2- 172.21.15.169:0/772715099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf5c1084e0 0x7fbf5c197eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:14.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 >> 172.21.15.169:0/772715099 conn(0x7fbf5c0fdf80 msgr2=0x7fbf5c10a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:14.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 shutdown_connections 2024-04-26T01:29:14.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:14.201+0000 7fbf627af700 1 -- 172.21.15.169:0/772715099 wait complete. 2024-04-26T01:29:14.801 DEBUG:teuthology.orchestra.run.smithi053:mon.b> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T01:29:14.804 DEBUG:teuthology.orchestra.run.smithi169:mon.c> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T01:29:14.806 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:29:14.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:29:15.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:15.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:15.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c;count:3 2024-04-26T01:29:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:15 smithi028 ceph-mon[28247]: Deploying daemon mon.c on smithi169 2024-04-26T01:29:15.944 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:17 smithi028 ceph-mon[28247]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.357+0000 7f1d72ea8700 1 -- 172.21.15.169:0/613410969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 msgr2=0x7f1d6c101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.357+0000 7f1d72ea8700 1 --2- 172.21.15.169:0/613410969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 0x7f1d6c101990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009910 tx=0x7f1d60009c20 comp rx=0 tx=0).stop 2024-04-26T01:29:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.357+0000 7f1d72ea8700 1 -- 172.21.15.169:0/613410969 shutdown_connections 2024-04-26T01:29:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.357+0000 7f1d72ea8700 1 --2- 172.21.15.169:0/613410969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 0x7f1d6c101990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.357+0000 7f1d72ea8700 1 -- 172.21.15.169:0/613410969 >> 172.21.15.169:0/613410969 conn(0x7f1d6c0fce40 msgr2=0x7f1d6c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.357+0000 7f1d72ea8700 1 -- 172.21.15.169:0/613410969 shutdown_connections 2024-04-26T01:29:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.357+0000 7f1d72ea8700 1 -- 172.21.15.169:0/613410969 wait complete. 2024-04-26T01:29:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.358+0000 7f1d72ea8700 1 Processor -- start 2024-04-26T01:29:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.358+0000 7f1d72ea8700 1 -- start start 2024-04-26T01:29:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.358+0000 7f1d72ea8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 0x7f1d6c198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.358+0000 7f1d72ea8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6c10b830 con 0x7f1d6c1015b0 2024-04-26T01:29:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.359+0000 7f1d70c44700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 0x7f1d6c198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.359+0000 7f1d70c44700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 0x7f1d6c198d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.169:37060/0 (socket says 172.21.15.169:37060) 2024-04-26T01:29:18.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.359+0000 7f1d70c44700 1 -- 172.21.15.169:0/2295154596 learned_addr learned my addr 172.21.15.169:0/2295154596 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:29:18.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.359+0000 7f1d70c44700 1 -- 172.21.15.169:0/2295154596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6c199240 con 0x7f1d6c1015b0 2024-04-26T01:29:18.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.360+0000 7f1d70c44700 1 --2- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 0x7f1d6c198d00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1d60006010 tx=0x7f1d60005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:18.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.360+0000 7f1d69ffb700 1 -- 172.21.15.169:0/2295154596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d60026070 con 0x7f1d6c1015b0 2024-04-26T01:29:18.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.360+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d6c199530 con 0x7f1d6c1015b0 2024-04-26T01:29:18.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.360+0000 7f1d69ffb700 1 -- 172.21.15.169:0/2295154596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d6000bba0 con 0x7f1d6c1015b0 2024-04-26T01:29:18.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.360+0000 7f1d69ffb700 1 -- 172.21.15.169:0/2295154596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d60003910 con 0x7f1d6c1015b0 2024-04-26T01:29:18.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.361+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d6c1999f0 con 0x7f1d6c1015b0 2024-04-26T01:29:18.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.361+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d6c102a90 con 0x7f1d6c1015b0 2024-04-26T01:29:18.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.362+0000 7f1d69ffb700 1 -- 172.21.15.169:0/2295154596 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1d60029070 con 0x7f1d6c1015b0 2024-04-26T01:29:18.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.362+0000 7f1d69ffb700 1 --2- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d5c039650 0x7f1d5c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:18.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.362+0000 7f1d69ffb700 1 -- 172.21.15.169:0/2295154596 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1d6001e070 con 0x7f1d6c1015b0 2024-04-26T01:29:18.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.363+0000 7f1d6bfff700 1 --2- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d5c039650 0x7f1d5c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:18.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.364+0000 7f1d6bfff700 1 --2- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d5c039650 0x7f1d5c03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1d58006fd0 tx=0x7f1d58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:18.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.378+0000 7f1d69ffb700 1 -- 172.21.15.169:0/2295154596 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1d60030020 con 0x7f1d6c1015b0 2024-04-26T01:29:18.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.690+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1d6c0640a0 con 0x7f1d6c1015b0 2024-04-26T01:29:18.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.691+0000 7f1d69ffb700 1 -- 172.21.15.169:0/2295154596 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f1d60013070 con 0x7f1d6c1015b0 2024-04-26T01:29:18.687 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:29:18.687 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","modified":"2024-04-26T01:27:11.642338Z","created":"2024-04-26T01:27:11.642338Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T01:29:18.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.693+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d5c039650 msgr2=0x7f1d5c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:18.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.693+0000 7f1d72ea8700 1 --2- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d5c039650 0x7f1d5c03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1d58006fd0 tx=0x7f1d58006e40 comp rx=0 tx=0).stop 2024-04-26T01:29:18.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.693+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 msgr2=0x7f1d6c198d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:18.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.693+0000 7f1d72ea8700 1 --2- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 0x7f1d6c198d00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1d60006010 tx=0x7f1d60005780 comp rx=0 tx=0).stop 2024-04-26T01:29:18.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.694+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 shutdown_connections 2024-04-26T01:29:18.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.694+0000 7f1d72ea8700 1 --2- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d5c039650 0x7f1d5c03bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:18.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.694+0000 7f1d72ea8700 1 --2- 172.21.15.169:0/2295154596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d6c1015b0 0x7f1d6c198d00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:18.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.694+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 >> 172.21.15.169:0/2295154596 conn(0x7f1d6c0fce40 msgr2=0x7f1d6c102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:18.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.694+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 shutdown_connections 2024-04-26T01:29:18.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:18.694+0000 7f1d72ea8700 1 -- 172.21.15.169:0/2295154596 wait complete. 2024-04-26T01:29:18.690 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-04-26T01:29:18.999 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:18 smithi169 systemd[1]: Starting Ceph mon.c for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:29:19.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:19 smithi028 ceph-mon[28247]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:19 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/2295154596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:19.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 podman[32832]: 2024-04-26 01:29:19.424372062 +0000 UTC m=+0.018480973 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:29:19.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 podman[32832]: 2024-04-26 01:29:19.522540293 +0000 UTC m=+0.116649203 container create edd4d4fc4da948ab7eda8eeb57a818d66a21cc0ff0b9c22fefebcbe6cf4d6bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:29:19.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 podman[32832]: 2024-04-26 01:29:19.73557856 +0000 UTC m=+0.329687463 container init edd4d4fc4da948ab7eda8eeb57a818d66a21cc0ff0b9c22fefebcbe6cf4d6bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 podman[32832]: 2024-04-26 01:29:19.738711123 +0000 UTC m=+0.332820034 container start edd4d4fc4da948ab7eda8eeb57a818d66a21cc0ff0b9c22fefebcbe6cf4d6bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-c, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pidfile_write: ignore empty --pid-file 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: load: jerasure load: lrc 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: RocksDB version: 7.9.2 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Git sha 0 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: DB SUMMARY 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: DB Session ID: Z1XA5MS30V3TPW26BBAU 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: CURRENT file: CURRENT 2024-04-26T01:29:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.error_if_exists: 0 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.create_if_missing: 0 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.paranoid_checks: 1 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.env: 0x55e852b2dcc0 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.info_log: 0x55e85390fbe0 2024-04-26T01:29:19.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.statistics: (nil) 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.use_fsync: 0 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_log_file_size: 0 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.allow_fallocate: 1 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.use_direct_reads: 0 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T01:29:19.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.db_log_dir: 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.wal_dir: 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.write_buffer_manager: 0x55e8538a30e0 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T01:29:19.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.unordered_write: 0 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.row_cache: None 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.wal_filter: None 2024-04-26T01:29:19.916 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.two_write_queues: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.wal_compression: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.atomic_flush: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.log_readahead_size: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T01:29:19.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_background_jobs: 2 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_background_compactions: -1 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_subcompactions: 1 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T01:29:19.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_open_files: -1 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_background_flushes: -1 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Compression algorithms supported: 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: kZSTD supported: 0 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: kXpressCompression supported: 0 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T01:29:19.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: kZlibCompression supported: 1 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: kSnappyCompression supported: 1 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: kLZ4Compression supported: 1 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: kBZip2Compression supported: 0 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.merge_operator: 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_filter: None 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T01:29:19.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e85390fdc0) 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: cache_index_and_filter_blocks: 1 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: pin_top_level_index_and_filter: 1 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: index_type: 0 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: data_block_index_type: 0 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: index_shortening: 1 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: checksum: 4 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: no_block_cache: 0 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_cache: 0x55e8538d51f0 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_cache_name: BinnedLRUCache 2024-04-26T01:29:19.921 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_cache_options: 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: capacity : 536870912 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: num_shard_bits : 4 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: strict_capacity_limit : 0 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: high_pri_pool_ratio: 0.000 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_cache_compressed: (nil) 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: persistent_cache: (nil) 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_size: 4096 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_size_deviation: 10 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_restart_interval: 16 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: index_block_restart_interval: 1 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: metadata_block_size: 4096 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: partition_filters: 0 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: use_delta_encoding: 1 2024-04-26T01:29:19.922 INFO:journalctl@ceph.mon.c.smithi169.stdout: filter_policy: bloomfilter 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: whole_key_filtering: 1 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: verify_compression: 0 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: read_amp_bytes_per_bit: 0 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: format_version: 5 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: enable_index_compression: 1 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_align: 0 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: max_auto_readahead_size: 262144 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: prepopulate_block_cache: 0 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: initial_auto_readahead_size: 8192 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression: NoCompression 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T01:29:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.num_levels: 7 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:29:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T01:29:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T01:29:19.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T01:29:19.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T01:29:19.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T01:29:19.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T01:29:19.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T01:29:19.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T01:29:19.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T01:29:19.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T01:29:19.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T01:29:19.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T01:29:19.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T01:29:19.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.inplace_update_support: 0 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.bloom_locality: 0 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.max_successive_merges: 0 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T01:29:19.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T01:29:19.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T01:29:19.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T01:29:19.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.ttl: 2592000 2024-04-26T01:29:19.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T01:29:19.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.enable_blob_files: false 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.min_blob_size: 0 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 214e9b7c-6d52-4bbd-a8ed-41d1c0492eaf 2024-04-26T01:29:19.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094959769568, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094959770151, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714094959, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "214e9b7c-6d52-4bbd-a8ed-41d1c0492eaf", "db_session_id": "Z1XA5MS30V3TPW26BBAU", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094959770256, "job": 1, "event": "recovery_finished"} 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e8539ea000 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: DB pointer 0x55e8539d4000 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** DB Stats ** 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T01:29:19.932 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** Compaction Stats [default] ** 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** Compaction Stats [default] ** 2024-04-26T01:29:19.933 INFO:journalctl@ceph.mon.c.smithi169.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:29:19.934 INFO:journalctl@ceph.mon.c.smithi169.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout: Block cache BinnedLRUCache@0x55e8538d51f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: using public_addr v2:172.21.15.169:0/0 -> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: starting mon.c rank -1 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(???) e0 preinit fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).mds e1 new map 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout: e1 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:29:19.935 INFO:journalctl@ceph.mon.c.smithi169.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout: legacy client fscid: -1 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout: No filesystems configured 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Added host smithi053 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:29:19.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Deploying cephadm binary to smithi169 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Added host smithi169 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2183329858' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2183329858' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T01:29:19.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c;count:3 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: Deploying daemon mon.c on smithi169 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:19.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/2295154596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:19.942 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 ceph-mon[32851]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T01:29:19.942 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 bash[32832]: edd4d4fc4da948ab7eda8eeb57a818d66a21cc0ff0b9c22fefebcbe6cf4d6bcb 2024-04-26T01:29:19.942 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:19 smithi169 systemd[1]: Started Ceph mon.c for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:29:20.328 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:29:20.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:29:20.563 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:29:25.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.071+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/2257696863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 msgr2=0x7ff0cc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:25.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.071+0000 7ff0e8e5e700 1 --2- 172.21.15.169:0/2257696863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 0x7ff0cc005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff0e406ad30 tx=0x7ff0d400feb0 comp rx=0 tx=0).stop 2024-04-26T01:29:25.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.071+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/2257696863 shutdown_connections 2024-04-26T01:29:25.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.071+0000 7ff0e8e5e700 1 --2- 172.21.15.169:0/2257696863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 0x7ff0cc005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:25.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.071+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/2257696863 >> 172.21.15.169:0/2257696863 conn(0x7ff0e40fcf80 msgr2=0x7ff0e40ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:25.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.071+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/2257696863 shutdown_connections 2024-04-26T01:29:25.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.071+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/2257696863 wait complete. 2024-04-26T01:29:25.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.072+0000 7ff0e8e5e700 1 Processor -- start 2024-04-26T01:29:25.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.072+0000 7ff0e8e5e700 1 -- start start 2024-04-26T01:29:25.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e8e5e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 0x7ff0e4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:25.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e8e5e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 0x7ff0e419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e8e5e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e410a4a0 con 0x7ff0cc0035c0 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e8e5e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e410a320 con 0x7ff0e4198500 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e1d9b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 0x7ff0e419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e1d9b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 0x7ff0e419c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49364/0 (socket says 172.21.15.169:49364) 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e1d9b700 1 -- 172.21.15.169:0/540631076 learned_addr learned my addr 172.21.15.169:0/540631076 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e1d9b700 1 -- 172.21.15.169:0/540631076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 msgr2=0x7ff0e419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e1d9b700 1 -- 172.21.15.169:0/540631076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 msgr2=0x7ff0e419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e1d9b700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 0x7ff0e419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e1d9b700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 0x7ff0e419c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.073+0000 7ff0e259c700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 0x7ff0e4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.074+0000 7ff0e259c700 1 -- 172.21.15.169:0/540631076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 msgr2=0x7ff0e419c890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.074+0000 7ff0e259c700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 0x7ff0e419c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:25.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.074+0000 7ff0e259c700 1 -- 172.21.15.169:0/540631076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e4198b40 con 0x7ff0cc0035c0 2024-04-26T01:29:25.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.074+0000 7ff0e259c700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 0x7ff0e4197fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4010010 tx=0x7ff0d40050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:25.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.075+0000 7ff0d37fe700 1 -- 172.21.15.169:0/540631076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff0d4004aa0 con 0x7ff0cc0035c0 2024-04-26T01:29:25.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.075+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e419cdd0 con 0x7ff0cc0035c0 2024-04-26T01:29:25.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.075+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e419d330 con 0x7ff0cc0035c0 2024-04-26T01:29:25.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.075+0000 7ff0d37fe700 1 -- 172.21.15.169:0/540631076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0d4004030 con 0x7ff0cc0035c0 2024-04-26T01:29:25.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.075+0000 7ff0d37fe700 1 -- 172.21.15.169:0/540631076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff0d40119e0 con 0x7ff0cc0035c0 2024-04-26T01:29:25.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.076+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e406b530 con 0x7ff0cc0035c0 2024-04-26T01:29:25.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.076+0000 7ff0d37fe700 1 -- 172.21.15.169:0/540631076 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff0d4023070 con 0x7ff0cc0035c0 2024-04-26T01:29:25.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.077+0000 7ff0d37fe700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0c40393f0 0x7ff0c403b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:25.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.077+0000 7ff0d37fe700 1 -- 172.21.15.169:0/540631076 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff0d4020050 con 0x7ff0cc0035c0 2024-04-26T01:29:25.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.077+0000 7ff0e1d9b700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0c40393f0 0x7ff0c403b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:25.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.079+0000 7ff0e1d9b700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0c40393f0 0x7ff0c403b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009180 tx=0x7ff0d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:25.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.094+0000 7ff0d37fe700 1 -- 172.21.15.169:0/540631076 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0d4028020 con 0x7ff0cc0035c0 2024-04-26T01:29:25.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:25.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:29:25.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: mon.a calling monitor election 2024-04-26T01:29:25.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: mon.c calling monitor election 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: monmap epoch 2 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: last_changed 2024-04-26T01:29:19.804576+0000 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:25.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: min_mon_release 19 (squid) 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: election_strategy: 1 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: fsmap 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: mgrmap e13: x(active, since 61s) 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: overall HEALTH_OK 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:25.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:25.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:29:25.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: mon.a calling monitor election 2024-04-26T01:29:25.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: mon.c calling monitor election 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: monmap epoch 2 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: last_changed 2024-04-26T01:29:19.804576+0000 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:25.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: min_mon_release 19 (squid) 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: election_strategy: 1 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: fsmap 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: mgrmap e13: x(active, since 61s) 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: overall HEALTH_OK 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:25.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.377+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff0e419d5b0 con 0x7ff0cc0035c0 2024-04-26T01:29:25.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.378+0000 7ff0d37fe700 1 -- 172.21.15.169:0/540631076 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7ff0d4003ba0 con 0x7ff0cc0035c0 2024-04-26T01:29:25.373 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:29:25.373 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":2,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","modified":"2024-04-26T01:29:19.804576Z","created":"2024-04-26T01:27:11.642338Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0c40393f0 msgr2=0x7ff0c403b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0c40393f0 0x7ff0c403b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009180 tx=0x7ff0d8008040 comp rx=0 tx=0).stop 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 msgr2=0x7ff0e4197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 0x7ff0e4197fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4010010 tx=0x7ff0d40050d0 comp rx=0 tx=0).stop 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 shutdown_connections 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0c40393f0 0x7ff0c403b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0cc0035c0 0x7ff0e4197fc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 --2- 172.21.15.169:0/540631076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e4198500 0x7ff0e419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 >> 172.21.15.169:0/540631076 conn(0x7ff0e40fcf80 msgr2=0x7ff0e40fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 shutdown_connections 2024-04-26T01:29:25.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:25.380+0000 7ff0e8e5e700 1 -- 172.21.15.169:0/540631076 wait complete. 2024-04-26T01:29:25.377 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 2 2024-04-26T01:29:25.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:25 smithi169 ceph-mon[32851]: Deploying daemon mon.b on smithi053 2024-04-26T01:29:25.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:25 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/540631076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:25.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:26.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:25 smithi028 ceph-mon[28247]: Deploying daemon mon.b on smithi053 2024-04-26T01:29:26.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:25 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/540631076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:26.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:26.932 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:29:26.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:29:27.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:26 smithi169 ceph-mon[32851]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:27.175 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:29:27.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:26 smithi028 ceph-mon[28247]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:28.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.467+0000 7f5e3bd87700 1 -- 172.21.15.169:0/2060118472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 msgr2=0x7f5e34101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:28.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.467+0000 7f5e3bd87700 1 --2- 172.21.15.169:0/2060118472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 0x7f5e34101250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009910 tx=0x7f5e30009c20 comp rx=0 tx=0).stop 2024-04-26T01:29:28.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.467+0000 7f5e3bd87700 1 -- 172.21.15.169:0/2060118472 shutdown_connections 2024-04-26T01:29:28.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.467+0000 7f5e3bd87700 1 --2- 172.21.15.169:0/2060118472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 0x7f5e34101250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:28.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.467+0000 7f5e3bd87700 1 -- 172.21.15.169:0/2060118472 >> 172.21.15.169:0/2060118472 conn(0x7f5e340778c0 msgr2=0x7f5e34077cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.467+0000 7f5e3bd87700 1 -- 172.21.15.169:0/2060118472 shutdown_connections 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e3bd87700 1 -- 172.21.15.169:0/2060118472 wait complete. 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e3bd87700 1 Processor -- start 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e3bd87700 1 -- start start 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e3bd87700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 0x7f5e34101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e3bd87700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 0x7f5e341027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e3bd87700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3410ca00 con 0x7f5e34100e70 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e3bd87700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3410c880 con 0x7f5e34102330 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e39322700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 0x7f5e341027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e39322700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 0x7f5e341027b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49368/0 (socket says 172.21.15.169:49368) 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.468+0000 7f5e39322700 1 -- 172.21.15.169:0/3186524283 learned_addr learned my addr 172.21.15.169:0/3186524283 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.469+0000 7f5e39322700 1 -- 172.21.15.169:0/3186524283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 msgr2=0x7f5e341027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T01:29:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.469+0000 7f5e39322700 1 -- 172.21.15.169:0/3186524283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 msgr2=0x7f5e341027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T01:29:28.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.469+0000 7f5e39322700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 0x7f5e341027b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T01:29:28.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.469+0000 7f5e39322700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 0x7f5e341027b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:29:28.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.469+0000 7f5e39b23700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 0x7f5e34101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.470+0000 7f5e39b23700 1 -- 172.21.15.169:0/3186524283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 msgr2=0x7f5e341027b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.470+0000 7f5e39b23700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 0x7f5e341027b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.470+0000 7f5e39b23700 1 -- 172.21.15.169:0/3186524283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e34103cc0 con 0x7f5e34100e70 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.470+0000 7f5e39b23700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 0x7f5e34101df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e30014000 tx=0x7f5e30005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.470+0000 7f5e2affd700 1 -- 172.21.15.169:0/3186524283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5e30005dc0 con 0x7f5e34100e70 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.471+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e341a4330 con 0x7f5e34100e70 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.471+0000 7f5e2affd700 1 -- 172.21.15.169:0/3186524283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e3000b6f0 con 0x7f5e34100e70 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.471+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e341a4890 con 0x7f5e34100e70 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.471+0000 7f5e2affd700 1 -- 172.21.15.169:0/3186524283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5e300038f0 con 0x7f5e34100e70 2024-04-26T01:29:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.471+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e3406a110 con 0x7f5e34100e70 2024-04-26T01:29:28.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.472+0000 7f5e2affd700 1 -- 172.21.15.169:0/3186524283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5e30029070 con 0x7f5e34100e70 2024-04-26T01:29:28.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.473+0000 7f5e2affd700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5e200393f0 0x7f5e2003b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:28.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.473+0000 7f5e2affd700 1 -- 172.21.15.169:0/3186524283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5e3001e070 con 0x7f5e34100e70 2024-04-26T01:29:28.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.473+0000 7f5e39322700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5e200393f0 0x7f5e2003b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.474+0000 7f5e39322700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5e200393f0 0x7f5e2003b8b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009180 tx=0x7f5e24008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:28.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.489+0000 7f5e2affd700 1 -- 172.21.15.169:0/3186524283 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e30030020 con 0x7f5e34100e70 2024-04-26T01:29:28.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 podman[32573]: 2024-04-26 01:29:28.692086241 +0000 UTC m=+0.191746657 container create 2cf2123eb492f35494937a26e7ea11caac1d947d0255577099fd19e0ea4b81b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T01:29:28.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.770+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5e341a4c00 con 0x7f5e34100e70 2024-04-26T01:29:28.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.771+0000 7f5e2affd700 1 -- 172.21.15.169:0/3186524283 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f5e30013070 con 0x7f5e34100e70 2024-04-26T01:29:28.766 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:29:28.766 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":2,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","modified":"2024-04-26T01:29:19.804576Z","created":"2024-04-26T01:27:11.642338Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T01:29:28.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5e200393f0 msgr2=0x7f5e2003b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:28.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5e200393f0 0x7f5e2003b8b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009180 tx=0x7f5e24008040 comp rx=0 tx=0).stop 2024-04-26T01:29:28.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 msgr2=0x7f5e34101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:28.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 0x7f5e34101df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e30014000 tx=0x7f5e30005030 comp rx=0 tx=0).stop 2024-04-26T01:29:28.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 shutdown_connections 2024-04-26T01:29:28.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5e200393f0 0x7f5e2003b8b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:28.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e34100e70 0x7f5e34101df0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:28.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 --2- 172.21.15.169:0/3186524283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e34102330 0x7f5e341027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:28.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 >> 172.21.15.169:0/3186524283 conn(0x7f5e340778c0 msgr2=0x7f5e34104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:28.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 shutdown_connections 2024-04-26T01:29:28.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:28.773+0000 7f5e3bd87700 1 -- 172.21.15.169:0/3186524283 wait complete. 2024-04-26T01:29:28.769 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 2 2024-04-26T01:29:29.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:28 smithi169 ceph-mon[32851]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:28 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/3186524283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:29.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 podman[32573]: 2024-04-26 01:29:28.919582628 +0000 UTC m=+0.419243042 container init 2cf2123eb492f35494937a26e7ea11caac1d947d0255577099fd19e0ea4b81b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 podman[32573]: 2024-04-26 01:29:28.922650684 +0000 UTC m=+0.422311098 container start 2cf2123eb492f35494937a26e7ea11caac1d947d0255577099fd19e0ea4b81b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pidfile_write: ignore empty --pid-file 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: load: jerasure load: lrc 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: RocksDB version: 7.9.2 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Git sha 0 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-26T01:29:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: DB SUMMARY 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: DB Session ID: JTRKXOKR09GU6KXY3Z9I 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: CURRENT file: CURRENT 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.error_if_exists: 0 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.create_if_missing: 0 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.paranoid_checks: 1 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T01:29:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T01:29:29.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.env: 0x55a9a7595cc0 2024-04-26T01:29:29.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T01:29:29.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.info_log: 0x55a9a9c2bbe0 2024-04-26T01:29:29.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.statistics: (nil) 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.use_fsync: 0 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_log_file_size: 0 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.allow_fallocate: 1 2024-04-26T01:29:29.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.use_direct_reads: 0 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.db_log_dir: 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.wal_dir: 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T01:29:29.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.write_buffer_manager: 0x55a9a9bbf0e0 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T01:29:29.223 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.unordered_write: 0 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.row_cache: None 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.wal_filter: None 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.two_write_queues: 0 2024-04-26T01:29:29.224 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.wal_compression: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.atomic_flush: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.log_readahead_size: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T01:29:29.225 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_background_jobs: 2 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_background_compactions: -1 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_subcompactions: 1 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T01:29:29.226 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_open_files: -1 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_background_flushes: -1 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Compression algorithms supported: 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: kZSTD supported: 0 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: kXpressCompression supported: 0 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T01:29:29.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: kZlibCompression supported: 1 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: kSnappyCompression supported: 1 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: kLZ4Compression supported: 1 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: kBZip2Compression supported: 0 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.merge_operator: 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_filter: None 2024-04-26T01:29:29.228 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a9a9c2bdc0) 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout: cache_index_and_filter_blocks: 1 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout: pin_top_level_index_and_filter: 1 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_type: 0 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_block_index_type: 0 2024-04-26T01:29:29.229 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_shortening: 1 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: checksum: 4 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: no_block_cache: 0 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache: 0x55a9a9bf11f0 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_name: BinnedLRUCache 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_options: 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: capacity : 536870912 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: num_shard_bits : 4 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: strict_capacity_limit : 0 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: high_pri_pool_ratio: 0.000 2024-04-26T01:29:29.230 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_compressed: (nil) 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: persistent_cache: (nil) 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_size: 4096 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_size_deviation: 10 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_restart_interval: 16 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_block_restart_interval: 1 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: metadata_block_size: 4096 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: partition_filters: 0 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: use_delta_encoding: 1 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: filter_policy: bloomfilter 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: whole_key_filtering: 1 2024-04-26T01:29:29.231 INFO:journalctl@ceph.mon.b.smithi053.stdout: verify_compression: 0 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout: read_amp_bytes_per_bit: 0 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout: format_version: 5 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout: enable_index_compression: 1 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_align: 0 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout: max_auto_readahead_size: 262144 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout: prepopulate_block_cache: 0 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout: initial_auto_readahead_size: 8192 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T01:29:29.232 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression: NoCompression 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.num_levels: 7 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T01:29:29.233 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:29:29.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T01:29:29.235 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T01:29:29.236 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T01:29:29.236 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T01:29:29.237 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T01:29:29.237 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T01:29:29.237 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T01:29:29.237 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T01:29:29.237 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T01:29:29.237 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T01:29:29.238 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.inplace_update_support: 0 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T01:29:29.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.bloom_locality: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.max_successive_merges: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.ttl: 2592000 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.enable_blob_files: false 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.min_blob_size: 0 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T01:29:29.240 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2bc7c8c4-09ca-4252-b67f-117637109aaa 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094968953361, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T01:29:29.241 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094968953974, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714094968, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2bc7c8c4-09ca-4252-b67f-117637109aaa", "db_session_id": "JTRKXOKR09GU6KXY3Z9I", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714094968954087, "job": 1, "event": "recovery_finished"} 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a9a9d06000 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: DB pointer 0x55a9a9cf0000 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** DB Stats ** 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:29:29.242 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-26T01:29:29.243 INFO:journalctl@ceph.mon.b.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:29:29.244 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout: Block cache BinnedLRUCache@0x55a9a9bf11f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: using public_addr v2:172.21.15.53:0/0 -> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: starting mon.b rank -1 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(???) e0 preinit fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).mds e1 new map 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-26T01:29:29.245 INFO:journalctl@ceph.mon.b.smithi053.stdout: e1 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout: legacy client fscid: -1 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout: No filesystems configured 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T01:29:29.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Added host smithi053 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.248 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Deploying cephadm binary to smithi169 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Added host smithi169 2024-04-26T01:29:29.249 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:29:29.250 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2183329858' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2183329858' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:29.251 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c;count:3 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Deploying daemon mon.c on smithi169 2024-04-26T01:29:29.252 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/2295154596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.a calling monitor election 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.c calling monitor election 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:29.253 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: monmap epoch 2 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: last_changed 2024-04-26T01:29:19.804576+0000 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: min_mon_release 19 (squid) 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: election_strategy: 1 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:29.254 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: fsmap 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mgrmap e13: x(active, since 61s) 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: overall HEALTH_OK 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: Deploying daemon mon.b on smithi053 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/540631076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:29.255 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.256 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.256 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/3186524283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:29.256 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 ceph-mon[32593]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T01:29:29.256 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 bash[32573]: 2cf2123eb492f35494937a26e7ea11caac1d947d0255577099fd19e0ea4b81b7 2024-04-26T01:29:29.256 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:28 smithi053 systemd[1]: Started Ceph mon.b for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:29:29.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:28 smithi028 ceph-mon[28247]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:29.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:28 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/3186524283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:29:30.443 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:29:30.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:29:30.687 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: mon.c calling monitor election 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: mon.a calling monitor election 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: monmap epoch 3 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: last_changed 2024-04-26T01:29:28.979735+0000 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: min_mon_release 19 (squid) 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: election_strategy: 1 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-04-26T01:29:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: fsmap 2024-04-26T01:29:34.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:34.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: mgrmap e13: x(active, since 70s) 2024-04-26T01:29:34.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: overall HEALTH_OK 2024-04-26T01:29:34.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:34.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:34.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: mon.c calling monitor election 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: mon.a calling monitor election 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:29:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: monmap epoch 3 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: last_changed 2024-04-26T01:29:28.979735+0000 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: min_mon_release 19 (squid) 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: election_strategy: 1 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: fsmap 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: mgrmap e13: x(active, since 70s) 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: overall HEALTH_OK 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:34.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:35.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.172+0000 7f28f4ab1700 1 -- 172.21.15.169:0/2902100033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80035c0 msgr2=0x7f28d8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:35.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.172+0000 7f28f4ab1700 1 --2- 172.21.15.169:0/2902100033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80035c0 0x7f28d8005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28f006ad30 tx=0x7f28e000feb0 comp rx=0 tx=0).stop 2024-04-26T01:29:35.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.172+0000 7f28f4ab1700 1 -- 172.21.15.169:0/2902100033 shutdown_connections 2024-04-26T01:29:35.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.172+0000 7f28f4ab1700 1 --2- 172.21.15.169:0/2902100033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80035c0 0x7f28d8005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:35.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.172+0000 7f28f4ab1700 1 -- 172.21.15.169:0/2902100033 >> 172.21.15.169:0/2902100033 conn(0x7f28f00fcf80 msgr2=0x7f28f00ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:35.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.172+0000 7f28f4ab1700 1 -- 172.21.15.169:0/2902100033 shutdown_connections 2024-04-26T01:29:35.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.172+0000 7f28f4ab1700 1 -- 172.21.15.169:0/2902100033 wait complete. 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.173+0000 7f28f4ab1700 1 Processor -- start 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.173+0000 7f28f4ab1700 1 -- start start 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28f4ab1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80035c0 0x7f28f01a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28f4ab1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28f01a4920 0x7f28f019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28f4ab1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28f019eae0 0x7f28f019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28f4ab1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28f0110a60 con 0x7f28d80035c0 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28f4ab1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f28f01108e0 con 0x7f28f01a4920 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28f4ab1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f28f0110be0 con 0x7f28f019eae0 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28f019eae0 0x7f28f019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28f019eae0 0x7f28f019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:41230/0 (socket says 172.21.15.169:41230) 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 -- 172.21.15.169:0/4265471232 learned_addr learned my addr 172.21.15.169:0/4265471232 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:29:35.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 -- 172.21.15.169:0/4265471232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28f01a4920 msgr2=0x7f28f019e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28f01a4920 0x7f28f019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 -- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80035c0 msgr2=0x7f28f01a43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80035c0 0x7f28f01a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 -- 172.21.15.169:0/4265471232 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f01ab2b0 con 0x7f28f019eae0 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28eed9d700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28f019eae0 0x7f28f019ef90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f28e800ba50 tx=0x7f28e800be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.174+0000 7f28df7fe700 1 -- 172.21.15.169:0/4265471232 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8019070 con 0x7f28f019eae0 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.175+0000 7f28df7fe700 1 -- 172.21.15.169:0/4265471232 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e8014440 con 0x7f28f019eae0 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.175+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f01ab4e0 con 0x7f28f019eae0 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.175+0000 7f28df7fe700 1 -- 172.21.15.169:0/4265471232 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8007590 con 0x7f28f019eae0 2024-04-26T01:29:35.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.175+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f01ab9d0 con 0x7f28f019eae0 2024-04-26T01:29:35.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.175+0000 7f28df7fe700 1 -- 172.21.15.169:0/4265471232 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f28e801c070 con 0x7f28f019eae0 2024-04-26T01:29:35.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.176+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f00640a0 con 0x7f28f019eae0 2024-04-26T01:29:35.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.176+0000 7f28df7fe700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f28d0039ae0 0x7f28d003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:35.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.176+0000 7f28df7fe700 1 -- 172.21.15.169:0/4265471232 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f28e804ce90 con 0x7f28f019eae0 2024-04-26T01:29:35.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.176+0000 7f28ee59c700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f28d0039ae0 0x7f28d003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:35.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.177+0000 7f28ee59c700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f28d0039ae0 0x7f28d003bfa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f28f006ad30 tx=0x7f28e0001e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:35.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.182+0000 7f28df7fe700 1 -- 172.21.15.169:0/4265471232 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28e8018170 con 0x7f28f019eae0 2024-04-26T01:29:35.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:35 smithi028 ceph-mon[28247]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:35.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:35.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:35.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:35.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:35.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:35 smithi169 ceph-mon[32851]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:35.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:35.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:35.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:35.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:35.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.471+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f28f002df80 con 0x7f28f019eae0 2024-04-26T01:29:35.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.471+0000 7f28df7fe700 1 -- 172.21.15.169:0/4265471232 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f28e80185a0 con 0x7f28f019eae0 2024-04-26T01:29:35.467 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-26T01:29:35.467 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":3,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","modified":"2024-04-26T01:29:28.979735Z","created":"2024-04-26T01:27:11.642338Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T01:29:35.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.473+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f28d0039ae0 msgr2=0x7f28d003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:35.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.473+0000 7f28f4ab1700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f28d0039ae0 0x7f28d003bfa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f28f006ad30 tx=0x7f28e0001e50 comp rx=0 tx=0).stop 2024-04-26T01:29:35.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.473+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28f019eae0 msgr2=0x7f28f019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:35.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.473+0000 7f28f4ab1700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28f019eae0 0x7f28f019ef90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f28e800ba50 tx=0x7f28e800be10 comp rx=0 tx=0).stop 2024-04-26T01:29:35.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.474+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 shutdown_connections 2024-04-26T01:29:35.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.474+0000 7f28f4ab1700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f28d0039ae0 0x7f28d003bfa0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:35.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.474+0000 7f28f4ab1700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28f019eae0 0x7f28f019ef90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:35.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.474+0000 7f28f4ab1700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80035c0 0x7f28f01a43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:35.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.474+0000 7f28f4ab1700 1 --2- 172.21.15.169:0/4265471232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28f01a4920 0x7f28f019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:35.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.474+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 >> 172.21.15.169:0/4265471232 conn(0x7f28f00fcf80 msgr2=0x7f28f0105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:35.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.474+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 shutdown_connections 2024-04-26T01:29:35.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:35.474+0000 7f28f4ab1700 1 -- 172.21.15.169:0/4265471232 wait complete. 2024-04-26T01:29:35.470 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 3 2024-04-26T01:29:36.232 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T01:29:36.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T01:29:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: mon.c calling monitor election 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: mon.a calling monitor election 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: monmap epoch 3 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: last_changed 2024-04-26T01:29:28.979735+0000 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: min_mon_release 19 (squid) 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: election_strategy: 1 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: fsmap 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: mgrmap e13: x(active, since 70s) 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: overall HEALTH_OK 2024-04-26T01:29:36.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:36.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:36.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:36.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:36.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:36.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:36.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:29:37.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: mon.b calling monitor election 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: mon.b calling monitor election 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: mon.c calling monitor election 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: mon.a calling monitor election 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: monmap epoch 3 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: last_changed 2024-04-26T01:29:28.979735+0000 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: min_mon_release 19 (squid) 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: election_strategy: 1 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: fsmap 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: mgrmap e13: x(active, since 72s) 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: overall HEALTH_OK 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:36 smithi028 ceph-mon[28247]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: mon.b calling monitor election 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: mon.b calling monitor election 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: mon.c calling monitor election 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: mon.a calling monitor election 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: monmap epoch 3 2024-04-26T01:29:37.131 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: last_changed 2024-04-26T01:29:28.979735+0000 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: min_mon_release 19 (squid) 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: election_strategy: 1 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: fsmap 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: mgrmap e13: x(active, since 72s) 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: overall HEALTH_OK 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:36 smithi169 ceph-mon[32851]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: mon.b calling monitor election 2024-04-26T01:29:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-26T01:29:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:29:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:29:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: mon.b calling monitor election 2024-04-26T01:29:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: mon.c calling monitor election 2024-04-26T01:29:37.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: mon.a calling monitor election 2024-04-26T01:29:37.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:29:37.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: monmap epoch 3 2024-04-26T01:29:37.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:37.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: last_changed 2024-04-26T01:29:28.979735+0000 2024-04-26T01:29:37.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: created 2024-04-26T01:27:11.642338+0000 2024-04-26T01:29:37.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: min_mon_release 19 (squid) 2024-04-26T01:29:37.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: election_strategy: 1 2024-04-26T01:29:37.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-26T01:29:37.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-04-26T01:29:37.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-04-26T01:29:37.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: fsmap 2024-04-26T01:29:37.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:29:37.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: mgrmap e13: x(active, since 72s) 2024-04-26T01:29:37.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: overall HEALTH_OK 2024-04-26T01:29:37.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:37.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:37.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:37.222 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:36 smithi053 ceph-mon[32593]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:29:38.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:38.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:29:38.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:38.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:38.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:38.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:37 smithi028 ceph-mon[28247]: Reconfiguring daemon mon.a on smithi028 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:29:38.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:38.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:38.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:38.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:37 smithi053 ceph-mon[32593]: Reconfiguring daemon mon.a on smithi028 2024-04-26T01:29:38.320 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.320 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:38.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:37 smithi169 ceph-mon[32851]: Reconfiguring daemon mon.a on smithi028 2024-04-26T01:29:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:39 smithi053 ceph-mon[32593]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:39.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:39 smithi028 ceph-mon[28247]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:39.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:39.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:39 smithi169 ceph-mon[32851]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:39.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.739+0000 7f48d1600700 1 -- 172.21.15.28:0/3577687523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48cc108540 msgr2=0x7f48cc108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.739+0000 7f48d1600700 1 --2- 172.21.15.28:0/3577687523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48cc108540 0x7f48cc108920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009910 tx=0x7f48bc009c20 comp rx=0 tx=0).stop 2024-04-26T01:29:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.739+0000 7f48d1600700 1 -- 172.21.15.28:0/3577687523 shutdown_connections 2024-04-26T01:29:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.739+0000 7f48d1600700 1 --2- 172.21.15.28:0/3577687523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48cc108540 0x7f48cc108920 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.739+0000 7f48d1600700 1 -- 172.21.15.28:0/3577687523 >> 172.21.15.28:0/3577687523 conn(0x7f48cc0fdfa0 msgr2=0x7f48cc1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.740+0000 7f48d1600700 1 -- 172.21.15.28:0/3577687523 shutdown_connections 2024-04-26T01:29:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.740+0000 7f48d1600700 1 -- 172.21.15.28:0/3577687523 wait complete. 2024-04-26T01:29:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.741+0000 7f48d1600700 1 Processor -- start 2024-04-26T01:29:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.741+0000 7f48d1600700 1 -- start start 2024-04-26T01:29:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48d1600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48cc19a510 0x7f48cc19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48d1600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc19ae30 0x7f48cc19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48d1600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48cc19f320 0x7f48cc19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48d1600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc10c9b0 con 0x7f48cc19f320 2024-04-26T01:29:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48d1600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc10c830 con 0x7f48cc19ae30 2024-04-26T01:29:39.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48d1600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc10cb30 con 0x7f48cc19a510 2024-04-26T01:29:39.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48caa00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48cc19a510 0x7f48cc19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:39.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48cb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48cc19f320 0x7f48cc19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:39.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48caa00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48cc19a510 0x7f48cc19a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52992/0 (socket says 172.21.15.28:52992) 2024-04-26T01:29:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48caa00700 1 -- 172.21.15.28:0/4286234387 learned_addr learned my addr 172.21.15.28:0/4286234387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:29:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.742+0000 7f48ca000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc19ae30 0x7f48cc19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48caa00700 1 -- 172.21.15.28:0/4286234387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc19ae30 msgr2=0x7f48cc19ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48caa00700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc19ae30 0x7f48cc19ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48caa00700 1 -- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48cc19f320 msgr2=0x7f48cc19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48caa00700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48cc19f320 0x7f48cc19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48caa00700 1 -- 172.21.15.28:0/4286234387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc1a7360 con 0x7f48cc19a510 2024-04-26T01:29:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48caa00700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48cc19a510 0x7f48cc19a8f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48bc014000 tx=0x7f48bc004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48c3400700 1 -- 172.21.15.28:0/4286234387 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc009e60 con 0x7f48cc19a510 2024-04-26T01:29:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48c3400700 1 -- 172.21.15.28:0/4286234387 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48bc00b750 con 0x7f48cc19a510 2024-04-26T01:29:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.743+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc1a75f0 con 0x7f48cc19a510 2024-04-26T01:29:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.744+0000 7f48c3400700 1 -- 172.21.15.28:0/4286234387 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc0038c0 con 0x7f48cc19a510 2024-04-26T01:29:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.744+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48cc1a7b00 con 0x7f48cc19a510 2024-04-26T01:29:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.745+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48cc0640a0 con 0x7f48cc19a510 2024-04-26T01:29:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.745+0000 7f48c3400700 1 -- 172.21.15.28:0/4286234387 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f48bc00bc00 con 0x7f48cc19a510 2024-04-26T01:29:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.745+0000 7f48c3400700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48b0039ae0 0x7f48b003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.746+0000 7f48c3400700 1 -- 172.21.15.28:0/4286234387 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f48bc04d9a0 con 0x7f48cc19a510 2024-04-26T01:29:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.746+0000 7f48ca000700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48b0039ae0 0x7f48b003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.746+0000 7f48ca000700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48b0039ae0 0x7f48b003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009ad0 tx=0x7f48b8009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.751+0000 7f48c3400700 1 -- 172.21.15.28:0/4286234387 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48bc013070 con 0x7f48cc19a510 2024-04-26T01:29:39.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.985+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f48cc1a7d60 con 0x7f48cc19a510 2024-04-26T01:29:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.985+0000 7f48c3400700 1 -- 172.21.15.28:0/4286234387 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f48bc021bd0 con 0x7f48cc19a510 2024-04-26T01:29:39.980 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:39.980 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-26T01:29:39.981 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:29:39.981 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2024-04-26T01:29:39.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48b0039ae0 msgr2=0x7f48b003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:39.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48b0039ae0 0x7f48b003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009ad0 tx=0x7f48b8009210 comp rx=0 tx=0).stop 2024-04-26T01:29:39.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48cc19a510 msgr2=0x7f48cc19a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:39.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48cc19a510 0x7f48cc19a8f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48bc014000 tx=0x7f48bc004f80 comp rx=0 tx=0).stop 2024-04-26T01:29:39.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 shutdown_connections 2024-04-26T01:29:39.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48b0039ae0 0x7f48b003bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48cc19f320 0x7f48cc19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48cc19a510 0x7f48cc19a8f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 --2- 172.21.15.28:0/4286234387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc19ae30 0x7f48cc19ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.989+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 >> 172.21.15.28:0/4286234387 conn(0x7f48cc0fdfa0 msgr2=0x7f48cc074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.990+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 shutdown_connections 2024-04-26T01:29:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:39.990+0000 7f48d1600700 1 -- 172.21.15.28:0/4286234387 wait complete. 2024-04-26T01:29:40.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:40.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:40.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:40.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:40.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:40.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:40 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/4286234387' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:40.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:40.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:40.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:40.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:40.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:40.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:40 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/4286234387' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:40.728 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T01:29:40.728 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:29:40.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:29:40.825 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:29:40.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:29:40.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:40.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:40.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:40.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:40.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:40.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:40 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/4286234387' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:40.967 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:29:40.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:29:40.996 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:29:40.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:29:41.061 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:29:41.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:29:41.091 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:29:41.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:29:41.157 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-04-26T01:29:41.157 INFO:tasks.cephadm:Adding mgr.y on smithi053 2024-04-26T01:29:41.157 INFO:tasks.cephadm:Adding mgr.z on smithi169 2024-04-26T01:29:41.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi028=x;smithi053=y;smithi169=z' 2024-04-26T01:29:41.434 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: Reconfiguring daemon mon.b on smithi053 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:41.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:41 smithi169 ceph-mon[32851]: Reconfiguring daemon mon.c on smithi169 2024-04-26T01:29:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:29:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: Reconfiguring daemon mon.b on smithi053 2024-04-26T01:29:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:29:41.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:41.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:41.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:41.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:41 smithi028 ceph-mon[28247]: Reconfiguring daemon mon.c on smithi169 2024-04-26T01:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: Reconfiguring daemon mon.b on smithi053 2024-04-26T01:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:29:41.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:41.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:41 smithi053 ceph-mon[32593]: Reconfiguring daemon mon.c on smithi169 2024-04-26T01:29:42.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:42 smithi028 ceph-mon[28247]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:42.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:42 smithi169 ceph-mon[32851]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:42.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.846+0000 7f897a7fc700 1 -- 172.21.15.169:0/1760733912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c005660 con 0x7f897c1025c0 2024-04-26T01:29:42.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.846+0000 7f8981836700 1 -- 172.21.15.169:0/1760733912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f897c1025c0 msgr2=0x7f897c1029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:42.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.846+0000 7f8981836700 1 --2- 172.21.15.169:0/1760733912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f897c1025c0 0x7f897c1029e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f896c009910 tx=0x7f896c009c20 comp rx=0 tx=0).stop 2024-04-26T01:29:42.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.846+0000 7f8981836700 1 -- 172.21.15.169:0/1760733912 shutdown_connections 2024-04-26T01:29:42.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.846+0000 7f8981836700 1 --2- 172.21.15.169:0/1760733912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f897c1025c0 0x7f897c1029e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:42.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.846+0000 7f8981836700 1 -- 172.21.15.169:0/1760733912 >> 172.21.15.169:0/1760733912 conn(0x7f897c0fdb60 msgr2=0x7f897c0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:42.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.846+0000 7f8981836700 1 -- 172.21.15.169:0/1760733912 shutdown_connections 2024-04-26T01:29:42.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.846+0000 7f8981836700 1 -- 172.21.15.169:0/1760733912 wait complete. 2024-04-26T01:29:42.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f8981836700 1 Processor -- start 2024-04-26T01:29:42.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f8981836700 1 -- start start 2024-04-26T01:29:42.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f8981836700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f897c1025c0 0x7f897c197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:42.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f8981836700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f897c198280 0x7f897c19d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f8981836700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f897c198700 0x7f897c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f8981836700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f897c108500 con 0x7f897c1025c0 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f8981836700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f897c108380 con 0x7f897c198280 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f8981836700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f897c108680 con 0x7f897c198700 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f897b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f897c198700 0x7f897c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f897b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f897c198700 0x7f897c19f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32832/0 (socket says 172.21.15.169:32832) 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.847+0000 7f897b7fe700 1 -- 172.21.15.169:0/1407504053 learned_addr learned my addr 172.21.15.169:0/1407504053 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f89727fc700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f897c198280 0x7f897c19d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f897b7fe700 1 -- 172.21.15.169:0/1407504053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f897c198280 msgr2=0x7f897c19d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f897affd700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f897c1025c0 0x7f897c197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f897b7fe700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f897c198280 0x7f897c19d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f897b7fe700 1 -- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f897c1025c0 msgr2=0x7f897c197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:42.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f897b7fe700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f897c1025c0 0x7f897c197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f897b7fe700 1 -- 172.21.15.169:0/1407504053 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c1a0160 con 0x7f897c198700 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f897b7fe700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f897c198700 0x7f897c19f860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f897400ba50 tx=0x7f897400be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f8980834700 1 -- 172.21.15.169:0/1407504053 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8974019070 con 0x7f897c198700 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897c1a0450 con 0x7f897c198700 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897c1a0a10 con 0x7f897c198700 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f8980834700 1 -- 172.21.15.169:0/1407504053 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8974014440 con 0x7f897c198700 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.848+0000 7f8980834700 1 -- 172.21.15.169:0/1407504053 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897401d610 con 0x7f897c198700 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.849+0000 7f8980834700 1 -- 172.21.15.169:0/1407504053 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f897401c070 con 0x7f897c198700 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.849+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895c005360 con 0x7f897c198700 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.849+0000 7f8980834700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8968039ae0 0x7f896803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:42.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.849+0000 7f8980834700 1 -- 172.21.15.169:0/1407504053 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f897404be50 con 0x7f897c198700 2024-04-26T01:29:42.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.850+0000 7f897affd700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8968039ae0 0x7f896803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:42.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.851+0000 7f897affd700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8968039ae0 0x7f896803bfa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f896c0052c0 tx=0x7f896c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:42.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:42.854+0000 7f8980834700 1 -- 172.21.15.169:0/1407504053 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f897401b2c0 con 0x7f897c198700 2024-04-26T01:29:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:42 smithi053 ceph-mon[32593]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:43.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.068+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi053=y;smithi169=z", "target": ["mon-mgr", ""]}) -- 0x7f895c000bf0 con 0x7f8968039ae0 2024-04-26T01:29:43.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.082+0000 7f8980834700 1 -- 172.21.15.169:0/1407504053 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f895c000bf0 con 0x7f8968039ae0 2024-04-26T01:29:43.077 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mgr update... 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8968039ae0 msgr2=0x7f896803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8968039ae0 0x7f896803bfa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f896c0052c0 tx=0x7f896c005fb0 comp rx=0 tx=0).stop 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f897c198700 msgr2=0x7f897c19f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f897c198700 0x7f897c19f860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f897400ba50 tx=0x7f897400be10 comp rx=0 tx=0).stop 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 shutdown_connections 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8968039ae0 0x7f896803bfa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f897c198700 0x7f897c19f860 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f897c1025c0 0x7f897c197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 --2- 172.21.15.169:0/1407504053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f897c198280 0x7f897c19d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:43.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 >> 172.21.15.169:0/1407504053 conn(0x7f897c0fdb60 msgr2=0x7f897c0fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:43.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 shutdown_connections 2024-04-26T01:29:43.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:29:43.084+0000 7f8981836700 1 -- 172.21.15.169:0/1407504053 wait complete. 2024-04-26T01:29:44.310 DEBUG:teuthology.orchestra.run.smithi053:mgr.y> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T01:29:44.313 DEBUG:teuthology.orchestra.run.smithi169:mgr.z> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T01:29:44.316 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T01:29:44.316 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:29:44.316 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:29:44.336 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi053=y;smithi169=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:44.336 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: Saving service mgr spec with placement smithi028=x;smithi053=y;smithi169=z;count:3 2024-04-26T01:29:44.336 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.336 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.336 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.336 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:44.336 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi053=y;smithi169=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: Saving service mgr spec with placement smithi028=x;smithi053=y;smithi169=z;count:3 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.337 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:44.338 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:44.379 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:29:44.379 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-04-26T01:29:44.435 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:29:44.436 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:44.436 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-26T01:29:44.436 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:44.436 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:44.436 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-26 01:28:36.865785328 +0000 2024-04-26T01:29:44.436 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-26 01:23:17.038927584 +0000 2024-04-26T01:29:44.436 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-26 01:23:17.039927557 +0000 2024-04-26T01:29:44.436 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-26 01:23:17.038927584 +0000 2024-04-26T01:29:44.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:29:44.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi053=y;smithi169=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: Saving service mgr spec with placement smithi028=x;smithi053=y;smithi169=z;count:3 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:44.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:44.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-26T01:29:44.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-26T01:29:44.524 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000300048 s, 1.7 MB/s 2024-04-26T01:29:44.528 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:29:44.589 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-04-26T01:29:44.639 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:29:44.639 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:44.640 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-26T01:29:44.640 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:44.640 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:44.640 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-26 01:28:36.944783115 +0000 2024-04-26T01:29:44.640 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-26 01:23:17.653910446 +0000 2024-04-26T01:29:44.640 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-26 01:23:17.653910446 +0000 2024-04-26T01:29:44.640 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-26 01:23:17.653910446 +0000 2024-04-26T01:29:44.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:29:44.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-26T01:29:44.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-26T01:29:44.715 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000266983 s, 1.9 MB/s 2024-04-26T01:29:44.718 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:29:44.776 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-26 01:28:37.007781351 +0000 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-26 01:23:18.259893559 +0000 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-26 01:23:18.259893559 +0000 2024-04-26T01:29:44.831 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-26 01:23:18.259893559 +0000 2024-04-26T01:29:44.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:29:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-26T01:29:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-26T01:29:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000316674 s, 1.6 MB/s 2024-04-26T01:29:44.920 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:29:44.979 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 691 Links: 1 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-26 01:28:37.079779334 +0000 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-26 01:23:18.887876058 +0000 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-26 01:23:18.888876030 +0000 2024-04-26T01:29:45.033 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-26 01:23:18.887876058 +0000 2024-04-26T01:29:45.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:29:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-26T01:29:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-26T01:29:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000226626 s, 2.3 MB/s 2024-04-26T01:29:45.115 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:29:45.170 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-26T01:29:45.170 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:29:45.189 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:29:45.190 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-04-26T01:29:45.249 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:29:45.249 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:45.249 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-26T01:29:45.249 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:45.249 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:45.249 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-26 01:29:28.285506089 +0000 2024-04-26T01:29:45.250 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-26 01:22:06.306692871 +0000 2024-04-26T01:29:45.250 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-26 01:22:06.306692871 +0000 2024-04-26T01:29:45.250 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-26 01:22:06.306692871 +0000 2024-04-26T01:29:45.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:29:45.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:45 smithi028 ceph-mon[28247]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:45.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:45 smithi028 ceph-mon[28247]: Deploying daemon mgr.z on smithi169 2024-04-26T01:29:45.316 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-26T01:29:45.316 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-26T01:29:45.316 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000130587 s, 3.9 MB/s 2024-04-26T01:29:45.318 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:29:45.362 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:45 smithi169 ceph-mon[32851]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:45.362 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:45 smithi169 ceph-mon[32851]: Deploying daemon mgr.z on smithi169 2024-04-26T01:29:45.377 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-26 01:29:28.286506069 +0000 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-26 01:22:06.649685553 +0000 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-26 01:22:06.649685553 +0000 2024-04-26T01:29:45.437 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-26 01:22:06.649685553 +0000 2024-04-26T01:29:45.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:29:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:45 smithi053 ceph-mon[32593]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:45 smithi053 ceph-mon[32593]: Deploying daemon mgr.z on smithi169 2024-04-26T01:29:45.494 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-26T01:29:45.494 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-26T01:29:45.494 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000108905 s, 4.7 MB/s 2024-04-26T01:29:45.495 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:29:45.555 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-04-26T01:29:45.612 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:29:45.612 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:45.612 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-26T01:29:45.612 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:45.613 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:45.613 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-26 01:29:28.286506069 +0000 2024-04-26T01:29:45.613 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-26 01:22:06.982678448 +0000 2024-04-26T01:29:45.613 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-26 01:22:06.982678448 +0000 2024-04-26T01:29:45.613 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-26 01:22:06.982678448 +0000 2024-04-26T01:29:45.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:29:45.679 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-26T01:29:45.679 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-26T01:29:45.679 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000124672 s, 4.1 MB/s 2024-04-26T01:29:45.681 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:29:45.739 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-04-26T01:29:45.796 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:29:45.797 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:45.797 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-26T01:29:45.797 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:45.797 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:45.797 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-26 01:29:28.286506069 +0000 2024-04-26T01:29:45.798 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-26 01:22:07.316671322 +0000 2024-04-26T01:29:45.798 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-26 01:22:07.316671322 +0000 2024-04-26T01:29:45.798 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-26 01:22:07.316671322 +0000 2024-04-26T01:29:45.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:29:45.863 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-26T01:29:45.863 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-26T01:29:45.863 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000118442 s, 4.3 MB/s 2024-04-26T01:29:45.865 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:29:45.924 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-26T01:29:45.925 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:29:45.941 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:29:45.942 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-04-26T01:29:45.998 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:29:45.998 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:45.998 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-26T01:29:45.998 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:45.999 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:45.999 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-04-26 01:29:45.651062864 +0000 2024-04-26T01:29:45.999 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-04-26 01:22:13.840482656 +0000 2024-04-26T01:29:45.999 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-04-26 01:22:13.840482656 +0000 2024-04-26T01:29:46.001 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-04-26 01:22:13.840482656 +0000 2024-04-26T01:29:46.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:29:46.086 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-04-26T01:29:46.086 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-04-26T01:29:46.086 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.00489561 s, 105 kB/s 2024-04-26T01:29:46.087 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:29:46.103 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-04-26T01:29:46.157 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:29:46.158 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:46.158 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T01:29:46.158 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:46.158 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:46.158 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-04-26 01:29:45.651062864 +0000 2024-04-26T01:29:46.158 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-04-26 01:22:14.174475384 +0000 2024-04-26T01:29:46.158 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-04-26 01:22:14.174475384 +0000 2024-04-26T01:29:46.158 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-04-26 01:22:14.174475384 +0000 2024-04-26T01:29:46.159 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:29:46.222 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-04-26T01:29:46.222 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-04-26T01:29:46.222 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000239515 s, 2.1 MB/s 2024-04-26T01:29:46.223 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:29:46.278 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-04-26T01:29:46.334 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:29:46.335 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:46.335 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-26T01:29:46.335 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:46.335 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:46.335 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-04-26 01:29:45.651062864 +0000 2024-04-26T01:29:46.335 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-04-26 01:22:14.500468287 +0000 2024-04-26T01:29:46.335 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-04-26 01:22:14.500468287 +0000 2024-04-26T01:29:46.335 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-04-26 01:22:14.500468287 +0000 2024-04-26T01:29:46.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:29:46.352 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:46 smithi169 systemd[1]: Started Ceph mgr.z for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:29:46.375 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-04-26T01:29:46.375 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-04-26T01:29:46.375 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000227816 s, 2.2 MB/s 2024-04-26T01:29:46.376 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:29:46.432 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-04-26T01:29:46.488 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:29:46.488 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:29:46.488 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-26T01:29:46.488 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:29:46.488 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:29:46.488 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-04-26 01:29:45.652062843 +0000 2024-04-26T01:29:46.488 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-04-26 01:22:14.851460646 +0000 2024-04-26T01:29:46.488 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-04-26 01:22:14.851460646 +0000 2024-04-26T01:29:46.489 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-04-26 01:22:14.851460646 +0000 2024-04-26T01:29:46.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:29:46.550 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-04-26T01:29:46.550 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-04-26T01:29:46.550 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000183526 s, 2.8 MB/s 2024-04-26T01:29:46.552 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:29:46.608 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-04-26T01:29:46.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:29:46.649 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:46.356+0000 7fd40b67e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:29:46.649 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:46.548+0000 7fd40b67e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:29:46.649 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:46.654+0000 7fd40b67e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:29:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:29:47.369 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:46.963+0000 7fd40b67e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:29:47.369 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:47.047+0000 7fd40b67e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:29:47.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:47.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:47.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:29:47.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:47.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:47.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:47 smithi028 ceph-mon[28247]: Deploying daemon mgr.y on smithi053 2024-04-26T01:29:47.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:47.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:47.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:29:47.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:47.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:47.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:47 smithi169 ceph-mon[32851]: Deploying daemon mgr.y on smithi053 2024-04-26T01:29:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:47.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:47.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:29:47.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:47.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:47.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:47 smithi053 ceph-mon[32593]: Deploying daemon mgr.y on smithi053 2024-04-26T01:29:48.583 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:48.249+0000 7fd40b67e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:29:48.583 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:48.416+0000 7fd40b67e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:29:48.583 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:48.588+0000 7fd40b67e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:29:48.642 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 systemd[1]: Starting Ceph mgr.y for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:29:48.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 podman[33959]: 2024-04-26 01:29:48.647066061 +0000 UTC m=+0.103447456 container create 4c2228c33d22bb0a794a33ac5f2179da8e600d61460237d38688b42fb5118e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:29:48.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 podman[33959]: 2024-04-26 01:29:48.561498371 +0000 UTC m=+0.017879766 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:29:48.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 podman[33959]: 2024-04-26 01:29:48.81587719 +0000 UTC m=+0.272258596 container init 4c2228c33d22bb0a794a33ac5f2179da8e600d61460237d38688b42fb5118e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:29:48.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 podman[33959]: 2024-04-26 01:29:48.818323754 +0000 UTC m=+0.274705148 container start 4c2228c33d22bb0a794a33ac5f2179da8e600d61460237d38688b42fb5118e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T01:29:48.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 ceph-mgr[33978]: -- 172.21.15.53:0/39521018 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e37b01c5a0 con 0x55e37aff0c00 2024-04-26T01:29:48.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 bash[33959]: 4c2228c33d22bb0a794a33ac5f2179da8e600d61460237d38688b42fb5118e42 2024-04-26T01:29:48.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 systemd[1]: Started Ceph mgr.y for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:29:49.158 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:48.875+0000 7fd40b67e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:29:49.159 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:48.966+0000 7fd40b67e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:29:49.267 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:48.974+0000 7f11319f2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:29:49.268 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:49.166+0000 7f11319f2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:29:49.268 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:49.272+0000 7f11319f2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:29:49.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:49 smithi028 ceph-mon[28247]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:49.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:49.599 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-mon[32593]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:49.599 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.599 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.599 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.599 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.599 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:49.659 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:49.172+0000 7fd40b67e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:29:49.659 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:49.369+0000 7fd40b67e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:29:49.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:49 smithi169 ceph-mon[32851]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:49.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:49.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:29:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:29:49.921 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-26T01:29:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-26T01:29:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0384933 s, 272 MB/s 2024-04-26T01:29:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-26T01:29:49.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:49.603+0000 7f11319f2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:29:49.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:49.692+0000 7f11319f2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:29:50.409 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:50.005+0000 7fd40b67e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:29:50.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-04-26T01:29:50.737 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:50.433+0000 7fd40b67e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:29:50.737 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:50.538+0000 7fd40b67e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:29:50.737 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:50.740+0000 7fd40b67e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:29:51.146 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:29:51.334 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:50.988+0000 7f11319f2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:29:51.334 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:51.163+0000 7f11319f2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:29:51.334 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:51.338+0000 7f11319f2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:29:51.594 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:51.266+0000 7fd40b67e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:29:51.594 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:51.356+0000 7fd40b67e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:29:51.594 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:51.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:51.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:51.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:51.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:51.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:51.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:51.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:51.631 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:29:51.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:51.908 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:51.598+0000 7fd40b67e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:29:51.908 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:51.731+0000 7fd40b67e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:29:51.909 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:51.824+0000 7fd40b67e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:29:51.936 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:51.636+0000 7f11319f2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:29:51.936 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:51.732+0000 7f11319f2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:29:51.936 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:51.941+0000 7f11319f2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:29:52.216 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:52.140+0000 7f11319f2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:29:52.236 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:51.938+0000 7fd40b67e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:29:52.236 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:52.240+0000 7fd40b67e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:29:52.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:52 smithi028 ceph-mon[28247]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:29:52.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:52 smithi028 ceph-mon[28247]: Reconfiguring daemon mgr.x on smithi028 2024-04-26T01:29:52.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:52 smithi169 ceph-mon[32851]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:29:52.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:52 smithi169 ceph-mon[32851]: Reconfiguring daemon mgr.x on smithi028 2024-04-26T01:29:52.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:52 smithi053 ceph-mon[32593]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:29:52.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:52 smithi053 ceph-mon[32593]: Reconfiguring daemon mgr.x on smithi028 2024-04-26T01:29:53.158 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:52.838+0000 7fd40b67e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:29:53.158 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:29:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:29:53.051+0000 7fd40b67e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:29:53.193 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:52.761+0000 7f11319f2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:29:53.466 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:53.197+0000 7f11319f2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:29:53.467 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:53.296+0000 7f11319f2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:29:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-mon[32593]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:29:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-mon[32593]: Standby manager daemon z started 2024-04-26T01:29:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:29:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:29:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:29:53.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:53 smithi028 ceph-mon[28247]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:53.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:53 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:29:53.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:53 smithi028 ceph-mon[28247]: Standby manager daemon z started 2024-04-26T01:29:53.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:53 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:29:53.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:53 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:29:53.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:53 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:29:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:53 smithi169 ceph-mon[32851]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:53 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:29:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:53 smithi169 ceph-mon[32851]: Standby manager daemon z started 2024-04-26T01:29:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:53 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:29:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:53 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:29:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:53 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.169:0/3838118806' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:29:53.966 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:53.504+0000 7f11319f2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:29:54.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 -- 172.21.15.28:0/3846165714 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ca4103520 msgr2=0x7f6ca4105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/3846165714 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ca4103520 0x7f6ca4105910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c94009990 tx=0x7f6c94009ca0 comp rx=0 tx=0).stop 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 -- 172.21.15.28:0/3846165714 shutdown_connections 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/3846165714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca4108780 0x7f6ca410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/3846165714 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ca4103520 0x7f6ca4105910 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/3846165714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ca4105e50 0x7f6ca4108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 -- 172.21.15.28:0/3846165714 >> 172.21.15.28:0/3846165714 conn(0x7f6ca40fcf30 msgr2=0x7f6ca40ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 -- 172.21.15.28:0/3846165714 shutdown_connections 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.791+0000 7f6ca9c00700 1 -- 172.21.15.28:0/3846165714 wait complete. 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.792+0000 7f6ca9c00700 1 Processor -- start 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.792+0000 7f6ca9c00700 1 -- start start 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca9c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca4103520 0x7f6ca419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca9c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ca4105e50 0x7f6ca419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca9c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ca4108780 0x7f6ca41a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca9c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410e580 con 0x7f6ca4103520 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca9c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410e400 con 0x7f6ca4105e50 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca9c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410e700 con 0x7f6ca4108780 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca2000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ca4105e50 0x7f6ca419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca2000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ca4105e50 0x7f6ca419b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:56312/0 (socket says 172.21.15.28:56312) 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca2000700 1 -- 172.21.15.28:0/2739424753 learned_addr learned my addr 172.21.15.28:0/2739424753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca2a00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca4103520 0x7f6ca419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.793+0000 7f6ca3400700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ca4108780 0x7f6ca41a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca2000700 1 -- 172.21.15.28:0/2739424753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ca4108780 msgr2=0x7f6ca41a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca2000700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ca4108780 0x7f6ca41a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca2000700 1 -- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca4103520 msgr2=0x7f6ca419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:29:54.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca2000700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca4103520 0x7f6ca419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca2000700 1 -- 172.21.15.28:0/2739424753 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca41a2a60 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca2a00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca4103520 0x7f6ca419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca2000700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ca4105e50 0x7f6ca419b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e9c0 tx=0x7f6c9000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6c9b400700 1 -- 172.21.15.28:0/2739424753 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9000cbd0 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6c9b400700 1 -- 172.21.15.28:0/2739424753 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c9000cd70 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca41a2d50 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6c9b400700 1 -- 172.21.15.28:0/2739424753 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90005680 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.794+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca41a3230 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.796+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca4069730 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.796+0000 7f6c9b400700 1 -- 172.21.15.28:0/2739424753 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f6c90004750 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.797+0000 7f6c9b400700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6c8806ea20 0x7f6c88070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.798+0000 7f6c9b400700 1 -- 172.21.15.28:0/2739424753 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6c9008dac0 con 0x7f6ca4105e50 2024-04-26T01:29:54.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.798+0000 7f6ca2a00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6c8806ea20 0x7f6c88070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:29:54.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.798+0000 7f6ca2a00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6c8806ea20 0x7f6c88070ee0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400b2e0 tx=0x7f6c9401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:29:54.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:53.802+0000 7f6c9b400700 1 -- 172.21.15.28:0/2739424753 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c9005c270 con 0x7f6ca4105e50 2024-04-26T01:29:54.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:54.053+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6ca4062e80 con 0x7f6c8806ea20 2024-04-26T01:29:54.397 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:54.046+0000 7f11319f2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:29:54.397 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:54.143+0000 7f11319f2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:29:54.717 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:54.401+0000 7f11319f2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:29:54.717 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:54.541+0000 7f11319f2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:29:54.717 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:54.642+0000 7f11319f2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:29:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T01:29:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:29:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:29:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:29:54.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:54.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:54.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:54.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:29:55.072 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:54.764+0000 7f11319f2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:29:55.073 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:55.076+0000 7f11319f2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:29:55.698 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:55 smithi053 ceph-mon[32593]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:55.698 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:55 smithi053 ceph-mon[32593]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:55.699 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:55.699 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:55.699 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:55.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:55 smithi028 ceph-mon[28247]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:55.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:55 smithi028 ceph-mon[28247]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:55.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:55.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:55.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:55.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:55 smithi169 ceph-mon[32851]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:55.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:55 smithi169 ceph-mon[32851]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:29:55.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:29:55.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:29:55.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:29:55.966 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:55.702+0000 7f11319f2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:29:55.967 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:29:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:29:55.924+0000 7f11319f2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:29:56.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:29:56.453+0000 7f6c9b400700 1 -- 172.21.15.28:0/2739424753 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f6c900581e0 con 0x7f6ca4105e50 2024-04-26T01:29:56.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:56 smithi028 ceph-mon[28247]: Standby manager daemon y started 2024-04-26T01:29:56.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:56 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:29:56.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:56 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:29:56.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:56 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:29:56.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:56 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:29:56.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:56 smithi053 ceph-mon[32593]: Standby manager daemon y started 2024-04-26T01:29:56.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:56 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:29:56.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:56 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:29:56.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:56 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:29:56.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:56 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:29:56.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:56 smithi169 ceph-mon[32851]: Standby manager daemon y started 2024-04-26T01:29:56.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:56 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:29:56.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:56 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:29:56.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:56 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:29:56.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:56 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.53:0/1634963266' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:29:57.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:57 smithi169 ceph-mon[32851]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:57.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:57 smithi169 ceph-mon[32851]: mgrmap e15: x(active, since 92s), standbys: y, z 2024-04-26T01:29:57.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:57 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:29:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:57 smithi053 ceph-mon[32593]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:57 smithi053 ceph-mon[32593]: mgrmap e15: x(active, since 92s), standbys: y, z 2024-04-26T01:29:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:57 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:29:57.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:57 smithi028 ceph-mon[28247]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:57.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:57 smithi028 ceph-mon[28247]: mgrmap e15: x(active, since 92s), standbys: y, z 2024-04-26T01:29:57.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:57 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:29:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:58 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2753002468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]: dispatch 2024-04-26T01:29:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:58 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]: dispatch 2024-04-26T01:29:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:58 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]': finished 2024-04-26T01:29:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:58 smithi053 ceph-mon[32593]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:29:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:58 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:29:58.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:58 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2753002468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]: dispatch 2024-04-26T01:29:58.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:58 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]: dispatch 2024-04-26T01:29:58.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:58 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]': finished 2024-04-26T01:29:58.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:58 smithi028 ceph-mon[28247]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:29:58.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:29:58.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:58 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2753002468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]: dispatch 2024-04-26T01:29:58.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:58 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]: dispatch 2024-04-26T01:29:58.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:58 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0017e56-53b2-4568-ab76-85ef9640ce85"}]': finished 2024-04-26T01:29:58.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:58 smithi169 ceph-mon[32851]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:29:58.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:58 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:29:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:59 smithi053 ceph-mon[32593]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:59.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:29:59 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2993892187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:29:59.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:59 smithi028 ceph-mon[28247]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:59.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:29:59 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2993892187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:29:59.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:59 smithi169 ceph-mon[32851]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:29:59.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:29:59 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2993892187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:30:00.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:00 smithi053 ceph-mon[32593]: overall HEALTH_OK 2024-04-26T01:30:00.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:00 smithi028 ceph-mon[28247]: overall HEALTH_OK 2024-04-26T01:30:00.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:00 smithi169 ceph-mon[32851]: overall HEALTH_OK 2024-04-26T01:30:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:01 smithi053 ceph-mon[32593]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:01.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:01 smithi028 ceph-mon[28247]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:01.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:01 smithi169 ceph-mon[32851]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:03.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:03 smithi028 ceph-mon[28247]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:03.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:03 smithi169 ceph-mon[32851]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:03 smithi053 ceph-mon[32593]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:05.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:05 smithi028 ceph-mon[28247]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:05.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:05 smithi169 ceph-mon[32851]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:05 smithi053 ceph-mon[32593]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:07 smithi053 ceph-mon[32593]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:07 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:30:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:07 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:07.728 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:07 smithi169 ceph-mon[32851]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:07.728 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:07 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:30:07.728 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:07 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:07 smithi028 ceph-mon[28247]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:07 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:30:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:07 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:08.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:08 smithi028 ceph-mon[28247]: Deploying daemon osd.0 on smithi028 2024-04-26T01:30:08.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:08 smithi169 ceph-mon[32851]: Deploying daemon osd.0 on smithi028 2024-04-26T01:30:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:08 smithi053 ceph-mon[32593]: Deploying daemon osd.0 on smithi028 2024-04-26T01:30:09.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:09 smithi028 ceph-mon[28247]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:09.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:09 smithi169 ceph-mon[32851]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:09 smithi053 ceph-mon[32593]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:11 smithi028 ceph-mon[28247]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:11.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:11 smithi169 ceph-mon[32851]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:11.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:11 smithi053 ceph-mon[32593]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:13.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:13 smithi028 ceph-mon[28247]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:13.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:13 smithi169 ceph-mon[32851]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:13.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:13 smithi053 ceph-mon[32593]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:15.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:15 smithi028 ceph-mon[28247]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:15.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:15 smithi169 ceph-mon[32851]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:15 smithi053 ceph-mon[32593]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:16.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:16 smithi028 ceph-mon[28247]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:16.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:16 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:16.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:16 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:16.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:16 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:16 smithi053 ceph-mon[32593]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:16 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:16 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:16 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:17.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:16 smithi169 ceph-mon[32851]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:17.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:16 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:17.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:16 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:17.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:16 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:18.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:18 smithi028 ceph-mon[28247]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:18.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:18 smithi053 ceph-mon[32593]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:18.991 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:18 smithi169 ceph-mon[32851]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:20.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:20.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:20.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:20.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:20.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:20.448 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-04-26T01:30:20.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.451+0000 7f6c9b400700 1 -- 172.21.15.28:0/2739424753 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6ca4062e80 con 0x7f6c8806ea20 2024-04-26T01:30:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6c8806ea20 msgr2=0x7f6c88070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6c8806ea20 0x7f6c88070ee0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400b2e0 tx=0x7f6c9401a040 comp rx=0 tx=0).stop 2024-04-26T01:30:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ca4105e50 msgr2=0x7f6ca419b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ca4105e50 0x7f6ca419b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e9c0 tx=0x7f6c9000ecd0 comp rx=0 tx=0).stop 2024-04-26T01:30:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 shutdown_connections 2024-04-26T01:30:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6c8806ea20 0x7f6c88070ee0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ca4108780 0x7f6ca41a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca4103520 0x7f6ca419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 --2- 172.21.15.28:0/2739424753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ca4105e50 0x7f6ca419b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 >> 172.21.15.28:0/2739424753 conn(0x7f6ca40fcf30 msgr2=0x7f6ca4102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:30:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 shutdown_connections 2024-04-26T01:30:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:30:20.455+0000 7f6ca9c00700 1 -- 172.21.15.28:0/2739424753 wait complete. 2024-04-26T01:30:21.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:20 smithi169 ceph-mon[32851]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:20 smithi169 ceph-mon[32851]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:30:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:21.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:20 smithi028 ceph-mon[28247]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:21.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:20 smithi028 ceph-mon[28247]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:30:21.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:21.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:21.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:20 smithi053 ceph-mon[32593]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:20 smithi053 ceph-mon[32593]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:30:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:21.222 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T01:30:21.226 INFO:tasks.cephadm:Deploying osd.1 on smithi053 with /dev/vg_nvme/lv_4... 2024-04-26T01:30:21.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:30:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:30:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:21 smithi053 ceph-mon[32593]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:30:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:21 smithi053 ceph-mon[32593]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:30:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:21 smithi053 ceph-mon[32593]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:30:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:21 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:22.271 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:21 smithi028 ceph-mon[28247]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:30:22.271 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:21 smithi028 ceph-mon[28247]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:30:22.271 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:21 smithi028 ceph-mon[28247]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:30:22.271 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:21 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:22.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:21 smithi169 ceph-mon[32851]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:30:22.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:21 smithi169 ceph-mon[32851]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:30:22.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:21 smithi169 ceph-mon[32851]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:30:22.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:21 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:22.930 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:30:22 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0[40623]: 2024-04-26T01:30:22.571+0000 7f11c2e00700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T01:30:22.930 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:30:22 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0[40623]: 2024-04-26T01:30:22.597+0000 7f11b8200700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:30:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:22 smithi053 ceph-mon[32593]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:22 smithi053 ceph-mon[32593]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:30:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:22 smithi053 ceph-mon[32593]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:30:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:22 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:22 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:22 smithi053 ceph-mon[32593]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' 2024-04-26T01:30:23.274 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:30:23.274 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:30:23.274 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-26T01:30:23.274 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-26T01:30:23.274 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00841196 s, 1.2 GB/s 2024-04-26T01:30:23.274 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-26T01:30:23.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:22 smithi028 ceph-mon[28247]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:23.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:22 smithi028 ceph-mon[28247]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:30:23.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:22 smithi028 ceph-mon[28247]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:30:23.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:22 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:23.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:22 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:23.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:22 smithi028 ceph-mon[28247]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' 2024-04-26T01:30:23.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:22 smithi169 ceph-mon[32851]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:23.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:22 smithi169 ceph-mon[32851]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:30:23.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:22 smithi169 ceph-mon[32851]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:30:23.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:22 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:23.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:22 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:23.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:22 smithi169 ceph-mon[32851]: from='osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886]' entity='osd.0' 2024-04-26T01:30:23.903 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_4 2024-04-26T01:30:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:30:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:23 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:30:24.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:23 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:30:24.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:24.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:23 smithi053 ceph-mon[32593]: osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] boot 2024-04-26T01:30:24.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:23 smithi053 ceph-mon[32593]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:30:24.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:24.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:23 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:30:24.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:23 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:30:24.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:24.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:23 smithi028 ceph-mon[28247]: osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] boot 2024-04-26T01:30:24.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:23 smithi028 ceph-mon[28247]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:30:24.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:24.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:23 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:30:24.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:23 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:30:24.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:24.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:23 smithi169 ceph-mon[32851]: osd.0 [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] boot 2024-04-26T01:30:24.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:23 smithi169 ceph-mon[32851]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:30:24.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:30:25.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:24 smithi053 ceph-mon[32593]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:25.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:24 smithi053 ceph-mon[32593]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:30:25.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:24 smithi028 ceph-mon[28247]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:25.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:24 smithi028 ceph-mon[28247]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:30:25.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:24 smithi169 ceph-mon[32851]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:30:25.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:24 smithi169 ceph-mon[32851]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:30:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 -- 172.21.15.53:0/348947307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248102690 msgr2=0x7fb248109f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/348947307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248102690 0x7fb248109f30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb24000b0a0 tx=0x7fb24000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:30:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 -- 172.21.15.53:0/348947307 shutdown_connections 2024-04-26T01:30:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/348947307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248102690 0x7fb248109f30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/348947307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb248103070 0x7fb248103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/348947307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248101cd0 0x7fb248102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 -- 172.21.15.53:0/348947307 >> 172.21.15.53:0/348947307 conn(0x7fb2480fc780 msgr2=0x7fb2480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:30:25.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 -- 172.21.15.53:0/348947307 shutdown_connections 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 -- 172.21.15.53:0/348947307 wait complete. 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.621+0000 7fb24e9f9700 1 Processor -- start 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24e9f9700 1 -- start start 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24e9f9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248101cd0 0x7fb248105a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24e9f9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb248102690 0x7fb2481040c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24e9f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248103070 0x7fb248104600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24e9f9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb24810d540 con 0x7fb248101cd0 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24e9f9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb24810d3c0 con 0x7fb248103070 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24e9f9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb24810d6c0 con 0x7fb248102690 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248103070 0x7fb248104600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248103070 0x7fb248104600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47338/0 (socket says 172.21.15.53:47338) 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 -- 172.21.15.53:0/3275585712 learned_addr learned my addr 172.21.15.53:0/3275585712 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb2477fe700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb248102690 0x7fb2481040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:30:25.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 -- 172.21.15.53:0/3275585712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb248102690 msgr2=0x7fb2481040c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb248102690 0x7fb2481040c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 -- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248101cd0 msgr2=0x7fb248105a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248101cd0 0x7fb248105a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 -- 172.21.15.53:0/3275585712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb248104eb0 con 0x7fb248103070 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.622+0000 7fb24cf96700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248103070 0x7fb248104600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb24000ced0 tx=0x7fb24000cfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.623+0000 7fb2457fa700 1 -- 172.21.15.53:0/3275585712 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240021070 con 0x7fb248103070 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.623+0000 7fb2457fa700 1 -- 172.21.15.53:0/3275585712 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb24001d900 con 0x7fb248103070 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.623+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2481a2da0 con 0x7fb248103070 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.623+0000 7fb2457fa700 1 -- 172.21.15.53:0/3275585712 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240005420 con 0x7fb248103070 2024-04-26T01:30:25.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.623+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2481a3250 con 0x7fb248103070 2024-04-26T01:30:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.624+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb248069730 con 0x7fb248103070 2024-04-26T01:30:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.625+0000 7fb2457fa700 1 -- 172.21.15.53:0/3275585712 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fb240024070 con 0x7fb248103070 2024-04-26T01:30:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.625+0000 7fb2457fa700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2340a3c30 0x7fb2340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:30:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.626+0000 7fb2457fa700 1 -- 172.21.15.53:0/3275585712 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fb24000f020 con 0x7fb248103070 2024-04-26T01:30:25.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.626+0000 7fb247fff700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2340a3c30 0x7fb2340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:30:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.627+0000 7fb247fff700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2340a3c30 0x7fb2340a60f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb24806afb0 tx=0x7fb230005ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:30:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.629+0000 7fb2457fa700 1 -- 172.21.15.53:0/3275585712 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb24002b020 con 0x7fb248103070 2024-04-26T01:30:25.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:25.847+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb2480fe910 con 0x7fb2340a3c30 2024-04-26T01:30:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:30:26.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:30:26.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:26.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:30:26.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:30:26.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:26.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:30:26.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:30:26.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:27.365 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:27.365 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:30:27.365 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi028 2024-04-26T01:30:27.365 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.365 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.365 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:30:27.366 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-26T01:30:27.366 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.366 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:27.366 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:27.366 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:27 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi028 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:27.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:27 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:27.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:30:27.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi028 2024-04-26T01:30:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:30:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-26T01:30:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:27 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:29.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:29 smithi169 ceph-mon[32851]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:29.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:29 smithi169 ceph-mon[32851]: from='client.? 172.21.15.53:0/3945677729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]: dispatch 2024-04-26T01:30:29.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:29 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]: dispatch 2024-04-26T01:30:29.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:29 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]': finished 2024-04-26T01:30:29.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:29 smithi169 ceph-mon[32851]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:30:29.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:29 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:29 smithi053 ceph-mon[32593]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:29 smithi053 ceph-mon[32593]: from='client.? 172.21.15.53:0/3945677729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]: dispatch 2024-04-26T01:30:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:29 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]: dispatch 2024-04-26T01:30:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:29 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]': finished 2024-04-26T01:30:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:29 smithi053 ceph-mon[32593]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:30:29.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:29 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:29.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:29 smithi028 ceph-mon[28247]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:29.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:29 smithi028 ceph-mon[28247]: from='client.? 172.21.15.53:0/3945677729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]: dispatch 2024-04-26T01:30:29.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:29 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]: dispatch 2024-04-26T01:30:29.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:29 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bfc13cc-6c4d-45ec-8b43-6a346569692f"}]': finished 2024-04-26T01:30:29.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:29 smithi028 ceph-mon[28247]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:30:29.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:29 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:30.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:30 smithi169 ceph-mon[32851]: from='client.? 172.21.15.53:0/3707663543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:30:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:30 smithi053 ceph-mon[32593]: from='client.? 172.21.15.53:0/3707663543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:30:30.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:30 smithi028 ceph-mon[28247]: from='client.? 172.21.15.53:0/3707663543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:30:31.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:31 smithi169 ceph-mon[32851]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:31 smithi053 ceph-mon[32593]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:31.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:31 smithi028 ceph-mon[28247]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:33.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:33 smithi053 ceph-mon[32593]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:33.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:33 smithi169 ceph-mon[32851]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:33 smithi028 ceph-mon[28247]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:35.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:35 smithi053 ceph-mon[32593]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:35.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:35 smithi028 ceph-mon[28247]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:35.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:35 smithi169 ceph-mon[32851]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:36.363 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:30:36.363 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:36.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:30:36.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:36.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:30:36.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:37.182 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:37 smithi053 ceph-mon[32593]: Deploying daemon osd.1 on smithi053 2024-04-26T01:30:37.182 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:37 smithi053 ceph-mon[32593]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:37.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:37 smithi169 ceph-mon[32851]: Deploying daemon osd.1 on smithi053 2024-04-26T01:30:37.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:37 smithi169 ceph-mon[32851]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:37.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:37 smithi028 ceph-mon[28247]: Deploying daemon osd.1 on smithi053 2024-04-26T01:30:37.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:37 smithi028 ceph-mon[28247]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:39.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:39 smithi169 ceph-mon[32851]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:39.409 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:39 smithi053 ceph-mon[32593]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:39.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:39 smithi028 ceph-mon[28247]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:41.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:41 smithi169 ceph-mon[32851]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:41.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:41 smithi053 ceph-mon[32593]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:41.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:41 smithi028 ceph-mon[28247]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:42.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:42.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:42.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:42.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:42.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:42.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:43.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:43 smithi169 ceph-mon[32851]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:43 smithi053 ceph-mon[32593]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:43.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:43 smithi028 ceph-mon[28247]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:44.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.797+0000 7fb2457fa700 1 -- 172.21.15.53:0/3275585712 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb2480fe910 con 0x7fb2340a3c30 2024-04-26T01:30:44.794 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 1 on host 'smithi053' 2024-04-26T01:30:44.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2340a3c30 msgr2=0x7fb2340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:44.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2340a3c30 0x7fb2340a60f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb24806afb0 tx=0x7fb230005ca0 comp rx=0 tx=0).stop 2024-04-26T01:30:44.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248103070 msgr2=0x7fb248104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:44.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248103070 0x7fb248104600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb24000ced0 tx=0x7fb24000cfb0 comp rx=0 tx=0).stop 2024-04-26T01:30:44.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 shutdown_connections 2024-04-26T01:30:44.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2340a3c30 0x7fb2340a60f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:44.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb248103070 0x7fb248104600 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:44.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb248101cd0 0x7fb248105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:44.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 --2- 172.21.15.53:0/3275585712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb248102690 0x7fb2481040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:44.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 >> 172.21.15.53:0/3275585712 conn(0x7fb2480fc780 msgr2=0x7fb2480fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:30:44.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 shutdown_connections 2024-04-26T01:30:44.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:30:44.800+0000 7fb24e9f9700 1 -- 172.21.15.53:0/3275585712 wait complete. 2024-04-26T01:30:45.099 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:45.099 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.099 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.100 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:45.100 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:45.100 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.100 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.100 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.100 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:45.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:45.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:45.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:45.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:45.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:45.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:45.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:45.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:45.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:30:45.435 DEBUG:teuthology.orchestra.run.smithi053:osd.1> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T01:30:45.437 INFO:tasks.cephadm:Deploying osd.2 on smithi169 with /dev/vg_nvme/lv_4... 2024-04-26T01:30:45.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:30:45.687 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:30:45.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:45 smithi053 ceph-mon[32593]: from='osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:30:45.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:45 smithi053 ceph-mon[32593]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:30:46.152 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:45 smithi169 ceph-mon[32851]: from='osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:30:46.152 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:45 smithi169 ceph-mon[32851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:30:46.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:45 smithi028 ceph-mon[28247]: from='osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:30:46.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:45 smithi028 ceph-mon[28247]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:30:46.926 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:46 smithi053 ceph-mon[32593]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:46.927 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:46 smithi053 ceph-mon[32593]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:30:46.927 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:46 smithi053 ceph-mon[32593]: from='osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:30:46.927 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:46 smithi053 ceph-mon[32593]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:30:46.927 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:46 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:46.927 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:46 smithi053 ceph-mon[32593]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:30:46.927 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:30:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1[37222]: 2024-04-26T01:30:46.890+0000 7fd5468eb700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T01:30:46.927 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:30:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1[37222]: 2024-04-26T01:30:46.894+0000 7fd5426de700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:30:47.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:46 smithi169 ceph-mon[32851]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:47.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:46 smithi169 ceph-mon[32851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:30:47.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:46 smithi169 ceph-mon[32851]: from='osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:30:47.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:46 smithi169 ceph-mon[32851]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:30:47.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:46 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:47.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:46 smithi169 ceph-mon[32851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:30:47.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:46 smithi028 ceph-mon[28247]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:47.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:46 smithi028 ceph-mon[28247]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:30:47.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:46 smithi028 ceph-mon[28247]: from='osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:30:47.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:46 smithi028 ceph-mon[28247]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:30:47.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:46 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:47.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:46 smithi028 ceph-mon[28247]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:30:47.319 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:30:47.319 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:30:47.319 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-04-26T01:30:47.319 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-04-26T01:30:47.319 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117243 s, 894 MB/s 2024-04-26T01:30:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-04-26T01:30:47.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi169:vg_nvme/lv_4 2024-04-26T01:30:48.128 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:30:48.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:47 smithi169 ceph-mon[32851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:30:48.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:47 smithi169 ceph-mon[32851]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:30:48.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:48.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:47 smithi053 ceph-mon[32593]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:30:48.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:47 smithi053 ceph-mon[32593]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:30:48.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:48.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:48.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:47 smithi028 ceph-mon[28247]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:30:48.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:47 smithi028 ceph-mon[28247]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:30:48.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:48.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:49.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:30:49.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:30:49.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:49.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:49.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] boot 2024-04-26T01:30:49.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:30:49.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:49.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi053 2024-04-26T01:30:49.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:30:49.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi053 to 17093M 2024-04-26T01:30:49.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:49.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:49.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] boot 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi053 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:30:49.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi053 to 17093M 2024-04-26T01:30:49.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:49.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:49.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:30:49.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:30:49.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: osd.1 [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] boot 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi053 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi053 to 17093M 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:49.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:30:49.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:30:49.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.678+0000 7f9e2e0af700 1 -- 172.21.15.169:0/3267678685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28102aa0 msgr2=0x7f9e28102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:49.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.678+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/3267678685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28102aa0 0x7f9e28102ec0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009990 tx=0x7f9e18009ca0 comp rx=0 tx=0).stop 2024-04-26T01:30:49.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 -- 172.21.15.169:0/3267678685 shutdown_connections 2024-04-26T01:30:49.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/3267678685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28104640 0x7f9e2810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:49.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/3267678685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28102aa0 0x7f9e28102ec0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:49.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/3267678685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e28103ca0 0x7f9e28104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:49.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 -- 172.21.15.169:0/3267678685 >> 172.21.15.169:0/3267678685 conn(0x7f9e280fe040 msgr2=0x7f9e28100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:30:49.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 -- 172.21.15.169:0/3267678685 shutdown_connections 2024-04-26T01:30:49.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 -- 172.21.15.169:0/3267678685 wait complete. 2024-04-26T01:30:49.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 Processor -- start 2024-04-26T01:30:49.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 -- start start 2024-04-26T01:30:49.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102aa0 0x7f9e28198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:30:49.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103ca0 0x7f9e28199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:30:49.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e28104640 0x7f9e281a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:30:49.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2810e610 con 0x7f9e28102aa0 2024-04-26T01:30:49.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2810e490 con 0x7f9e28104640 2024-04-26T01:30:49.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e2e0af700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2810e790 con 0x7f9e28103ca0 2024-04-26T01:30:49.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e1ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103ca0 0x7f9e28199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:30:49.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e1ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103ca0 0x7f9e28199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54228/0 (socket says 172.21.15.169:54228) 2024-04-26T01:30:49.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.679+0000 7f9e1ffff700 1 -- 172.21.15.169:0/590960997 learned_addr learned my addr 172.21.15.169:0/590960997 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:30:49.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e27fff700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e28104640 0x7f9e281a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:30:49.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e1ffff700 1 -- 172.21.15.169:0/590960997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e28104640 msgr2=0x7f9e281a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:49.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e1ffff700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e28104640 0x7f9e281a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:49.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e1ffff700 1 -- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102aa0 msgr2=0x7f9e28198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:30:49.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e1ffff700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102aa0 0x7f9e28198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:30:49.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e1ffff700 1 -- 172.21.15.169:0/590960997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e281a0950 con 0x7f9e28103ca0 2024-04-26T01:30:49.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e27fff700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e28104640 0x7f9e281a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:30:49.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e1ffff700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103ca0 0x7f9e28199290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000b530 tx=0x7f9e1000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:30:49.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e257fa700 1 -- 172.21.15.169:0/590960997 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10015020 con 0x7f9e28103ca0 2024-04-26T01:30:49.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e257fa700 1 -- 172.21.15.169:0/590960997 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e10011ce0 con 0x7f9e28103ca0 2024-04-26T01:30:49.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e281a0c40 con 0x7f9e28103ca0 2024-04-26T01:30:49.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e257fa700 1 -- 172.21.15.169:0/590960997 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10010d20 con 0x7f9e28103ca0 2024-04-26T01:30:49.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.680+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e281a1130 con 0x7f9e28103ca0 2024-04-26T01:30:49.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.681+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e28069730 con 0x7f9e28103ca0 2024-04-26T01:30:49.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.682+0000 7f9e257fa700 1 -- 172.21.15.169:0/590960997 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f9e10018020 con 0x7f9e28103ca0 2024-04-26T01:30:49.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.683+0000 7f9e257fa700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9e140a3bb0 0x7f9e140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:30:49.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.683+0000 7f9e257fa700 1 -- 172.21.15.169:0/590960997 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f9e100ce940 con 0x7f9e28103ca0 2024-04-26T01:30:49.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.683+0000 7f9e277fe700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9e140a3bb0 0x7f9e140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:30:49.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.685+0000 7f9e277fe700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9e140a3bb0 0x7f9e140a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800b5c0 tx=0x7f9e18003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:30:49.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.687+0000 7f9e257fa700 1 -- 172.21.15.169:0/590960997 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e1001f020 con 0x7f9e28103ca0 2024-04-26T01:30:49.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:30:49.898+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9e28062e80 con 0x7f9e140a3bb0 2024-04-26T01:30:50.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:49 smithi169 ceph-mon[32851]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:30:50.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:30:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:30:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:49 smithi053 ceph-mon[32593]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:30:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:30:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:30:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:50.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:49 smithi028 ceph-mon[28247]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:30:50.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:30:50.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:30:50.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:30:51.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:50 smithi053 ceph-mon[32593]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:51.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:50 smithi053 ceph-mon[32593]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:30:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:50 smithi028 ceph-mon[28247]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:50 smithi028 ceph-mon[28247]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:30:51.326 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:50 smithi169 ceph-mon[32851]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:51.327 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:50 smithi169 ceph-mon[32851]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:30:53.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:52 smithi169 ceph-mon[32851]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:52 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/1568975168' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]: dispatch 2024-04-26T01:30:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:52 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]: dispatch 2024-04-26T01:30:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:52 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]': finished 2024-04-26T01:30:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:52 smithi169 ceph-mon[32851]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:30:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:52 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:30:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:52 smithi053 ceph-mon[32593]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:52 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/1568975168' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]: dispatch 2024-04-26T01:30:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:52 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]: dispatch 2024-04-26T01:30:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:52 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]': finished 2024-04-26T01:30:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:52 smithi053 ceph-mon[32593]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:30:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:52 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:30:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:52 smithi028 ceph-mon[28247]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:52 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/1568975168' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]: dispatch 2024-04-26T01:30:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:52 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]: dispatch 2024-04-26T01:30:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:52 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "147b31a0-5128-4e08-a252-ada0832abff2"}]': finished 2024-04-26T01:30:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:52 smithi028 ceph-mon[28247]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:30:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:52 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:30:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:53 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/1176292035' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:30:54.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:53 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/1176292035' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:30:54.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:53 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/1176292035' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:30:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:54 smithi053 ceph-mon[32593]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:55.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:54 smithi028 ceph-mon[28247]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:55.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:54 smithi169 ceph-mon[32851]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:56 smithi053 ceph-mon[32593]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:57.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:56 smithi169 ceph-mon[32851]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:57.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:56 smithi028 ceph-mon[28247]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:59.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:58 smithi053 ceph-mon[32593]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:59.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:58 smithi169 ceph-mon[32851]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:30:59.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:58 smithi028 ceph-mon[28247]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:00.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:59 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:31:00.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:59 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:00.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:30:59 smithi169 ceph-mon[32851]: Deploying daemon osd.2 on smithi169 2024-04-26T01:31:00.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:59 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:31:00.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:59 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:00.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:30:59 smithi053 ceph-mon[32593]: Deploying daemon osd.2 on smithi169 2024-04-26T01:31:00.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:59 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:31:00.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:59 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:00.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:30:59 smithi028 ceph-mon[28247]: Deploying daemon osd.2 on smithi169 2024-04-26T01:31:01.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:00 smithi169 ceph-mon[32851]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:01.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:00 smithi053 ceph-mon[32593]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:01.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:00 smithi028 ceph-mon[28247]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:03.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:02 smithi028 ceph-mon[28247]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:03.325 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:02 smithi169 ceph-mon[32851]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:02 smithi053 ceph-mon[32593]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:05.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:05 smithi028 ceph-mon[28247]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:05.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:05 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:05.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:05 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:05.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:05 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:05.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:05 smithi169 ceph-mon[32851]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:05.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:05 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:05.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:05 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:05.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:05 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:05 smithi053 ceph-mon[32593]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:05 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:05 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:05 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:07.755 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:07 smithi053 ceph-mon[32593]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:07 smithi028 ceph-mon[28247]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:07.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:07 smithi169 ceph-mon[32851]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:08.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.040+0000 7f9e257fa700 1 -- 172.21.15.169:0/590960997 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9e28062e80 con 0x7f9e140a3bb0 2024-04-26T01:31:08.035 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 2 on host 'smithi169' 2024-04-26T01:31:08.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9e140a3bb0 msgr2=0x7f9e140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:08.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9e140a3bb0 0x7f9e140a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800b5c0 tx=0x7f9e18003860 comp rx=0 tx=0).stop 2024-04-26T01:31:08.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103ca0 msgr2=0x7f9e28199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103ca0 0x7f9e28199290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000b530 tx=0x7f9e1000b8f0 comp rx=0 tx=0).stop 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 shutdown_connections 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9e140a3bb0 0x7f9e140a6070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e28104640 0x7f9e281a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102aa0 0x7f9e28198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 --2- 172.21.15.169:0/590960997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103ca0 0x7f9e28199290 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 >> 172.21.15.169:0/590960997 conn(0x7f9e280fe040 msgr2=0x7f9e281070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 shutdown_connections 2024-04-26T01:31:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:31:08.042+0000 7f9e2e0af700 1 -- 172.21.15.169:0/590960997 wait complete. 2024-04-26T01:31:08.686 DEBUG:teuthology.orchestra.run.smithi169:osd.2> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T01:31:08.689 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-04-26T01:31:08.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:31:08.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:31:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:08 smithi169 ceph-mon[32851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:08.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:31:08.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:08 smithi053 ceph-mon[32593]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:31:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:31:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:08 smithi028 ceph-mon[28247]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:31:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:31:10.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:09 smithi028 ceph-mon[28247]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:31:10.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:09 smithi028 ceph-mon[28247]: from='osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:31:10.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:09 smithi028 ceph-mon[28247]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:31:10.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:10.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:09 smithi028 ceph-mon[28247]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:31:10.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:09 smithi169 ceph-mon[32851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:31:10.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:09 smithi169 ceph-mon[32851]: from='osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:31:10.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:09 smithi169 ceph-mon[32851]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:31:10.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:10.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:09 smithi169 ceph-mon[32851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:31:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:09 smithi053 ceph-mon[32593]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:31:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:09 smithi053 ceph-mon[32593]: from='osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:31:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:09 smithi053 ceph-mon[32593]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:31:10.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:10.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:09 smithi053 ceph-mon[32593]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:31:10.760 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2[38152]: 2024-04-26T01:31:10.413+0000 7eff0ab86700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T01:31:10.760 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2[38152]: 2024-04-26T01:31:10.417+0000 7eff02971700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:31:11.153 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:11 smithi169 ceph-mon[32851]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:11.153 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:11 smithi169 ceph-mon[32851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:31:11.153 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:11 smithi169 ceph-mon[32851]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:31:11.153 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:11.153 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:11.153 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:11 smithi169 ceph-mon[32851]: from='osd.2 ' entity='osd.2' 2024-04-26T01:31:11.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:11 smithi028 ceph-mon[28247]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:11.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:11 smithi028 ceph-mon[28247]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:31:11.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:11 smithi028 ceph-mon[28247]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:31:11.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:11.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:11.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:11 smithi028 ceph-mon[28247]: from='osd.2 ' entity='osd.2' 2024-04-26T01:31:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:11 smithi053 ceph-mon[32593]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:11 smithi053 ceph-mon[32593]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:31:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:11 smithi053 ceph-mon[32593]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:31:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:11 smithi053 ceph-mon[32593]: from='osd.2 ' entity='osd.2' 2024-04-26T01:31:12.180 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:31:12.180 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:31:12.180 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-26T01:31:12.180 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-26T01:31:12.180 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0324584 s, 323 MB/s 2024-04-26T01:31:12.180 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-26T01:31:12.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:31:12.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:31:12.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:12.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] boot 2024-04-26T01:31:12.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:31:12.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:12.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi169 2024-04-26T01:31:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:31:12.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] boot 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi169 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:31:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:31:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:31:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: osd.2 [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] boot 2024-04-26T01:31:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:31:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:31:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi169 2024-04-26T01:31:12.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:12.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:12.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:12.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:12.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:31:12.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-04-26T01:31:13.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:13 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi169 to 17088M 2024-04-26T01:31:13.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:13 smithi028 ceph-mon[28247]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:13.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:31:13.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:13 smithi028 ceph-mon[28247]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:31:13.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:31:13.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:13 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi169 to 17088M 2024-04-26T01:31:13.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:13 smithi169 ceph-mon[32851]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:13.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:31:13.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:13 smithi169 ceph-mon[32851]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:31:13.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:31:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:31:13.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:13 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi169 to 17088M 2024-04-26T01:31:13.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:13 smithi053 ceph-mon[32593]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:31:13.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:31:13.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:13 smithi053 ceph-mon[32593]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:31:13.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:31:13.868 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:31:13 smithi028 sudo[45200]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:31:13.869 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:31:13 smithi028 sudo[45200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:13.869 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:31:13 smithi028 sudo[45200]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:13.869 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:31:13 smithi028 sudo[45200]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.119 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 sudo[45220]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:31:14.119 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 sudo[45220]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.119 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 sudo[45220]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.119 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 sudo[45220]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.119 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:31:13 smithi028 sudo[45205]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:31:14.119 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:31:13 smithi028 sudo[45205]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.119 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:31:13 smithi028 sudo[45205]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.119 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:31:13 smithi028 sudo[45205]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.212 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:31:13 smithi053 sudo[41285]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:31:14.213 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:31:13 smithi053 sudo[41285]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.213 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:31:13 smithi053 sudo[41285]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.213 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:31:13 smithi053 sudo[41285]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.213 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:31:13 smithi053 sudo[41288]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:31:14.213 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:31:13 smithi053 sudo[41288]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.213 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:31:13 smithi053 sudo[41288]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.213 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:31:13 smithi053 sudo[41288]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42065]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:31:14.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42065]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42065]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42065]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42068]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:31:14.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42068]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42068]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42068]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.409 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42059]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:31:14.409 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42059]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.409 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42059]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.409 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42059]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.409 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42062]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:31:14.410 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42062]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.410 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42062]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.410 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:31:14 smithi169 sudo[42062]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 sudo[41291]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:31:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 sudo[41291]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 sudo[41291]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 sudo[41291]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 sudo[41294]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:31:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 sudo[41294]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 sudo[41294]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 sudo[41294]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 sudo[45223]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:31:14.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 sudo[45223]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:31:14.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 sudo[45223]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:31:14.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 sudo[45223]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:31:14.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:31:14.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:31:14.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:14.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:14 smithi028 ceph-mon[28247]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:14.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:14 smithi169 ceph-mon[32851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:31:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:31:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:31:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:14 smithi053 ceph-mon[32593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:31:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.656+0000 7f33fda00700 1 -- 172.21.15.28:0/2107705418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f80754a0 msgr2=0x7f33f8075920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.656+0000 7f33fda00700 1 --2- 172.21.15.28:0/2107705418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f80754a0 0x7f33f8075920 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009990 tx=0x7f33e4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:31:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.656+0000 7f33fda00700 1 -- 172.21.15.28:0/2107705418 shutdown_connections 2024-04-26T01:31:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.656+0000 7f33fda00700 1 --2- 172.21.15.28:0/2107705418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33f810a990 0x7f33f810ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.656+0000 7f33fda00700 1 --2- 172.21.15.28:0/2107705418 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33f8076a70 0x7f33f8074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.656+0000 7f33fda00700 1 --2- 172.21.15.28:0/2107705418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f80754a0 0x7f33f8075920 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.656+0000 7f33fda00700 1 -- 172.21.15.28:0/2107705418 >> 172.21.15.28:0/2107705418 conn(0x7f33f80fe060 msgr2=0x7f33f81004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.657+0000 7f33fda00700 1 -- 172.21.15.28:0/2107705418 shutdown_connections 2024-04-26T01:31:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.657+0000 7f33fda00700 1 -- 172.21.15.28:0/2107705418 wait complete. 2024-04-26T01:31:15.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.658+0000 7f33fda00700 1 Processor -- start 2024-04-26T01:31:15.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.658+0000 7f33fda00700 1 -- start start 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.658+0000 7f33fda00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33f80754a0 0x7f33f819ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.658+0000 7f33fda00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33f8076a70 0x7f33f819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.658+0000 7f33fda00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f810a990 0x7f33f81a2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.658+0000 7f33fda00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33f81107a0 con 0x7f33f810a990 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.658+0000 7f33fda00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f33f8110620 con 0x7f33f80754a0 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.658+0000 7f33fda00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f33f8110920 con 0x7f33f8076a70 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33f80754a0 0x7f33f819ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33f8076a70 0x7f33f819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f810a990 0x7f33f81a2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33f8076a70 0x7f33f819b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:45418/0 (socket says 172.21.15.28:45418) 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33f80754a0 0x7f33f819ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38626/0 (socket says 172.21.15.28:38626) 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 -- 172.21.15.28:0/2514424035 learned_addr learned my addr 172.21.15.28:0/2514424035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 -- 172.21.15.28:0/2514424035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33f80754a0 msgr2=0x7f33f819ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33f80754a0 0x7f33f819ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 -- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f810a990 msgr2=0x7f33f81a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f810a990 0x7f33f81a2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 -- 172.21.15.28:0/2514424035 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f81a2ac0 con 0x7f33f8076a70 2024-04-26T01:31:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6a00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33f80754a0 0x7f33f819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:31:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.659+0000 7f33f6000700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33f8076a70 0x7f33f819b3a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f33e400b760 tx=0x7f33e40057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.660+0000 7f33ef400700 1 -- 172.21.15.28:0/2514424035 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e4027070 con 0x7f33f8076a70 2024-04-26T01:31:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.660+0000 7f33ef400700 1 -- 172.21.15.28:0/2514424035 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e4022470 con 0x7f33f8076a70 2024-04-26T01:31:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.660+0000 7f33ef400700 1 -- 172.21.15.28:0/2514424035 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e4017650 con 0x7f33f8076a70 2024-04-26T01:31:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.660+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f81a2d50 con 0x7f33f8076a70 2024-04-26T01:31:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.660+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f81a3150 con 0x7f33f8076a70 2024-04-26T01:31:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.661+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33f8069730 con 0x7f33f8076a70 2024-04-26T01:31:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.663+0000 7f33ef400700 1 -- 172.21.15.28:0/2514424035 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f33e402a070 con 0x7f33f8076a70 2024-04-26T01:31:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.664+0000 7f33ef400700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f33dc0a3c30 0x7f33dc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.665+0000 7f33ef400700 1 -- 172.21.15.28:0/2514424035 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f33e40cffd0 con 0x7f33f8076a70 2024-04-26T01:31:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.665+0000 7f33f6a00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f33dc0a3c30 0x7f33dc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:15.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.665+0000 7f33f6a00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f33dc0a3c30 0x7f33dc0a60f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f33f8076e50 tx=0x7f33e80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.667+0000 7f33ef400700 1 -- 172.21.15.28:0/2514424035 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33e4013070 con 0x7f33f8076a70 2024-04-26T01:31:15.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:15 smithi028 ceph-mon[28247]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:31:15.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:15.907+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f33f8062e80 con 0x7f33dc0a3c30 2024-04-26T01:31:15.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:15 smithi169 ceph-mon[32851]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:31:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:15 smithi053 ceph-mon[32593]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:31:16.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:16.625+0000 7f33ef400700 1 -- 172.21.15.28:0/2514424035 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f33e4009d10 con 0x7f33f8076a70 2024-04-26T01:31:16.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:16 smithi028 ceph-mon[28247]: pgmap v97: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:16.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:16 smithi028 ceph-mon[28247]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:31:16.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:16 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:31:16.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:16 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:31:16.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:16 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:16.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:16 smithi169 ceph-mon[32851]: pgmap v97: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:16.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:16 smithi169 ceph-mon[32851]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:31:16.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:16 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:31:16.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:16 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:31:16.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:16 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:16 smithi053 ceph-mon[32593]: pgmap v97: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:16 smithi053 ceph-mon[32593]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:31:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:16 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:31:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:16 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:31:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:16 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:17.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:17 smithi169 ceph-mon[32851]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-26T01:31:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:17 smithi028 ceph-mon[28247]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-26T01:31:17.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:17 smithi053 ceph-mon[32593]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-26T01:31:18.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:18 smithi028 ceph-mon[28247]: pgmap v98: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:18.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:18 smithi169 ceph-mon[32851]: pgmap v98: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:18.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:18 smithi053 ceph-mon[32593]: pgmap v98: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:20 smithi053 ceph-mon[32593]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 6.3 KiB/s rd, 336 KiB/s wr, 21 op/s 2024-04-26T01:31:20.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:20 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3657471687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0929291f-f56e-4c6b-813d-886624278d0e"}]: dispatch 2024-04-26T01:31:20.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:20 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3657471687' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0929291f-f56e-4c6b-813d-886624278d0e"}]': finished 2024-04-26T01:31:20.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:20 smithi053 ceph-mon[32593]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T01:31:20.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:21.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:20 smithi028 ceph-mon[28247]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 6.3 KiB/s rd, 336 KiB/s wr, 21 op/s 2024-04-26T01:31:21.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:20 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3657471687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0929291f-f56e-4c6b-813d-886624278d0e"}]: dispatch 2024-04-26T01:31:21.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:20 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3657471687' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0929291f-f56e-4c6b-813d-886624278d0e"}]': finished 2024-04-26T01:31:21.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:20 smithi028 ceph-mon[28247]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T01:31:21.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:21.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:20 smithi169 ceph-mon[32851]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 6.3 KiB/s rd, 336 KiB/s wr, 21 op/s 2024-04-26T01:31:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:20 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3657471687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0929291f-f56e-4c6b-813d-886624278d0e"}]: dispatch 2024-04-26T01:31:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:20 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3657471687' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0929291f-f56e-4c6b-813d-886624278d0e"}]': finished 2024-04-26T01:31:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:20 smithi169 ceph-mon[32851]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T01:31:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:21 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3405479919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:31:22.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:21 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3405479919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:31:22.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:21 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3405479919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:31:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:22 smithi053 ceph-mon[32593]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 240 KiB/s wr, 15 op/s 2024-04-26T01:31:23.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:22 smithi028 ceph-mon[28247]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 240 KiB/s wr, 15 op/s 2024-04-26T01:31:23.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:22 smithi169 ceph-mon[32851]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 240 KiB/s wr, 15 op/s 2024-04-26T01:31:24.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:24 smithi028 ceph-mon[28247]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 187 KiB/s wr, 11 op/s 2024-04-26T01:31:25.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:24 smithi169 ceph-mon[32851]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 187 KiB/s wr, 11 op/s 2024-04-26T01:31:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:24 smithi053 ceph-mon[32593]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 187 KiB/s wr, 11 op/s 2024-04-26T01:31:27.018 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:26 smithi053 ceph-mon[32593]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 169 KiB/s wr, 10 op/s 2024-04-26T01:31:27.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:26 smithi028 ceph-mon[28247]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 169 KiB/s wr, 10 op/s 2024-04-26T01:31:27.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:26 smithi169 ceph-mon[32851]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 169 KiB/s wr, 10 op/s 2024-04-26T01:31:29.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:28 smithi028 ceph-mon[28247]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 169 KiB/s wr, 10 op/s 2024-04-26T01:31:29.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:28 smithi169 ceph-mon[32851]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 169 KiB/s wr, 10 op/s 2024-04-26T01:31:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:28 smithi053 ceph-mon[32593]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 169 KiB/s wr, 10 op/s 2024-04-26T01:31:30.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:29 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:31:30.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:29 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:30.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:29 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:31:30.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:29 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:29 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:31:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:29 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:31.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:30 smithi028 ceph-mon[28247]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:31.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:30 smithi028 ceph-mon[28247]: Deploying daemon osd.3 on smithi028 2024-04-26T01:31:31.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:30 smithi169 ceph-mon[32851]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:31.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:30 smithi169 ceph-mon[32851]: Deploying daemon osd.3 on smithi028 2024-04-26T01:31:31.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:30 smithi053 ceph-mon[32593]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:31.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:30 smithi053 ceph-mon[32593]: Deploying daemon osd.3 on smithi028 2024-04-26T01:31:32.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:32 smithi028 ceph-mon[28247]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:33.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:32 smithi169 ceph-mon[32851]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:33.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:32 smithi053 ceph-mon[32593]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:34.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:34 smithi028 ceph-mon[28247]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:35.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:34 smithi169 ceph-mon[32851]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:35.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:34 smithi053 ceph-mon[32593]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:37.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:36 smithi028 ceph-mon[28247]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:37.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:36 smithi169 ceph-mon[32851]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:36 smithi053 ceph-mon[32593]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:39.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:38 smithi028 ceph-mon[28247]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:39.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:38 smithi169 ceph-mon[32851]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:38 smithi053 ceph-mon[32593]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:39.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:39.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:39.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:40.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:40.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:40.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:40.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:40.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:40 smithi028 ceph-mon[28247]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:40 smithi169 ceph-mon[32851]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:40 smithi053 ceph-mon[32593]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:43.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.711+0000 7f33ef400700 1 -- 172.21.15.28:0/2514424035 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f33f8062e80 con 0x7f33dc0a3c30 2024-04-26T01:31:43.707 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-04-26T01:31:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f33dc0a3c30 msgr2=0x7f33dc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f33dc0a3c30 0x7f33dc0a60f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f33f8076e50 tx=0x7f33e80073d0 comp rx=0 tx=0).stop 2024-04-26T01:31:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33f8076a70 msgr2=0x7f33f819b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33f8076a70 0x7f33f819b3a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f33e400b760 tx=0x7f33e40057c0 comp rx=0 tx=0).stop 2024-04-26T01:31:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 shutdown_connections 2024-04-26T01:31:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f33dc0a3c30 0x7f33dc0a60f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f810a990 0x7f33f81a2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33f80754a0 0x7f33f819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 --2- 172.21.15.28:0/2514424035 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33f8076a70 0x7f33f819b3a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.714+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 >> 172.21.15.28:0/2514424035 conn(0x7f33f80fe060 msgr2=0x7f33f8106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.715+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 shutdown_connections 2024-04-26T01:31:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:31:43.715+0000 7f33fda00700 1 -- 172.21.15.28:0/2514424035 wait complete. 2024-04-26T01:31:43.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:43 smithi028 ceph-mon[28247]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:43.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:43.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:43.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:43 smithi028 ceph-mon[28247]: from='osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:31:43.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:43 smithi028 ceph-mon[28247]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:31:43.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:43 smithi169 ceph-mon[32851]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:43.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:43.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:43.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:43 smithi169 ceph-mon[32851]: from='osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:31:43.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:43 smithi169 ceph-mon[32851]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:31:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:43 smithi053 ceph-mon[32593]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:43 smithi053 ceph-mon[32593]: from='osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:31:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:43 smithi053 ceph-mon[32593]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:31:44.508 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T01:31:44.513 INFO:tasks.cephadm:Deploying osd.4 on smithi053 with /dev/vg_nvme/lv_3... 2024-04-26T01:31:44.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:31:44.764 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:31:44.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:44 smithi028 ceph-mon[28247]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:31:44.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:44 smithi028 ceph-mon[28247]: from='osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:31:44.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:44 smithi028 ceph-mon[28247]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:31:44.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:44.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:44 smithi028 ceph-mon[28247]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:31:44.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:44.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:44.887 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:44 smithi053 ceph-mon[32593]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:31:44.887 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:44 smithi053 ceph-mon[32593]: from='osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:31:44.887 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:44 smithi053 ceph-mon[32593]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:31:44.887 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:44.887 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:44 smithi053 ceph-mon[32593]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:31:44.887 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:44.887 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:44.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:44 smithi169 ceph-mon[32851]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:31:44.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:44 smithi169 ceph-mon[32851]: from='osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:31:44.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:44 smithi169 ceph-mon[32851]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:31:44.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:44.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:44 smithi169 ceph-mon[32851]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:31:44.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:44.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:45.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:45 smithi028 ceph-mon[28247]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:45.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:45 smithi028 ceph-mon[28247]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:31:45.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:45 smithi028 ceph-mon[28247]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:31:45.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:45 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:45.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:45 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:45.857 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:45 smithi053 ceph-mon[32593]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:45.857 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:45 smithi053 ceph-mon[32593]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:31:45.857 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:45 smithi053 ceph-mon[32593]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:31:45.857 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:45 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:45.858 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:45 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:45.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:45 smithi169 ceph-mon[32851]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:31:45.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:45 smithi169 ceph-mon[32851]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:31:45.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:45 smithi169 ceph-mon[32851]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:31:45.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:45 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:45.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:45 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:46.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:46 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:31:46.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:46 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:31:46.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:46 smithi053 ceph-mon[32593]: osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] boot 2024-04-26T01:31:46.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:46 smithi053 ceph-mon[32593]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T01:31:46.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:46 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:46.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:46 smithi053 ceph-mon[32593]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T01:31:46.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:46 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:31:46.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:46 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:31:46.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:46 smithi028 ceph-mon[28247]: osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] boot 2024-04-26T01:31:46.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:46 smithi028 ceph-mon[28247]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T01:31:46.741 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:46 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:46.741 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:46 smithi028 ceph-mon[28247]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T01:31:46.862 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:31:46.862 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:31:46.862 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-26T01:31:46.862 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-26T01:31:46.862 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0084035 s, 1.2 GB/s 2024-04-26T01:31:46.862 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-26T01:31:46.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:46 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:31:46.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:46 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:31:46.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:46 smithi169 ceph-mon[32851]: osd.3 [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] boot 2024-04-26T01:31:46.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:46 smithi169 ceph-mon[32851]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T01:31:46.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:46 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:31:46.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:46 smithi169 ceph-mon[32851]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T01:31:47.648 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:47 smithi053 ceph-mon[32593]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:31:47.648 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:47 smithi053 ceph-mon[32593]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:31:47.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_3 2024-04-26T01:31:47.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:47 smithi028 ceph-mon[28247]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:31:47.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:47 smithi028 ceph-mon[28247]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:31:47.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:47 smithi169 ceph-mon[32851]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:31:47.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:47 smithi169 ceph-mon[32851]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:31:47.938 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:31:49.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:48 smithi028 ceph-mon[28247]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:31:49.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:48 smithi028 ceph-mon[28247]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:31:49.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:48 smithi053 ceph-mon[32593]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:31:49.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:48 smithi053 ceph-mon[32593]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:31:49.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:48 smithi169 ceph-mon[32851]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:31:49.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:48 smithi169 ceph-mon[32851]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:31:49.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 -- 172.21.15.53:0/1315315353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4102bc0 msgr2=0x7efdd4103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:49.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1315315353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4102bc0 0x7efdd4103020 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efdc80099c0 tx=0x7efdc8009cd0 comp rx=0 tx=0).stop 2024-04-26T01:31:49.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 -- 172.21.15.53:0/1315315353 shutdown_connections 2024-04-26T01:31:49.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1315315353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4103560 0x7efdd4109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:49.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1315315353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efdd41019c0 0x7efdd4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:49.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1315315353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4102bc0 0x7efdd4103020 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:49.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 -- 172.21.15.53:0/1315315353 >> 172.21.15.53:0/1315315353 conn(0x7efdd40fcf60 msgr2=0x7efdd40ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:49.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 -- 172.21.15.53:0/1315315353 shutdown_connections 2024-04-26T01:31:49.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 -- 172.21.15.53:0/1315315353 wait complete. 2024-04-26T01:31:49.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 Processor -- start 2024-04-26T01:31:49.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.544+0000 7efdda9b9700 1 -- start start 2024-04-26T01:31:49.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdda9b9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efdd41019c0 0x7efdd4198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:49.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdda9b9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4102bc0 0x7efdd4199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:49.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdda9b9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4103560 0x7efdd41a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:49.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdda9b9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdd410d4b0 con 0x7efdd4102bc0 2024-04-26T01:31:49.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdda9b9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efdd410d330 con 0x7efdd4103560 2024-04-26T01:31:49.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdda9b9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7efdd410d630 con 0x7efdd41019c0 2024-04-26T01:31:49.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4103560 0x7efdd41a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4103560 0x7efdd41a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43720/0 (socket says 172.21.15.53:43720) 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 -- 172.21.15.53:0/1150161414 learned_addr learned my addr 172.21.15.53:0/1150161414 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd3fff700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efdd41019c0 0x7efdd4198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd37fe700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4102bc0 0x7efdd4199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 -- 172.21.15.53:0/1150161414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efdd41019c0 msgr2=0x7efdd4198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efdd41019c0 0x7efdd4198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 -- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4102bc0 msgr2=0x7efdd4199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4102bc0 0x7efdd4199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 -- 172.21.15.53:0/1150161414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd41a0880 con 0x7efdd4103560 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd37fe700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4102bc0 0x7efdd4199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd8f56700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4103560 0x7efdd41a01b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00b970 tx=0x7efdcc00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.545+0000 7efdd17fa700 1 -- 172.21.15.53:0/1150161414 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc019070 con 0x7efdd4103560 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.546+0000 7efdd17fa700 1 -- 172.21.15.53:0/1150161414 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdcc00cde0 con 0x7efdd4103560 2024-04-26T01:31:49.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.546+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd41a0b70 con 0x7efdd4103560 2024-04-26T01:31:49.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.546+0000 7efdd17fa700 1 -- 172.21.15.53:0/1150161414 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc007370 con 0x7efdd4103560 2024-04-26T01:31:49.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.546+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd41a10b0 con 0x7efdd4103560 2024-04-26T01:31:49.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.547+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd4069730 con 0x7efdd4103560 2024-04-26T01:31:49.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.548+0000 7efdd17fa700 1 -- 172.21.15.53:0/1150161414 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efdcc01c070 con 0x7efdd4103560 2024-04-26T01:31:49.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.549+0000 7efdd17fa700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7efdbc0a3cc0 0x7efdbc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:49.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.549+0000 7efdd17fa700 1 -- 172.21.15.53:0/1150161414 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7efdcc0ce650 con 0x7efdd4103560 2024-04-26T01:31:49.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.549+0000 7efdd3fff700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7efdbc0a3cc0 0x7efdbc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:49.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.550+0000 7efdd3fff700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7efdbc0a3cc0 0x7efdbc0a6180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efdd419a110 tx=0x7efdc400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:49.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.554+0000 7efdd17fa700 1 -- 172.21.15.53:0/1150161414 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efdcc09c250 con 0x7efdd4103560 2024-04-26T01:31:49.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:31:49.771+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7efdd40ff160 con 0x7efdbc0a3cc0 2024-04-26T01:31:50.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:31:50.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:31:50.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:31:50.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:50.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:31:50.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:31:50.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:50.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:50 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi028 2024-04-26T01:31:50.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:50 smithi053 ceph-mon[32593]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:31:50.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:50 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-26T01:31:50.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:50 smithi053 ceph-mon[32593]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:31:50.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:50 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:50.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:50 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:50.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:50 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:50.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:50 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:51.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:50 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi028 2024-04-26T01:31:51.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:50 smithi169 ceph-mon[32851]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:31:51.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:50 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-26T01:31:51.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:50 smithi169 ceph-mon[32851]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:31:51.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:50 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:51.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:50 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:51.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:50 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:51.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:50 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:50 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi028 2024-04-26T01:31:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:50 smithi028 ceph-mon[28247]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:31:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:50 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-26T01:31:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:50 smithi028 ceph-mon[28247]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:31:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:50 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:31:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:50 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:50 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:31:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:50 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:31:53.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:52 smithi169 ceph-mon[32851]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-26T01:31:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:52 smithi169 ceph-mon[32851]: from='client.? 172.21.15.53:0/3474604049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]: dispatch 2024-04-26T01:31:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:52 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]: dispatch 2024-04-26T01:31:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:52 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]': finished 2024-04-26T01:31:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:52 smithi169 ceph-mon[32851]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T01:31:53.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:52 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:31:53.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:52 smithi053 ceph-mon[32593]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-26T01:31:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:52 smithi053 ceph-mon[32593]: from='client.? 172.21.15.53:0/3474604049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]: dispatch 2024-04-26T01:31:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:52 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]: dispatch 2024-04-26T01:31:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:52 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]': finished 2024-04-26T01:31:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:52 smithi053 ceph-mon[32593]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T01:31:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:52 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:31:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:52 smithi028 ceph-mon[28247]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-26T01:31:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:52 smithi028 ceph-mon[28247]: from='client.? 172.21.15.53:0/3474604049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]: dispatch 2024-04-26T01:31:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:52 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]: dispatch 2024-04-26T01:31:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:52 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3adbfa0d-bced-4cd1-bf74-06e2fd42a015"}]': finished 2024-04-26T01:31:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:52 smithi028 ceph-mon[28247]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T01:31:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:52 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:31:54.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:53 smithi169 ceph-mon[32851]: from='client.? 172.21.15.53:0/2442143269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:31:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:53 smithi053 ceph-mon[32593]: from='client.? 172.21.15.53:0/2442143269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:31:54.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:53 smithi028 ceph-mon[28247]: from='client.? 172.21.15.53:0/2442143269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:31:55.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:54 smithi169 ceph-mon[32851]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:31:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:54 smithi053 ceph-mon[32593]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:31:55.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:54 smithi028 ceph-mon[28247]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:31:57.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:56 smithi169 ceph-mon[32851]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T01:31:57.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:56 smithi053 ceph-mon[32593]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T01:31:57.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:56 smithi028 ceph-mon[28247]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T01:31:59.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:31:58 smithi169 ceph-mon[32851]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T01:31:59.198 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:31:58 smithi053 ceph-mon[32593]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T01:31:59.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:31:58 smithi028 ceph-mon[28247]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T01:32:01.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:00 smithi169 ceph-mon[32851]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:01.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:00 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:32:01.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:00 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:01.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:00 smithi169 ceph-mon[32851]: Deploying daemon osd.4 on smithi053 2024-04-26T01:32:01.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:00 smithi053 ceph-mon[32593]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:01.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:00 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:32:01.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:00 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:01.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:00 smithi053 ceph-mon[32593]: Deploying daemon osd.4 on smithi053 2024-04-26T01:32:01.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:00 smithi028 ceph-mon[28247]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:01.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:00 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:32:01.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:00 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:01.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:00 smithi028 ceph-mon[28247]: Deploying daemon osd.4 on smithi053 2024-04-26T01:32:03.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:02 smithi169 ceph-mon[32851]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:02 smithi053 ceph-mon[32593]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:03.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:02 smithi028 ceph-mon[28247]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:05.106 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:04 smithi053 ceph-mon[32593]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:05.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:04 smithi169 ceph-mon[32851]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:05.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:04 smithi028 ceph-mon[28247]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:07.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:06 smithi028 ceph-mon[28247]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:07.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:06 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:07.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:06 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:07.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:06 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:07.139 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:06 smithi053 ceph-mon[32593]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:07.139 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:06 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:07.139 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:06 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:07.139 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:06 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:07.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:06 smithi169 ceph-mon[32851]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:07.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:06 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:07.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:06 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:07.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:06 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:08 smithi028 ceph-mon[28247]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:09.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:09.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:08 smithi169 ceph-mon[32851]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:09.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:09.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:09.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:08 smithi053 ceph-mon[32593]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:09.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:09.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:09.545 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 4 on host 'smithi053' 2024-04-26T01:32:09.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.550+0000 7efdd17fa700 1 -- 172.21.15.53:0/1150161414 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7efdd40ff160 con 0x7efdbc0a3cc0 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.552+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7efdbc0a3cc0 msgr2=0x7efdbc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.552+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7efdbc0a3cc0 0x7efdbc0a6180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efdd419a110 tx=0x7efdc400b410 comp rx=0 tx=0).stop 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.552+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4103560 msgr2=0x7efdd41a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.552+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4103560 0x7efdd41a01b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00b970 tx=0x7efdcc00bc80 comp rx=0 tx=0).stop 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.553+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 shutdown_connections 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.553+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7efdbc0a3cc0 0x7efdbc0a6180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.553+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efdd4103560 0x7efdd41a01b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.553+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efdd41019c0 0x7efdd4198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.553+0000 7efdda9b9700 1 --2- 172.21.15.53:0/1150161414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdd4102bc0 0x7efdd4199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.553+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 >> 172.21.15.53:0/1150161414 conn(0x7efdd40fcf60 msgr2=0x7efdd40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.553+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 shutdown_connections 2024-04-26T01:32:09.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:32:09.553+0000 7efdda9b9700 1 -- 172.21.15.53:0/1150161414 wait complete. 2024-04-26T01:32:10.207 DEBUG:teuthology.orchestra.run.smithi053:osd.4> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T01:32:10.209 INFO:tasks.cephadm:Deploying osd.5 on smithi169 with /dev/vg_nvme/lv_3... 2024-04-26T01:32:10.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:32:10.443 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:32:10.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:10 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:10.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:10 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:10.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:10 smithi053 ceph-mon[32593]: from='osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:32:10.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:10 smithi053 ceph-mon[32593]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:32:10.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:10.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:10.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:10 smithi028 ceph-mon[28247]: from='osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:32:10.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:10 smithi028 ceph-mon[28247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:32:10.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:10 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:10.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:10 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:10.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:10 smithi169 ceph-mon[32851]: from='osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:32:10.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:10 smithi169 ceph-mon[32851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:32:11.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:11 smithi053 ceph-mon[32593]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:11.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:11 smithi053 ceph-mon[32593]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:32:11.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:11 smithi053 ceph-mon[32593]: from='osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:32:11.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:11 smithi053 ceph-mon[32593]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T01:32:11.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:11.708 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:11 smithi053 ceph-mon[32593]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:32:11.708 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:32:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4[44022]: 2024-04-26T01:32:11.521+0000 7f665546d700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T01:32:11.708 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:32:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4[44022]: 2024-04-26T01:32:11.528+0000 7f6651260700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:32:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:11 smithi028 ceph-mon[28247]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:11 smithi028 ceph-mon[28247]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:32:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:11 smithi028 ceph-mon[28247]: from='osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:32:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:11 smithi028 ceph-mon[28247]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T01:32:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:11 smithi028 ceph-mon[28247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:32:11.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:11 smithi169 ceph-mon[32851]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:11.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:11 smithi169 ceph-mon[32851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:32:11.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:11 smithi169 ceph-mon[32851]: from='osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:32:11.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:11 smithi169 ceph-mon[32851]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T01:32:11.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:11.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:11 smithi169 ceph-mon[32851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:32:12.362 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:32:12.362 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:32:12.362 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-04-26T01:32:12.362 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-04-26T01:32:12.362 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114932 s, 912 MB/s 2024-04-26T01:32:12.363 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-04-26T01:32:12.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:12 smithi169 ceph-mon[32851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:32:12.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:12 smithi169 ceph-mon[32851]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:32:12.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:12.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:12.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:12 smithi053 ceph-mon[32593]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:32:12.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:12 smithi053 ceph-mon[32593]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:32:12.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:12.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:13.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi169:vg_nvme/lv_3 2024-04-26T01:32:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:12 smithi028 ceph-mon[28247]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:32:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:12 smithi028 ceph-mon[28247]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:32:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:13.266 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:32:13.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] boot 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:32:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:32:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] boot 2024-04-26T01:32:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T01:32:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:14.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:32:14.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:32:14.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: osd.4 [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] boot 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:14.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:14.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 -- 172.21.15.169:0/2673423159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8104640 msgr2=0x7fdef810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:14.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2673423159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8104640 0x7fdef810aef0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdef000b0a0 tx=0x7fdef000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 -- 172.21.15.169:0/2673423159 shutdown_connections 2024-04-26T01:32:14.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2673423159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8104640 0x7fdef810aef0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2673423159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8102aa0 0x7fdef8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2673423159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef8103ca0 0x7fdef8104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 -- 172.21.15.169:0/2673423159 >> 172.21.15.169:0/2673423159 conn(0x7fdef80fe040 msgr2=0x7fdef8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 -- 172.21.15.169:0/2673423159 shutdown_connections 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.783+0000 7fdeff24b700 1 -- 172.21.15.169:0/2673423159 wait complete. 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdeff24b700 1 Processor -- start 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdeff24b700 1 -- start start 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdeff24b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8102aa0 0x7fdef8198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdeff24b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8103ca0 0x7fdef8199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdeff24b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef8104640 0x7fdef81a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdeff24b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdef810e630 con 0x7fdef8102aa0 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdeff24b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdef810e4b0 con 0x7fdef8104640 2024-04-26T01:32:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdeff24b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdef810e7b0 con 0x7fdef8103ca0 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8103ca0 0x7fdef8199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8103ca0 0x7fdef8199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52488/0 (socket says 172.21.15.169:52488) 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 -- 172.21.15.169:0/2427990731 learned_addr learned my addr 172.21.15.169:0/2427990731 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdefd7e8700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef8104640 0x7fdef81a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 -- 172.21.15.169:0/2427990731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef8104640 msgr2=0x7fdef81a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef8104640 0x7fdef81a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 -- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8102aa0 msgr2=0x7fdef8198d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8102aa0 0x7fdef8198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 -- 172.21.15.169:0/2427990731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef81a0990 con 0x7fdef8103ca0 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.784+0000 7fdef7fff700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8103ca0 0x7fdef8199270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdee400b560 tx=0x7fdee400b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.785+0000 7fdef5ffb700 1 -- 172.21.15.169:0/2427990731 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee4015020 con 0x7fdef8103ca0 2024-04-26T01:32:14.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.785+0000 7fdef5ffb700 1 -- 172.21.15.169:0/2427990731 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee4011ce0 con 0x7fdef8103ca0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.785+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef81a0c80 con 0x7fdef8103ca0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.785+0000 7fdef5ffb700 1 -- 172.21.15.169:0/2427990731 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee4010da0 con 0x7fdef8103ca0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.785+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef81a11c0 con 0x7fdef8103ca0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.786+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef8069770 con 0x7fdef8103ca0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.787+0000 7fdef5ffb700 1 -- 172.21.15.169:0/2427990731 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdee4018020 con 0x7fdef8103ca0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.788+0000 7fdef5ffb700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdee80a3cc0 0x7fdee80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.788+0000 7fdef5ffb700 1 -- 172.21.15.169:0/2427990731 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fdee40ceb60 con 0x7fdef8103ca0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.788+0000 7fdefcfe7700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdee80a3cc0 0x7fdee80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.789+0000 7fdefcfe7700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdee80a3cc0 0x7fdee80a6180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdef8103b00 tx=0x7fdeec006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:14.793+0000 7fdef5ffb700 1 -- 172.21.15.169:0/2427990731 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdee401f020 con 0x7fdef8103ca0 2024-04-26T01:32:14.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:14 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi053 2024-04-26T01:32:14.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:14 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi053 to 8546M 2024-04-26T01:32:14.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:14 smithi169 ceph-mon[32851]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T01:32:14.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:14.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:14.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:14 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi053 2024-04-26T01:32:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:14 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi053 to 8546M 2024-04-26T01:32:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:14 smithi053 ceph-mon[32593]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T01:32:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:14.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:15.000+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fdef8062e80 con 0x7fdee80a3cc0 2024-04-26T01:32:15.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:14 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi053 2024-04-26T01:32:15.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:14 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi053 to 8546M 2024-04-26T01:32:15.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:14 smithi028 ceph-mon[28247]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T01:32:15.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:15.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:15.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:15.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:15.862 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:15 smithi169 ceph-mon[32851]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:15.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:15 smithi169 ceph-mon[32851]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:15.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:15 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:32:15.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:15 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:32:15.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:15 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:15 smithi053 ceph-mon[32593]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:15 smithi053 ceph-mon[32593]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:15 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:32:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:15 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:32:15.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:15 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:16.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:15 smithi028 ceph-mon[28247]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:16.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:15 smithi028 ceph-mon[28247]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:16.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:32:16.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:32:16.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:15 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:16.813 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:16 smithi169 ceph-mon[32851]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:16 smithi053 ceph-mon[32593]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:17.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:16 smithi028 ceph-mon[28247]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:18.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:17 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/1135307394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]: dispatch 2024-04-26T01:32:18.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:17 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]: dispatch 2024-04-26T01:32:18.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:17 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]': finished 2024-04-26T01:32:18.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:17 smithi169 ceph-mon[32851]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T01:32:18.025 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:17 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:18.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:17 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/1135307394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]: dispatch 2024-04-26T01:32:18.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:17 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]: dispatch 2024-04-26T01:32:18.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:17 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]': finished 2024-04-26T01:32:18.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:17 smithi028 ceph-mon[28247]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T01:32:18.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:17 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:17 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/1135307394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]: dispatch 2024-04-26T01:32:18.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:17 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]: dispatch 2024-04-26T01:32:18.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:17 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9211a41-97a0-4477-8dfb-7354f6020bfd"}]': finished 2024-04-26T01:32:18.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:17 smithi053 ceph-mon[32593]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T01:32:18.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:17 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:19.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:18 smithi028 ceph-mon[28247]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:19.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:18 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/3352055179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:32:19.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:18 smithi169 ceph-mon[32851]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:19.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:18 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/3352055179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:32:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:18 smithi053 ceph-mon[32593]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:18 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/3352055179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:32:21.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:20 smithi028 ceph-mon[28247]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:21.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:20 smithi169 ceph-mon[32851]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:20 smithi053 ceph-mon[32593]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:23.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:22 smithi028 ceph-mon[28247]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:23.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:22 smithi169 ceph-mon[32851]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:23.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:22 smithi053 ceph-mon[32593]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:24.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:24 smithi169 ceph-mon[32851]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:25.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:24 smithi028 ceph-mon[28247]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:24 smithi053 ceph-mon[32593]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:26.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:32:26.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:26.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:25 smithi028 ceph-mon[28247]: Deploying daemon osd.5 on smithi169 2024-04-26T01:32:26.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:32:26.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:26.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:25 smithi169 ceph-mon[32851]: Deploying daemon osd.5 on smithi169 2024-04-26T01:32:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:32:26.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:26.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:25 smithi053 ceph-mon[32593]: Deploying daemon osd.5 on smithi169 2024-04-26T01:32:27.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:26 smithi028 ceph-mon[28247]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:27.104 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:26 smithi169 ceph-mon[32851]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:27.118 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:26 smithi053 ceph-mon[32593]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:29.150 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:28 smithi169 ceph-mon[32851]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:29.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:28 smithi053 ceph-mon[32593]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:29.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:28 smithi028 ceph-mon[28247]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:31.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:31 smithi169 ceph-mon[32851]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:31.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:31 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:31.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:31 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:31.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:31 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:31 smithi053 ceph-mon[32593]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:31 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:31 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:31 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:31 smithi028 ceph-mon[28247]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:31 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:31 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:31 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:33.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:33 smithi169 ceph-mon[32851]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:33.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:33.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:33.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:33.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:33.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:33 smithi053 ceph-mon[32593]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:33.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:33 smithi028 ceph-mon[28247]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:33.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:33.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:33.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:33.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:33.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:34.133 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 5 on host 'smithi169' 2024-04-26T01:32:34.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.137+0000 7fdef5ffb700 1 -- 172.21.15.169:0/2427990731 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdef8062e80 con 0x7fdee80a3cc0 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdee80a3cc0 msgr2=0x7fdee80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdee80a3cc0 0x7fdee80a6180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdef8103b00 tx=0x7fdeec006d20 comp rx=0 tx=0).stop 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8103ca0 msgr2=0x7fdef8199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8103ca0 0x7fdef8199270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdee400b560 tx=0x7fdee400b920 comp rx=0 tx=0).stop 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 shutdown_connections 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdee80a3cc0 0x7fdee80a6180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef8104640 0x7fdef81a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8102aa0 0x7fdef8198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 --2- 172.21.15.169:0/2427990731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef8103ca0 0x7fdef8199270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 >> 172.21.15.169:0/2427990731 conn(0x7fdef80fe040 msgr2=0x7fdef8107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:34.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 shutdown_connections 2024-04-26T01:32:34.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:32:34.140+0000 7fdeff24b700 1 -- 172.21.15.169:0/2427990731 wait complete. 2024-04-26T01:32:34.780 DEBUG:teuthology.orchestra.run.smithi169:osd.5> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T01:32:34.782 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-04-26T01:32:34.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:32:35.254 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:32:35.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:35 smithi028 ceph-mon[28247]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:35.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:35.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:35.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:35.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:35 smithi028 ceph-mon[28247]: from='osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:32:35.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:35 smithi028 ceph-mon[28247]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:32:35.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:35 smithi169 ceph-mon[32851]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:35.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:35.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:35.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:35.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:35 smithi169 ceph-mon[32851]: from='osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:32:35.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:35 smithi169 ceph-mon[32851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:32:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:35 smithi053 ceph-mon[32593]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:35 smithi053 ceph-mon[32593]: from='osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:32:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:35 smithi053 ceph-mon[32593]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:32:36.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:36 smithi169 ceph-mon[32851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:32:36.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:36 smithi169 ceph-mon[32851]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T01:32:36.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:36.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:36 smithi169 ceph-mon[32851]: from='osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:32:36.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:36 smithi169 ceph-mon[32851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:32:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:36 smithi053 ceph-mon[32593]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:32:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:36 smithi053 ceph-mon[32593]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T01:32:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:36 smithi053 ceph-mon[32593]: from='osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:32:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:36 smithi053 ceph-mon[32593]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:32:36.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:36 smithi028 ceph-mon[28247]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:32:36.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:36 smithi028 ceph-mon[28247]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T01:32:36.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:36.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:36 smithi028 ceph-mon[28247]: from='osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:32:36.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:36 smithi028 ceph-mon[28247]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:32:36.908 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:32:36 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:32:36.554+0000 7fb1492f8700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T01:32:36.908 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:32:36 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:32:36.561+0000 7fb1410e3700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:32:37.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:37 smithi169 ceph-mon[32851]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:37.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:37 smithi169 ceph-mon[32851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:32:37.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:37 smithi169 ceph-mon[32851]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:32:37.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:37.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:37 smithi169 ceph-mon[32851]: from='osd.5 ' entity='osd.5' 2024-04-26T01:32:37.337 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:37 smithi169 ceph-mon[32851]: osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] boot 2024-04-26T01:32:37.338 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:37 smithi169 ceph-mon[32851]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T01:32:37.338 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:37.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:37 smithi028 ceph-mon[28247]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:37.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:37 smithi028 ceph-mon[28247]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:32:37.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:37 smithi028 ceph-mon[28247]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:32:37.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:37.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:37 smithi028 ceph-mon[28247]: from='osd.5 ' entity='osd.5' 2024-04-26T01:32:37.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:37 smithi028 ceph-mon[28247]: osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] boot 2024-04-26T01:32:37.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:37 smithi028 ceph-mon[28247]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T01:32:37.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:37 smithi053 ceph-mon[32593]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:32:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:37 smithi053 ceph-mon[32593]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:32:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:37 smithi053 ceph-mon[32593]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:32:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:37 smithi053 ceph-mon[32593]: from='osd.5 ' entity='osd.5' 2024-04-26T01:32:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:37 smithi053 ceph-mon[32593]: osd.5 [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] boot 2024-04-26T01:32:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:37 smithi053 ceph-mon[32593]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T01:32:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:32:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:32:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:32:38.407 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-26T01:32:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-26T01:32:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0360898 s, 291 MB/s 2024-04-26T01:32:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-26T01:32:38.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:38 smithi028 ceph-mon[28247]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:38 smithi169 ceph-mon[32851]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T01:32:38.460 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:32:38.461 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:38 smithi053 ceph-mon[32593]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T01:32:39.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-04-26T01:32:39.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:39 smithi169 ceph-mon[32851]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:32:39.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:39 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi169 2024-04-26T01:32:39.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:39 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi169 to 8544M 2024-04-26T01:32:39.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:39 smithi169 ceph-mon[32851]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T01:32:39.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:39 smithi028 ceph-mon[28247]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:32:39.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:39 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi169 2024-04-26T01:32:39.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:39 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi169 to 8544M 2024-04-26T01:32:39.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:39 smithi028 ceph-mon[28247]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T01:32:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:39 smithi053 ceph-mon[32593]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:32:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:39 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi169 2024-04-26T01:32:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:39 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi169 to 8544M 2024-04-26T01:32:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:39 smithi053 ceph-mon[32593]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T01:32:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:32:40.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:40 smithi028 ceph-mon[28247]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T01:32:40.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:40 smithi028 ceph-mon[28247]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:32:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:40 smithi053 ceph-mon[32593]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T01:32:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:40 smithi053 ceph-mon[32593]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:32:40.604 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:40 smithi169 ceph-mon[32851]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T01:32:40.604 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:40 smithi169 ceph-mon[32851]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:32:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:41 smithi053 ceph-mon[32593]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:41.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:41 smithi053 ceph-mon[32593]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:32:41.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:41 smithi028 ceph-mon[28247]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:41.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:41 smithi028 ceph-mon[28247]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:32:41.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:41 smithi169 ceph-mon[32851]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:41.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:41 smithi169 ceph-mon[32851]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:32:42.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 -- 172.21.15.28:0/3445727133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a64101390 msgr2=0x7f6a641017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:42.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 --2- 172.21.15.28:0/3445727133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a64101390 0x7f6a641017b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6a540098f0 tx=0x7f6a54009c00 comp rx=0 tx=0).stop 2024-04-26T01:32:42.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 -- 172.21.15.28:0/3445727133 shutdown_connections 2024-04-26T01:32:42.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 --2- 172.21.15.28:0/3445727133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a641026b0 0x7f6a64102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:42.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 --2- 172.21.15.28:0/3445727133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a64101390 0x7f6a641017b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:42.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 --2- 172.21.15.28:0/3445727133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64101cf0 0x7f6a64102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:42.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 -- 172.21.15.28:0/3445727133 >> 172.21.15.28:0/3445727133 conn(0x7f6a640fcf30 msgr2=0x7f6a640ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:42.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 -- 172.21.15.28:0/3445727133 shutdown_connections 2024-04-26T01:32:42.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.391+0000 7f6a69c00700 1 -- 172.21.15.28:0/3445727133 wait complete. 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.392+0000 7f6a69c00700 1 Processor -- start 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a69c00700 1 -- start start 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a69c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a64101390 0x7f6a6419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a69c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64101cf0 0x7f6a6419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a69c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a641026b0 0x7f6a641a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a69c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a641106e0 con 0x7f6a64101cf0 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a69c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64110560 con 0x7f6a64101390 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a69c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64110860 con 0x7f6a641026b0 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a62000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64101cf0 0x7f6a6419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a62a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a64101390 0x7f6a6419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a62000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64101cf0 0x7f6a6419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50928/0 (socket says 172.21.15.28:50928) 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a62000700 1 -- 172.21.15.28:0/557953071 learned_addr learned my addr 172.21.15.28:0/557953071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:32:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.393+0000 7f6a63400700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a641026b0 0x7f6a641a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.394+0000 7f6a62a00700 1 -- 172.21.15.28:0/557953071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a641026b0 msgr2=0x7f6a641a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.394+0000 7f6a62a00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a641026b0 0x7f6a641a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.394+0000 7f6a62a00700 1 -- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64101cf0 msgr2=0x7f6a6419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.394+0000 7f6a62a00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64101cf0 0x7f6a6419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.394+0000 7f6a62a00700 1 -- 172.21.15.28:0/557953071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a641a2a40 con 0x7f6a64101390 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.394+0000 7f6a63400700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a641026b0 0x7f6a641a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.394+0000 7f6a62000700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64101cf0 0x7f6a6419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.394+0000 7f6a62a00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a64101390 0x7f6a6419ade0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009810 tx=0x7f6a54005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.395+0000 7f6a5b400700 1 -- 172.21.15.28:0/557953071 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54005510 con 0x7f6a64101390 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.395+0000 7f6a5b400700 1 -- 172.21.15.28:0/557953071 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a540056b0 con 0x7f6a64101390 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.395+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a641a2cd0 con 0x7f6a64101390 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.395+0000 7f6a5b400700 1 -- 172.21.15.28:0/557953071 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54003a80 con 0x7f6a64101390 2024-04-26T01:32:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.395+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a641a3180 con 0x7f6a64101390 2024-04-26T01:32:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.396+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a64069730 con 0x7f6a64101390 2024-04-26T01:32:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.398+0000 7f6a5b400700 1 -- 172.21.15.28:0/557953071 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6a54003c40 con 0x7f6a64101390 2024-04-26T01:32:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.399+0000 7f6a5b400700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6a480a3cc0 0x7f6a480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.400+0000 7f6a5b400700 1 -- 172.21.15.28:0/557953071 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f6a540d07e0 con 0x7f6a64101390 2024-04-26T01:32:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.400+0000 7f6a62000700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6a480a3cc0 0x7f6a480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.400+0000 7f6a62000700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6a480a3cc0 0x7f6a480a6180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005950 tx=0x7f6a50009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.402+0000 7f6a5b400700 1 -- 172.21.15.28:0/557953071 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a54013030 con 0x7f6a64101390 2024-04-26T01:32:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:32:42.642+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6a64062e80 con 0x7f6a480a3cc0 2024-04-26T01:32:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:43 smithi053 ceph-mon[32593]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:32:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:32:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:43.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:43 smithi028 ceph-mon[28247]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:43.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:32:43.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:32:43.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:43.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:43 smithi169 ceph-mon[32851]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 184 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:43.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:32:43.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:32:43.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:44.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:44 smithi028 ceph-mon[28247]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:44 smithi053 ceph-mon[32593]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:44.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:44 smithi169 ceph-mon[32851]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:45 smithi053 ceph-mon[32593]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:45.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:45 smithi028 ceph-mon[28247]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:45.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:45 smithi169 ceph-mon[32851]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T01:32:46.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:46 smithi028 ceph-mon[28247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T01:32:46.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:46 smithi028 ceph-mon[28247]: Cluster is now healthy 2024-04-26T01:32:46.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:46 smithi169 ceph-mon[32851]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T01:32:46.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:46 smithi169 ceph-mon[32851]: Cluster is now healthy 2024-04-26T01:32:46.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:46 smithi053 ceph-mon[32593]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T01:32:46.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:46 smithi053 ceph-mon[32593]: Cluster is now healthy 2024-04-26T01:32:47.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:47 smithi028 ceph-mon[28247]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:32:47.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:47 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/145900357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]: dispatch 2024-04-26T01:32:47.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:47 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]: dispatch 2024-04-26T01:32:47.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:47 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]': finished 2024-04-26T01:32:47.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:47 smithi028 ceph-mon[28247]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T01:32:47.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:32:47.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:47 smithi169 ceph-mon[32851]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:32:47.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:47 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/145900357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]: dispatch 2024-04-26T01:32:47.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:47 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]: dispatch 2024-04-26T01:32:47.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:47 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]': finished 2024-04-26T01:32:47.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:47 smithi169 ceph-mon[32851]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T01:32:47.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:32:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:47 smithi053 ceph-mon[32593]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:32:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:47 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/145900357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]: dispatch 2024-04-26T01:32:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:47 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]: dispatch 2024-04-26T01:32:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:47 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05c575f6-3cd7-46be-9c3e-66fcc47b4bf2"}]': finished 2024-04-26T01:32:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:47 smithi053 ceph-mon[32593]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T01:32:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:32:48.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:48 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2275739656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:32:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:48 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2275739656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:32:48.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:48 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2275739656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:32:49.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:49 smithi169 ceph-mon[32851]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T01:32:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:49 smithi053 ceph-mon[32593]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T01:32:49.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:49 smithi028 ceph-mon[28247]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T01:32:51.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:51 smithi169 ceph-mon[32851]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T01:32:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:51 smithi053 ceph-mon[32593]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T01:32:51.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:51 smithi028 ceph-mon[28247]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T01:32:53.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:53 smithi028 ceph-mon[28247]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:32:53.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:53 smithi169 ceph-mon[32851]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:32:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:53 smithi053 ceph-mon[32593]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:32:55.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:55 smithi028 ceph-mon[28247]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:32:55.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:55 smithi169 ceph-mon[32851]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:32:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:55 smithi053 ceph-mon[32593]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:32:57.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:57 smithi169 ceph-mon[32851]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:32:57.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:57 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:32:57.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:57 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:57 smithi053 ceph-mon[32593]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:32:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:57 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:32:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:57 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:57.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:57 smithi028 ceph-mon[28247]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:32:57.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:57 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:32:57.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:57 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:58.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:58 smithi028 ceph-mon[28247]: Deploying daemon osd.6 on smithi028 2024-04-26T01:32:58.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:58 smithi169 ceph-mon[32851]: Deploying daemon osd.6 on smithi028 2024-04-26T01:32:58.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:58 smithi053 ceph-mon[32593]: Deploying daemon osd.6 on smithi028 2024-04-26T01:32:59.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:32:59 smithi169 ceph-mon[32851]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:32:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:32:59 smithi053 ceph-mon[32593]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:32:59.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:32:59 smithi028 ceph-mon[28247]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:01.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:01 smithi169 ceph-mon[32851]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:01 smithi053 ceph-mon[32593]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:01.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:01 smithi028 ceph-mon[28247]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:03.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:03 smithi028 ceph-mon[28247]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:03.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:03 smithi169 ceph-mon[32851]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:03.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:03 smithi053 ceph-mon[32593]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:05.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:05 smithi169 ceph-mon[32851]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:05 smithi053 ceph-mon[32593]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:05.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:05 smithi028 ceph-mon[28247]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:06.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:06 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:06.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:06 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:06.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:06 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:06.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:06 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:06.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:06 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:06.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:06 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:06.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:06 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:06.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:06 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:06.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:06 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:07.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:07 smithi028 ceph-mon[28247]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:07.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:07 smithi169 ceph-mon[32851]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:07 smithi053 ceph-mon[32593]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:09.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:09 smithi028 ceph-mon[28247]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:09.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:09 smithi169 ceph-mon[32851]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:09.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:09 smithi053 ceph-mon[32593]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:11.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:11 smithi028 ceph-mon[28247]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:11.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:11.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:11.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:11 smithi028 ceph-mon[28247]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:33:11.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:11 smithi169 ceph-mon[32851]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:11.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:11.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:11.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:11 smithi169 ceph-mon[32851]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:33:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:11 smithi053 ceph-mon[32593]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:11 smithi053 ceph-mon[32593]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:33:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.602+0000 7f6a5b400700 1 -- 172.21.15.28:0/557953071 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6a64062e80 con 0x7f6a480a3cc0 2024-04-26T01:33:11.598 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-04-26T01:33:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6a480a3cc0 msgr2=0x7f6a480a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6a480a3cc0 0x7f6a480a6180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005950 tx=0x7f6a50009310 comp rx=0 tx=0).stop 2024-04-26T01:33:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a64101390 msgr2=0x7f6a6419ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a64101390 0x7f6a6419ade0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009810 tx=0x7f6a54005270 comp rx=0 tx=0).stop 2024-04-26T01:33:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 shutdown_connections 2024-04-26T01:33:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6a480a3cc0 0x7f6a480a6180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a641026b0 0x7f6a641a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a64101390 0x7f6a6419ade0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 --2- 172.21.15.28:0/557953071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64101cf0 0x7f6a6419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.606+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 >> 172.21.15.28:0/557953071 conn(0x7f6a640fcf30 msgr2=0x7f6a640fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.607+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 shutdown_connections 2024-04-26T01:33:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:33:11.607+0000 7f6a69c00700 1 -- 172.21.15.28:0/557953071 wait complete. 2024-04-26T01:33:12.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:12.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:33:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T01:33:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:33:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:12.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:12.547 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T01:33:12.556 INFO:tasks.cephadm:Deploying osd.7 on smithi053 with /dev/vg_nvme/lv_2... 2024-04-26T01:33:12.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:33:12.780 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:33:12.716+0000 7f046fa00700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T01:33:12.780 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:33:12 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:33:12.740+0000 7f0465800700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:33:12.803 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:33:13.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:13 smithi028 ceph-mon[28247]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:13.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:13 smithi028 ceph-mon[28247]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:33:13.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:13 smithi028 ceph-mon[28247]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T01:33:13.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:13.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:13.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:13 smithi028 ceph-mon[28247]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' 2024-04-26T01:33:13.192 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:13 smithi053 ceph-mon[32593]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:13.192 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:13 smithi053 ceph-mon[32593]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:33:13.192 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:13 smithi053 ceph-mon[32593]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T01:33:13.192 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:13.193 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:13.193 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:13 smithi053 ceph-mon[32593]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' 2024-04-26T01:33:13.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:13 smithi169 ceph-mon[32851]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:13.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:13 smithi169 ceph-mon[32851]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:33:13.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:13 smithi169 ceph-mon[32851]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T01:33:13.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:13.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:13.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:13 smithi169 ceph-mon[32851]: from='osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315]' entity='osd.6' 2024-04-26T01:33:14.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:14 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:33:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:14 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:33:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:14 smithi028 ceph-mon[28247]: osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] boot 2024-04-26T01:33:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:14 smithi028 ceph-mon[28247]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T01:33:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:14.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:14 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:33:14.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:14 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:33:14.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:14.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:14 smithi053 ceph-mon[32593]: osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] boot 2024-04-26T01:33:14.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:14 smithi053 ceph-mon[32593]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T01:33:14.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:14.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:14 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:33:14.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:14 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:33:14.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:14.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:14 smithi169 ceph-mon[32851]: osd.6 [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] boot 2024-04-26T01:33:14.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:14 smithi169 ceph-mon[32851]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T01:33:14.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:33:15.144 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:33:15.144 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:33:15.145 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-26T01:33:15.145 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-26T01:33:15.145 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00824485 s, 1.3 GB/s 2024-04-26T01:33:15.145 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 2024-04-26T01:33:15.145 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-26T01:33:15.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:15 smithi028 ceph-mon[28247]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:15.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:15 smithi028 ceph-mon[28247]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T01:33:15.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:15 smithi169 ceph-mon[32851]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:15.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:15 smithi169 ceph-mon[32851]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T01:33:15.453 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:15 smithi053 ceph-mon[32593]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:33:15.454 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:15 smithi053 ceph-mon[32593]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T01:33:15.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_2 2024-04-26T01:33:16.035 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:33:17.181 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:17 smithi053 ceph-mon[32593]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:17.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:17 smithi169 ceph-mon[32851]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:17.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:17 smithi028 ceph-mon[28247]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:17.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 -- 172.21.15.53:0/3928445878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c4103550 msgr2=0x7fd5c4105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:17.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/3928445878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c4103550 0x7fd5c4105940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400b0a0 tx=0x7fd5b400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:33:17.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 -- 172.21.15.53:0/3928445878 shutdown_connections 2024-04-26T01:33:17.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/3928445878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5c41087b0 0x7fd5c410ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:17.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/3928445878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c4103550 0x7fd5c4105940 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:17.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/3928445878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5c4105e80 0x7fd5c4108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:17.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 -- 172.21.15.53:0/3928445878 >> 172.21.15.53:0/3928445878 conn(0x7fd5c40fcfa0 msgr2=0x7fd5c40ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:17.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 -- 172.21.15.53:0/3928445878 shutdown_connections 2024-04-26T01:33:17.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.718+0000 7fd5c9a32700 1 -- 172.21.15.53:0/3928445878 wait complete. 2024-04-26T01:33:17.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c9a32700 1 Processor -- start 2024-04-26T01:33:17.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c9a32700 1 -- start start 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c9a32700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5c4103550 0x7fd5c4102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c9a32700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5c4105e80 0x7fd5c4101500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c9a32700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c41087b0 0x7fd5c4101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c9a32700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c410e610 con 0x7fd5c4103550 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c9a32700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c410e490 con 0x7fd5c41087b0 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c9a32700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c410e790 con 0x7fd5c4105e80 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c41087b0 0x7fd5c4101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c41087b0 0x7fd5c4101a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46388/0 (socket says 172.21.15.53:46388) 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c37fe700 1 -- 172.21.15.53:0/2898069789 learned_addr learned my addr 172.21.15.53:0/2898069789 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c27fc700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5c4105e80 0x7fd5c4101500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:17.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c37fe700 1 -- 172.21.15.53:0/2898069789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5c4105e80 msgr2=0x7fd5c4101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c37fe700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5c4105e80 0x7fd5c4101500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c37fe700 1 -- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5c4103550 msgr2=0x7fd5c4102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c37fe700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5c4103550 0x7fd5c4102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.719+0000 7fd5c37fe700 1 -- 172.21.15.53:0/2898069789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c4102310 con 0x7fd5c41087b0 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.720+0000 7fd5c37fe700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c41087b0 0x7fd5c4101a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00b480 tx=0x7fd5bc00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.720+0000 7fd5c8a30700 1 -- 172.21.15.53:0/2898069789 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc015020 con 0x7fd5c41087b0 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.720+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c41a2e20 con 0x7fd5c41087b0 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.720+0000 7fd5c8a30700 1 -- 172.21.15.53:0/2898069789 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5bc011ce0 con 0x7fd5c41087b0 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.720+0000 7fd5c8a30700 1 -- 172.21.15.53:0/2898069789 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc010e60 con 0x7fd5c41087b0 2024-04-26T01:33:17.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.720+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c41a32f0 con 0x7fd5c41087b0 2024-04-26T01:33:17.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.721+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c4069730 con 0x7fd5c41087b0 2024-04-26T01:33:17.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.722+0000 7fd5c8a30700 1 -- 172.21.15.53:0/2898069789 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd5bc018020 con 0x7fd5c41087b0 2024-04-26T01:33:17.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.722+0000 7fd5c8a30700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd5b00a3cc0 0x7fd5b00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:17.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.723+0000 7fd5c8a30700 1 -- 172.21.15.53:0/2898069789 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fd5bc0ceea0 con 0x7fd5c41087b0 2024-04-26T01:33:17.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.723+0000 7fd5c2ffd700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd5b00a3cc0 0x7fd5b00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:17.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.724+0000 7fd5c2ffd700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd5b00a3cc0 0x7fd5b00a6180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400b070 tx=0x7fd5b4009d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:17.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.727+0000 7fd5c8a30700 1 -- 172.21.15.53:0/2898069789 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd5bc01f020 con 0x7fd5c41087b0 2024-04-26T01:33:17.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:17.943+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd5c4062e80 con 0x7fd5b00a3cc0 2024-04-26T01:33:19.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:19.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi028 2024-04-26T01:33:19.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:33:19.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi028 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:33:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:33:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi028 2024-04-26T01:33:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-26T01:33:19.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:19.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:19.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:19.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:19.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:33:19.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:33:19.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:21.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:20 smithi169 ceph-mon[32851]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:21.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:20 smithi169 ceph-mon[32851]: from='client.? 172.21.15.53:0/3896966266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]: dispatch 2024-04-26T01:33:21.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:20 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]: dispatch 2024-04-26T01:33:21.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:20 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]': finished 2024-04-26T01:33:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:20 smithi169 ceph-mon[32851]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T01:33:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:21.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:20 smithi053 ceph-mon[32593]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:21.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:20 smithi053 ceph-mon[32593]: from='client.? 172.21.15.53:0/3896966266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]: dispatch 2024-04-26T01:33:21.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:20 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]: dispatch 2024-04-26T01:33:21.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:20 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]': finished 2024-04-26T01:33:21.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:20 smithi053 ceph-mon[32593]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T01:33:21.159 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:21.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:20 smithi028 ceph-mon[28247]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:21.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:20 smithi028 ceph-mon[28247]: from='client.? 172.21.15.53:0/3896966266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]: dispatch 2024-04-26T01:33:21.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:20 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]: dispatch 2024-04-26T01:33:21.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:20 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "630a21f2-701f-4b41-8d73-88c86c371714"}]': finished 2024-04-26T01:33:21.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:20 smithi028 ceph-mon[28247]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T01:33:21.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:22.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:21 smithi169 ceph-mon[32851]: from='client.? 172.21.15.53:0/1444930077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:33:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:21 smithi053 ceph-mon[32593]: from='client.? 172.21.15.53:0/1444930077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:33:22.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:21 smithi028 ceph-mon[28247]: from='client.? 172.21.15.53:0/1444930077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:33:23.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:22 smithi169 ceph-mon[32851]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:23.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:22 smithi053 ceph-mon[32593]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:23.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:22 smithi028 ceph-mon[28247]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:25.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:24 smithi169 ceph-mon[32851]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:25.186 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:24 smithi053 ceph-mon[32593]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:25.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:24 smithi028 ceph-mon[28247]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:27.158 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:26 smithi053 ceph-mon[32593]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:27.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:26 smithi169 ceph-mon[32851]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:27.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:26 smithi028 ceph-mon[28247]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:29.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:28 smithi169 ceph-mon[32851]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:29.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:33:29.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:29.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:28 smithi169 ceph-mon[32851]: Deploying daemon osd.7 on smithi053 2024-04-26T01:33:29.161 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:28 smithi053 ceph-mon[32593]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:29.161 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:33:29.161 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:29.161 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:28 smithi053 ceph-mon[32593]: Deploying daemon osd.7 on smithi053 2024-04-26T01:33:29.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:28 smithi028 ceph-mon[28247]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:29.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:33:29.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:29.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:28 smithi028 ceph-mon[28247]: Deploying daemon osd.7 on smithi053 2024-04-26T01:33:31.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:30 smithi169 ceph-mon[32851]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:31.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:30 smithi053 ceph-mon[32593]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:31.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:30 smithi028 ceph-mon[28247]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:33.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:32 smithi169 ceph-mon[32851]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:32 smithi053 ceph-mon[32593]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:33.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:32 smithi028 ceph-mon[28247]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:35.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:35 smithi169 ceph-mon[32851]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:35.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:35.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:35.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:35 smithi053 ceph-mon[32593]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:35.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:35.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:35.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:35.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:35 smithi028 ceph-mon[28247]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:35.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:35.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:35.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:37 smithi053 ceph-mon[32593]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:37.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:37 smithi028 ceph-mon[28247]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:37.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:37 smithi169 ceph-mon[32851]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:38.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.428+0000 7fd5c8a30700 1 -- 172.21.15.53:0/2898069789 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd5c4062e80 con 0x7fd5b00a3cc0 2024-04-26T01:33:38.423 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 7 on host 'smithi053' 2024-04-26T01:33:38.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd5b00a3cc0 msgr2=0x7fd5b00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd5b00a3cc0 0x7fd5b00a6180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400b070 tx=0x7fd5b4009d00 comp rx=0 tx=0).stop 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c41087b0 msgr2=0x7fd5c4101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c41087b0 0x7fd5c4101a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00b480 tx=0x7fd5bc00b840 comp rx=0 tx=0).stop 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 shutdown_connections 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd5b00a3cc0 0x7fd5b00a6180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5c41087b0 0x7fd5c4101a40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5c4103550 0x7fd5c4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.430+0000 7fd5c9a32700 1 --2- 172.21.15.53:0/2898069789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5c4105e80 0x7fd5c4101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.431+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 >> 172.21.15.53:0/2898069789 conn(0x7fd5c40fcfa0 msgr2=0x7fd5c4103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.431+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 shutdown_connections 2024-04-26T01:33:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:33:38.431+0000 7fd5c9a32700 1 -- 172.21.15.53:0/2898069789 wait complete. 2024-04-26T01:33:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:33:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:33:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:38.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:38.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:38.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:39.116 DEBUG:teuthology.orchestra.run.smithi053:osd.7> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T01:33:39.118 INFO:tasks.cephadm:Deploying osd.8 on smithi169 with /dev/vg_nvme/lv_2... 2024-04-26T01:33:39.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:33:39.362 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:33:39.688 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:39 smithi053 ceph-mon[32593]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:39.688 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:39 smithi053 ceph-mon[32593]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:33:39.688 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:39 smithi053 ceph-mon[32593]: from='osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:33:39.689 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:39 smithi053 ceph-mon[32593]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T01:33:39.689 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:39.689 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:39 smithi053 ceph-mon[32593]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:33:39.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:39 smithi169 ceph-mon[32851]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:39.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:39 smithi169 ceph-mon[32851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:33:39.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:39 smithi169 ceph-mon[32851]: from='osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:33:39.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:39 smithi169 ceph-mon[32851]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T01:33:39.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:39.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:39 smithi169 ceph-mon[32851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:33:39.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:39 smithi028 ceph-mon[28247]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:39.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:39 smithi028 ceph-mon[28247]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:33:39.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:39 smithi028 ceph-mon[28247]: from='osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:33:39.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:39 smithi028 ceph-mon[28247]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T01:33:39.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:39.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:39 smithi028 ceph-mon[28247]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:33:40.279 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:33:40.000+0000 7fdd1daf8700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T01:33:40.279 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:33:40.010+0000 7fdd158e3700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:33:40.579 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-mon[32593]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:33:40.579 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-mon[32593]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T01:33:40.579 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:40.579 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-mon[32593]: from='osd.7 ' entity='osd.7' 2024-04-26T01:33:40.579 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-mon[32593]: osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] boot 2024-04-26T01:33:40.580 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-mon[32593]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T01:33:40.580 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:40.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:40 smithi028 ceph-mon[28247]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:33:40.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:40 smithi028 ceph-mon[28247]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T01:33:40.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:40.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:40 smithi028 ceph-mon[28247]: from='osd.7 ' entity='osd.7' 2024-04-26T01:33:40.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:40 smithi028 ceph-mon[28247]: osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] boot 2024-04-26T01:33:40.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:40 smithi028 ceph-mon[28247]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T01:33:40.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:40.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:40 smithi169 ceph-mon[32851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:33:40.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:40 smithi169 ceph-mon[32851]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T01:33:40.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:40.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:40 smithi169 ceph-mon[32851]: from='osd.7 ' entity='osd.7' 2024-04-26T01:33:40.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:40 smithi169 ceph-mon[32851]: osd.7 [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] boot 2024-04-26T01:33:40.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:40 smithi169 ceph-mon[32851]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T01:33:40.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:33:41.492 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:33:41.492 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:33:41.492 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-04-26T01:33:41.492 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-04-26T01:33:41.492 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134468 s, 780 MB/s 2024-04-26T01:33:41.492 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-04-26T01:33:41.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:41 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:33:41.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:41 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:33:41.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:41 smithi169 ceph-mon[32851]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:41.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:41 smithi169 ceph-mon[32851]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T01:33:41.687 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:41 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:33:41.687 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:41 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:33:41.687 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:41 smithi053 ceph-mon[32593]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:41.687 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:41 smithi053 ceph-mon[32593]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T01:33:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:41 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:33:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:41 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:33:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:41 smithi028 ceph-mon[28247]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:33:41.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:41 smithi028 ceph-mon[28247]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T01:33:42.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi169:vg_nvme/lv_2 2024-04-26T01:33:42.366 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:33:43.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:43.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi053 2024-04-26T01:33:43.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi053 to 5697M 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:43 smithi169 ceph-mon[32851]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi053 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi053 to 5697M 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:43.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:43 smithi053 ceph-mon[32593]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T01:33:43.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi053 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi053 to 5697M 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:33:43.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:43 smithi028 ceph-mon[28247]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 -- 172.21.15.169:0/258855279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 msgr2=0x7f8c5410aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/258855279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 0x7f8c5410aee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00b0a0 tx=0x7f8c4c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 -- 172.21.15.169:0/258855279 shutdown_connections 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/258855279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 0x7f8c5410aee0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/258855279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c54102a90 0x7f8c54102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/258855279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c54103c90 0x7f8c541040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 -- 172.21.15.169:0/258855279 >> 172.21.15.169:0/258855279 conn(0x7f8c540fe030 msgr2=0x7f8c54100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 -- 172.21.15.169:0/258855279 shutdown_connections 2024-04-26T01:33:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 -- 172.21.15.169:0/258855279 wait complete. 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 Processor -- start 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.010+0000 7f8c58ef8700 1 -- start start 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c58ef8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c54102a90 0x7f8c54198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c58ef8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c54103c90 0x7f8c54199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c58ef8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 0x7f8c541a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c58ef8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5410e620 con 0x7f8c54103c90 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c58ef8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5410e4a0 con 0x7f8c54102a90 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c58ef8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5410e7a0 con 0x7f8c54104630 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 0x7f8c541a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 0x7f8c541a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50730/0 (socket says 172.21.15.169:50730) 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 -- 172.21.15.169:0/4154579845 learned_addr learned my addr 172.21.15.169:0/4154579845 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:33:44.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c5259c700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c54102a90 0x7f8c54198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 -- 172.21.15.169:0/4154579845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c54102a90 msgr2=0x7f8c54198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c54102a90 0x7f8c54198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 -- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c54103c90 msgr2=0x7f8c54199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c54103c90 0x7f8c54199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 -- 172.21.15.169:0/4154579845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c541a0940 con 0x7f8c54104630 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c52d9d700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 0x7f8c541a0270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00b070 tx=0x7f8c4c004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c437fe700 1 -- 172.21.15.169:0/4154579845 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c021070 con 0x7f8c54104630 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c437fe700 1 -- 172.21.15.169:0/4154579845 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c4c004020 con 0x7f8c54104630 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c541a0bd0 con 0x7f8c54104630 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c437fe700 1 -- 172.21.15.169:0/4154579845 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c01d690 con 0x7f8c54104630 2024-04-26T01:33:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.011+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c541a1100 con 0x7f8c54104630 2024-04-26T01:33:44.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.013+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c54069770 con 0x7f8c54104630 2024-04-26T01:33:44.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.014+0000 7f8c437fe700 1 -- 172.21.15.169:0/4154579845 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8c4c0044f0 con 0x7f8c54104630 2024-04-26T01:33:44.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.014+0000 7f8c437fe700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8c3c0a3c40 0x7f8c3c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:44.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.015+0000 7f8c437fe700 1 -- 172.21.15.169:0/4154579845 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f8c4c00f020 con 0x7f8c54104630 2024-04-26T01:33:44.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.015+0000 7f8c5259c700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8c3c0a3c40 0x7f8c3c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:44.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.016+0000 7f8c5259c700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8c3c0a3c40 0x7f8c3c0a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8c44006fd0 tx=0x7f8c44008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:44.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.019+0000 7f8c437fe700 1 -- 172.21.15.169:0/4154579845 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c4c02a020 con 0x7f8c54104630 2024-04-26T01:33:44.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:33:44.233+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8c54062e80 con 0x7f8c3c0a3c40 2024-04-26T01:33:44.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:44 smithi169 ceph-mon[32851]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T01:33:44.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:44 smithi169 ceph-mon[32851]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:33:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:44 smithi053 ceph-mon[32593]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T01:33:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:44 smithi053 ceph-mon[32593]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:33:44.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:44 smithi028 ceph-mon[28247]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T01:33:44.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:44 smithi028 ceph-mon[28247]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:33:45.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:45 smithi169 ceph-mon[32851]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:45.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:45 smithi169 ceph-mon[32851]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:45.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:45 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:33:45.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:45 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:33:45.123 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:45 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:45.123 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:45 smithi169 ceph-mon[32851]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:33:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:45 smithi053 ceph-mon[32593]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:45 smithi053 ceph-mon[32593]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:45 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:33:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:45 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:33:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:45 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:45 smithi053 ceph-mon[32593]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:33:45.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:45 smithi028 ceph-mon[28247]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:45.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:45 smithi028 ceph-mon[28247]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:45.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:45 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:33:45.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:45 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:33:45.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:45 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:45.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:45 smithi028 ceph-mon[28247]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:33:46.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:46 smithi169 ceph-mon[32851]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T01:33:46.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:46 smithi169 ceph-mon[32851]: Cluster is now healthy 2024-04-26T01:33:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:46 smithi053 ceph-mon[32593]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T01:33:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:46 smithi053 ceph-mon[32593]: Cluster is now healthy 2024-04-26T01:33:46.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:46 smithi028 ceph-mon[28247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T01:33:46.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:46 smithi028 ceph-mon[28247]: Cluster is now healthy 2024-04-26T01:33:47.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:47 smithi169 ceph-mon[32851]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:47.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:47 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/3023070745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]: dispatch 2024-04-26T01:33:47.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:47 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]: dispatch 2024-04-26T01:33:47.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:47 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]': finished 2024-04-26T01:33:47.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:47 smithi169 ceph-mon[32851]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T01:33:47.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:47 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:33:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:47 smithi053 ceph-mon[32593]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:47 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/3023070745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]: dispatch 2024-04-26T01:33:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:47 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]: dispatch 2024-04-26T01:33:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:47 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]': finished 2024-04-26T01:33:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:47 smithi053 ceph-mon[32593]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T01:33:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:47 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:33:47.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:47 smithi028 ceph-mon[28247]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:47.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:47 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/3023070745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]: dispatch 2024-04-26T01:33:47.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:47 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]: dispatch 2024-04-26T01:33:47.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:47 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e807449-7760-4d62-801e-bf02535de567"}]': finished 2024-04-26T01:33:47.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:47 smithi028 ceph-mon[28247]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T01:33:47.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:47 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:33:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:48 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/722640798' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:33:48.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:48 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/722640798' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:33:48.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:48 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/722640798' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:33:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:49 smithi053 ceph-mon[32593]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:49.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:49 smithi028 ceph-mon[28247]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:49.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:49 smithi169 ceph-mon[32851]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:51.295 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:51 smithi169 ceph-mon[32851]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:51 smithi053 ceph-mon[32593]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:51.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:51 smithi028 ceph-mon[28247]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:53.311 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:53 smithi169 ceph-mon[32851]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:53.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:53 smithi053 ceph-mon[32593]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:53.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:53 smithi028 ceph-mon[28247]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:55.364 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:55 smithi169 ceph-mon[32851]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:55.364 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:33:55.365 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:55.365 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:55 smithi169 ceph-mon[32851]: Deploying daemon osd.8 on smithi169 2024-04-26T01:33:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:55 smithi053 ceph-mon[32593]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:33:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:55 smithi053 ceph-mon[32593]: Deploying daemon osd.8 on smithi169 2024-04-26T01:33:55.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:55 smithi028 ceph-mon[28247]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:55.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:33:55.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:55.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:55 smithi028 ceph-mon[28247]: Deploying daemon osd.8 on smithi169 2024-04-26T01:33:57.460 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:57 smithi169 ceph-mon[32851]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:57 smithi053 ceph-mon[32593]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:57.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:57 smithi028 ceph-mon[28247]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:33:59 smithi053 ceph-mon[32593]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:59.512 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:33:59 smithi169 ceph-mon[32851]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:33:59.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:33:59 smithi028 ceph-mon[28247]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:01.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:01 smithi169 ceph-mon[32851]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:01.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:01 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:01.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:01 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:01.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:01 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:01 smithi053 ceph-mon[32593]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:01 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:01 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:01 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:01.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:01 smithi028 ceph-mon[28247]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:01.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:01 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:01.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:01 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:01.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:01 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:03.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:03 smithi169 ceph-mon[32851]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:03 smithi053 ceph-mon[32593]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:03.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:03 smithi028 ceph-mon[28247]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:03.671 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 8 on host 'smithi169' 2024-04-26T01:34:03.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.675+0000 7f8c437fe700 1 -- 172.21.15.169:0/4154579845 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8c54062e80 con 0x7f8c3c0a3c40 2024-04-26T01:34:03.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8c3c0a3c40 msgr2=0x7f8c3c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:03.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8c3c0a3c40 0x7f8c3c0a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8c44006fd0 tx=0x7f8c44008040 comp rx=0 tx=0).stop 2024-04-26T01:34:03.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 msgr2=0x7f8c541a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:03.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 0x7f8c541a0270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00b070 tx=0x7f8c4c004ef0 comp rx=0 tx=0).stop 2024-04-26T01:34:03.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 shutdown_connections 2024-04-26T01:34:03.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8c3c0a3c40 0x7f8c3c0a6100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:03.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c54104630 0x7f8c541a0270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:03.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c54102a90 0x7f8c54198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:03.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 --2- 172.21.15.169:0/4154579845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c54103c90 0x7f8c54199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:03.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 >> 172.21.15.169:0/4154579845 conn(0x7f8c540fe030 msgr2=0x7f8c541070b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:03.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 shutdown_connections 2024-04-26T01:34:03.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:34:03.678+0000 7f8c58ef8700 1 -- 172.21.15.169:0/4154579845 wait complete. 2024-04-26T01:34:04.351 DEBUG:teuthology.orchestra.run.smithi169:osd.8> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T01:34:04.354 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-04-26T01:34:04.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:34:04.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:04 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:04 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:04 smithi028 ceph-mon[28247]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:04 smithi028 ceph-mon[28247]: from='osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:34:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:04 smithi028 ceph-mon[28247]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:34:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:04 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:04 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:04.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:04 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:04 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:04 smithi169 ceph-mon[32851]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:04.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:04 smithi169 ceph-mon[32851]: from='osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:34:04.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:04 smithi169 ceph-mon[32851]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:34:04.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:04 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:04 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:04 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:04 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:04 smithi053 ceph-mon[32593]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:04.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:04 smithi053 ceph-mon[32593]: from='osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:34:04.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:04 smithi053 ceph-mon[32593]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:34:04.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:04 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:04.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:04 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:05 smithi053 ceph-mon[32593]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:34:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:05 smithi053 ceph-mon[32593]: from='osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:34:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:05 smithi053 ceph-mon[32593]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T01:34:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:05 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:05 smithi053 ceph-mon[32593]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:34:06.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:05 smithi028 ceph-mon[28247]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:34:06.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:05 smithi028 ceph-mon[28247]: from='osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:34:06.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:05 smithi028 ceph-mon[28247]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T01:34:06.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:05 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:06.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:05 smithi028 ceph-mon[28247]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:34:06.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:05 smithi169 ceph-mon[32851]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:34:06.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:05 smithi169 ceph-mon[32851]: from='osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:34:06.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:05 smithi169 ceph-mon[32851]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T01:34:06.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:05 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:06.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:05 smithi169 ceph-mon[32851]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:34:06.111 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:34:05 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:34:05.672+0000 7f6decf0b700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T01:34:06.112 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:34:05 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:34:05.682+0000 7f6de8cfe700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:34:06.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:06 smithi169 ceph-mon[32851]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:34:06.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:06 smithi169 ceph-mon[32851]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T01:34:06.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:06 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:06.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:06 smithi169 ceph-mon[32851]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:06.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:06 smithi169 ceph-mon[32851]: osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] boot 2024-04-26T01:34:06.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:06 smithi169 ceph-mon[32851]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T01:34:06.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:06 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:06.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:06 smithi053 ceph-mon[32593]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:34:06.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:06 smithi053 ceph-mon[32593]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T01:34:06.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:06 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:06.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:06 smithi053 ceph-mon[32593]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:06.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:06 smithi053 ceph-mon[32593]: osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] boot 2024-04-26T01:34:06.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:06 smithi053 ceph-mon[32593]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T01:34:06.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:06 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:06.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:06 smithi028 ceph-mon[28247]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:34:06.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:06 smithi028 ceph-mon[28247]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T01:34:07.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:06 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:07.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:06 smithi028 ceph-mon[28247]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:34:07.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:06 smithi028 ceph-mon[28247]: osd.8 [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] boot 2024-04-26T01:34:07.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:06 smithi028 ceph-mon[28247]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T01:34:07.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:06 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:34:07.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:07 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:34:07.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:07 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:34:07.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:07 smithi169 ceph-mon[32851]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T01:34:07.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:07 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:34:07.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:07 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:34:07.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:07 smithi028 ceph-mon[28247]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T01:34:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:34:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:34:07.937 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-26T01:34:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-26T01:34:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0384948 s, 272 MB/s 2024-04-26T01:34:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-26T01:34:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:07 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:34:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:07 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:34:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:07 smithi053 ceph-mon[32593]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T01:34:09.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-04-26T01:34:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi169 2024-04-26T01:34:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi169 to 5696M 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi169 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi169 to 5696M 2024-04-26T01:34:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T01:34:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:09.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi169 2024-04-26T01:34:09.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi169 to 5696M 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:09.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:10 smithi053 ceph-mon[32593]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:34:10.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:10 smithi053 ceph-mon[32593]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:10.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:10 smithi028 ceph-mon[28247]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:34:10.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:10 smithi028 ceph-mon[28247]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:11.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:10 smithi169 ceph-mon[32851]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:34:11.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:10 smithi169 ceph-mon[32851]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:11.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.866+0000 7fe542c00700 1 -- 172.21.15.28:0/998172843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe53c103ca0 msgr2=0x7fe53c104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.866+0000 7fe542c00700 1 --2- 172.21.15.28:0/998172843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe53c103ca0 0x7fe53c104100 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe524009990 tx=0x7fe524009ca0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.867+0000 7fe542c00700 1 -- 172.21.15.28:0/998172843 shutdown_connections 2024-04-26T01:34:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.867+0000 7fe542c00700 1 --2- 172.21.15.28:0/998172843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe53c104640 0x7fe53c10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.867+0000 7fe542c00700 1 --2- 172.21.15.28:0/998172843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c102aa0 0x7fe53c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.867+0000 7fe542c00700 1 --2- 172.21.15.28:0/998172843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe53c103ca0 0x7fe53c104100 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.867+0000 7fe542c00700 1 -- 172.21.15.28:0/998172843 >> 172.21.15.28:0/998172843 conn(0x7fe53c0fe040 msgr2=0x7fe53c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.867+0000 7fe542c00700 1 -- 172.21.15.28:0/998172843 shutdown_connections 2024-04-26T01:34:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.867+0000 7fe542c00700 1 -- 172.21.15.28:0/998172843 wait complete. 2024-04-26T01:34:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.868+0000 7fe542c00700 1 Processor -- start 2024-04-26T01:34:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.868+0000 7fe542c00700 1 -- start start 2024-04-26T01:34:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.868+0000 7fe542c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c102aa0 0x7fe53c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.868+0000 7fe542c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe53c103ca0 0x7fe53c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.868+0000 7fe542c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe53c104640 0x7fe53c1a23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.868+0000 7fe542c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c10e560 con 0x7fe53c102aa0 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.868+0000 7fe542c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c10e3e0 con 0x7fe53c104640 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.868+0000 7fe542c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c10e6e0 con 0x7fe53c103ca0 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe53be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c102aa0 0x7fe53c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe540c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe53c104640 0x7fe53c1a23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe53be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c102aa0 0x7fe53c19ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53098/0 (socket says 172.21.15.28:53098) 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe53be00700 1 -- 172.21.15.28:0/2426950034 learned_addr learned my addr 172.21.15.28:0/2426950034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe533e00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe53c103ca0 0x7fe53c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe540c00700 1 -- 172.21.15.28:0/2426950034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe53c103ca0 msgr2=0x7fe53c19b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe540c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe53c103ca0 0x7fe53c19b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe540c00700 1 -- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c102aa0 msgr2=0x7fe53c19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe540c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c102aa0 0x7fe53c19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.869+0000 7fe540c00700 1 -- 172.21.15.28:0/2426950034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53c1a2a80 con 0x7fe53c104640 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.870+0000 7fe540c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe53c104640 0x7fe53c1a23e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe53400e900 tx=0x7fe53400ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.870+0000 7fe539600700 1 -- 172.21.15.28:0/2426950034 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534004220 con 0x7fe53c104640 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.870+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe53c1a2d70 con 0x7fe53c104640 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.870+0000 7fe539600700 1 -- 172.21.15.28:0/2426950034 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5340043c0 con 0x7fe53c104640 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.870+0000 7fe539600700 1 -- 172.21.15.28:0/2426950034 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534003d80 con 0x7fe53c104640 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.870+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe53c1a3250 con 0x7fe53c104640 2024-04-26T01:34:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.871+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe53c069760 con 0x7fe53c104640 2024-04-26T01:34:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.873+0000 7fe539600700 1 -- 172.21.15.28:0/2426950034 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe53400c780 con 0x7fe53c104640 2024-04-26T01:34:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.874+0000 7fe539600700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe5200a3cc0 0x7fe5200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.875+0000 7fe539600700 1 -- 172.21.15.28:0/2426950034 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fe5340cfb70 con 0x7fe53c104640 2024-04-26T01:34:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.875+0000 7fe53be00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe5200a3cc0 0x7fe5200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.875+0000 7fe53be00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe5200a3cc0 0x7fe5200a6180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009da0 tx=0x7fe52c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:11.879+0000 7fe539600700 1 -- 172.21.15.28:0/2426950034 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe53c069760 con 0x7fe53c104640 2024-04-26T01:34:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:11 smithi053 ceph-mon[32593]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:34:12.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:11 smithi028 ceph-mon[28247]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:34:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:12.121+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe53c10ab90 con 0x7fe5200a3cc0 2024-04-26T01:34:12.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:11 smithi169 ceph-mon[32851]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:34:12.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:12 smithi053 ceph-mon[32593]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-26T01:34:12.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:12 smithi053 ceph-mon[32593]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:12.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:12.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:12.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:12 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:12 smithi028 ceph-mon[28247]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-26T01:34:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:12 smithi028 ceph-mon[28247]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:13.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:12 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:13.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:12 smithi169 ceph-mon[32851]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-26T01:34:13.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:12 smithi169 ceph-mon[32851]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:12 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:14.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:14 smithi028 ceph-mon[28247]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:34:14.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:14 smithi053 ceph-mon[32593]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:34:15.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:14 smithi169 ceph-mon[32851]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:34:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:16 smithi053 ceph-mon[32593]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T01:34:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:16 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/469558500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]: dispatch 2024-04-26T01:34:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:16 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]: dispatch 2024-04-26T01:34:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:16 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]': finished 2024-04-26T01:34:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:16 smithi053 ceph-mon[32593]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T01:34:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:16 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:17.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:16 smithi028 ceph-mon[28247]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T01:34:17.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:16 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/469558500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]: dispatch 2024-04-26T01:34:17.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:16 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]: dispatch 2024-04-26T01:34:17.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:16 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]': finished 2024-04-26T01:34:17.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:16 smithi028 ceph-mon[28247]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T01:34:17.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:16 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:17.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:16 smithi169 ceph-mon[32851]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T01:34:17.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:16 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/469558500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]: dispatch 2024-04-26T01:34:17.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:16 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]: dispatch 2024-04-26T01:34:17.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:16 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41a7832e-3641-4f51-b26d-c928e268c72f"}]': finished 2024-04-26T01:34:17.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:16 smithi169 ceph-mon[32851]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T01:34:17.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:16 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:18.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:17 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2597513346' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:18.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:17 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2597513346' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:17 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2597513346' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:19.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:18 smithi028 ceph-mon[28247]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:34:19.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:18 smithi169 ceph-mon[32851]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:34:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:18 smithi053 ceph-mon[32593]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:34:21.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:20 smithi028 ceph-mon[28247]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:21.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:20 smithi169 ceph-mon[32851]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:20 smithi053 ceph-mon[32593]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:22.988 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:22 smithi028 ceph-mon[28247]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:23.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:22 smithi169 ceph-mon[32851]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:23.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:22 smithi053 ceph-mon[32593]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:25.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:24 smithi028 ceph-mon[28247]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:25.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:24 smithi169 ceph-mon[32851]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:25.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:24 smithi053 ceph-mon[32593]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:26.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:26 smithi028 ceph-mon[28247]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:26.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:26 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:34:26.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:26 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:27.019 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:26 smithi053 ceph-mon[32593]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:27.019 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:26 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:34:27.019 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:26 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:27.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:26 smithi169 ceph-mon[32851]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:27.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:26 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:34:27.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:26 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:28.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:27 smithi028 ceph-mon[28247]: Deploying daemon osd.9 on smithi028 2024-04-26T01:34:28.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:27 smithi169 ceph-mon[32851]: Deploying daemon osd.9 on smithi028 2024-04-26T01:34:28.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:27 smithi053 ceph-mon[32593]: Deploying daemon osd.9 on smithi028 2024-04-26T01:34:28.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:28 smithi028 ceph-mon[28247]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:29.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:28 smithi169 ceph-mon[32851]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:28 smithi053 ceph-mon[32593]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:31.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:30 smithi028 ceph-mon[28247]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:31.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:30 smithi169 ceph-mon[32851]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:31.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:30 smithi053 ceph-mon[32593]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:32.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:32 smithi028 ceph-mon[28247]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:33.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:32 smithi169 ceph-mon[32851]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:32 smithi053 ceph-mon[32593]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:35.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:34 smithi028 ceph-mon[28247]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:35.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:34 smithi169 ceph-mon[32851]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:35.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:34 smithi053 ceph-mon[32593]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:36.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:36.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:36.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:36.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:36.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:36.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:36.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:36.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:36.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:36 smithi028 ceph-mon[28247]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:37.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:36 smithi169 ceph-mon[32851]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:36 smithi053 ceph-mon[32593]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:38.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:38 smithi028 ceph-mon[28247]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:39.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:38 smithi169 ceph-mon[32851]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:38 smithi053 ceph-mon[32593]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:40.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:39 smithi169 ceph-mon[32851]: from='osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:34:40.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:39 smithi169 ceph-mon[32851]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:34:40.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:40.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:39 smithi053 ceph-mon[32593]: from='osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:34:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:39 smithi053 ceph-mon[32593]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:34:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:40.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:39 smithi028 ceph-mon[28247]: from='osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:34:40.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:39 smithi028 ceph-mon[28247]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:34:40.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:40.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.986+0000 7fe539600700 1 -- 172.21.15.28:0/2426950034 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe53c10ab90 con 0x7fe5200a3cc0 2024-04-26T01:34:40.983 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-04-26T01:34:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe5200a3cc0 msgr2=0x7fe5200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe5200a3cc0 0x7fe5200a6180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009da0 tx=0x7fe52c00b040 comp rx=0 tx=0).stop 2024-04-26T01:34:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe53c104640 msgr2=0x7fe53c1a23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe53c104640 0x7fe53c1a23e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe53400e900 tx=0x7fe53400ec10 comp rx=0 tx=0).stop 2024-04-26T01:34:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 shutdown_connections 2024-04-26T01:34:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe5200a3cc0 0x7fe5200a6180 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe53c104640 0x7fe53c1a23e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c102aa0 0x7fe53c19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 --2- 172.21.15.28:0/2426950034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe53c103ca0 0x7fe53c19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 >> 172.21.15.28:0/2426950034 conn(0x7fe53c0fe040 msgr2=0x7fe53c109470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 shutdown_connections 2024-04-26T01:34:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:34:40.990+0000 7fe542c00700 1 -- 172.21.15.28:0/2426950034 wait complete. 2024-04-26T01:34:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:40 smithi169 ceph-mon[32851]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:40 smithi169 ceph-mon[32851]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:34:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:40 smithi169 ceph-mon[32851]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T01:34:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:40 smithi169 ceph-mon[32851]: from='osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:34:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:40 smithi169 ceph-mon[32851]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:34:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:41.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:40 smithi053 ceph-mon[32593]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:40 smithi053 ceph-mon[32593]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:34:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:40 smithi053 ceph-mon[32593]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T01:34:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:40 smithi053 ceph-mon[32593]: from='osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:34:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:40 smithi053 ceph-mon[32593]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:34:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:41.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:40 smithi028 ceph-mon[28247]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:41.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:40 smithi028 ceph-mon[28247]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:34:41.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:40 smithi028 ceph-mon[28247]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T01:34:41.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:40 smithi028 ceph-mon[28247]: from='osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:34:41.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:41.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:40 smithi028 ceph-mon[28247]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-26T01:34:41.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:41.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:41.920 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T01:34:41.925 INFO:tasks.cephadm:Deploying osd.10 on smithi053 with /dev/vg_nvme/lv_1... 2024-04-26T01:34:41.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:34:42.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:41 smithi028 ceph-mon[28247]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:34:42.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:41 smithi028 ceph-mon[28247]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T01:34:42.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:41 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:41 smithi169 ceph-mon[32851]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:41 smithi169 ceph-mon[32851]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:42.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:42.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:41 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:42.183 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:34:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:41 smithi053 ceph-mon[32593]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-26T01:34:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:41 smithi053 ceph-mon[32593]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T01:34:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:41 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:42 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:34:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:42 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:34:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:42 smithi053 ceph-mon[32593]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:42 smithi053 ceph-mon[32593]: osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] boot 2024-04-26T01:34:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:42 smithi053 ceph-mon[32593]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T01:34:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:43.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:42 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:34:43.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:42 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:34:43.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:42 smithi028 ceph-mon[28247]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:43.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:43.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:42 smithi028 ceph-mon[28247]: osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] boot 2024-04-26T01:34:43.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:42 smithi028 ceph-mon[28247]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T01:34:43.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:43.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:42 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:34:43.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:42 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:34:43.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:42 smithi169 ceph-mon[32851]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:34:43.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:43.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:42 smithi169 ceph-mon[32851]: osd.9 [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] boot 2024-04-26T01:34:43.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:42 smithi169 ceph-mon[32851]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T01:34:43.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:34:44.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:43 smithi028 ceph-mon[28247]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T01:34:44.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:43 smithi053 ceph-mon[32593]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T01:34:44.322 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:34:44.322 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:34:44.322 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-26T01:34:44.322 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-26T01:34:44.322 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00836806 s, 1.3 GB/s 2024-04-26T01:34:44.322 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-26T01:34:44.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:43 smithi169 ceph-mon[32851]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T01:34:44.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_1 2024-04-26T01:34:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:44 smithi053 ceph-mon[32593]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:45.238 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:34:45.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:44 smithi028 ceph-mon[28247]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:45.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:44 smithi169 ceph-mon[32851]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:46.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 -- 172.21.15.53:0/435796091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 msgr2=0x7f7884109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:46.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 --2- 172.21.15.53:0/435796091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 0x7f7884109e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f787c00b0d0 tx=0x7f787c00b3e0 comp rx=0 tx=0).stop 2024-04-26T01:34:46.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 -- 172.21.15.53:0/435796091 shutdown_connections 2024-04-26T01:34:46.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 --2- 172.21.15.53:0/435796091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 0x7f7884109e10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:46.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 --2- 172.21.15.53:0/435796091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78841019c0 0x7f7884101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:46.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 --2- 172.21.15.53:0/435796091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7884102bc0 0x7f7884103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:46.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 -- 172.21.15.53:0/435796091 >> 172.21.15.53:0/435796091 conn(0x7f78840fcf60 msgr2=0x7f78840ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:46.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 -- 172.21.15.53:0/435796091 shutdown_connections 2024-04-26T01:34:46.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.992+0000 7f788be18700 1 -- 172.21.15.53:0/435796091 wait complete. 2024-04-26T01:34:46.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788be18700 1 Processor -- start 2024-04-26T01:34:46.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788be18700 1 -- start start 2024-04-26T01:34:46.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788be18700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78841019c0 0x7f788406bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788be18700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7884102bc0 0x7f788419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788be18700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 0x7f78841a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788be18700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f788410d5a0 con 0x7f7884102bc0 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788be18700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f788410d420 con 0x7f7884103560 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788be18700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f788410d720 con 0x7f78841019c0 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788a3b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 0x7f78841a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788a3b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 0x7f78841a00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47718/0 (socket says 172.21.15.53:47718) 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f788a3b5700 1 -- 172.21.15.53:0/2138433508 learned_addr learned my addr 172.21.15.53:0/2138433508 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.993+0000 7f7889bb4700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78841019c0 0x7f788406bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f788a3b5700 1 -- 172.21.15.53:0/2138433508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78841019c0 msgr2=0x7f788406bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f788a3b5700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78841019c0 0x7f788406bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f788a3b5700 1 -- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7884102bc0 msgr2=0x7f788419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:46.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f788a3b5700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7884102bc0 0x7f788419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:46.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f788a3b5700 1 -- 172.21.15.53:0/2138433508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78841a0980 con 0x7f7884103560 2024-04-26T01:34:46.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f7889bb4700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78841019c0 0x7f788406bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:34:46.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f788a3b5700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 0x7f78841a00b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f787c004d50 tx=0x7f787c004e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:46.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f787affd700 1 -- 172.21.15.53:0/2138433508 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c01f030 con 0x7f7884103560 2024-04-26T01:34:46.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.994+0000 7f787affd700 1 -- 172.21.15.53:0/2138433508 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787c007bc0 con 0x7f7884103560 2024-04-26T01:34:46.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.995+0000 7f787affd700 1 -- 172.21.15.53:0/2138433508 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c01b690 con 0x7f7884103560 2024-04-26T01:34:46.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.995+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78841a0c10 con 0x7f7884103560 2024-04-26T01:34:46.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.995+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78841a1090 con 0x7f7884103560 2024-04-26T01:34:46.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.995+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f788402df80 con 0x7f7884103560 2024-04-26T01:34:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.997+0000 7f787affd700 1 -- 172.21.15.53:0/2138433508 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f787c022030 con 0x7f7884103560 2024-04-26T01:34:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.997+0000 7f787affd700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f78700a3c40 0x7f78700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.997+0000 7f787affd700 1 -- 172.21.15.53:0/2138433508 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f787c00f020 con 0x7f7884103560 2024-04-26T01:34:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.998+0000 7f7889bb4700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f78700a3c40 0x7f78700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:46.999+0000 7f7889bb4700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f78700a3c40 0x7f78700a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7884102a20 tx=0x7f7880008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:46.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:47.001+0000 7f787affd700 1 -- 172.21.15.53:0/2138433508 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f787c029070 con 0x7f7884103560 2024-04-26T01:34:46.997 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:46 smithi053 ceph-mon[32593]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:47.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:34:47.220+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7884062e80 con 0x7f78700a3c40 2024-04-26T01:34:47.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:46 smithi028 ceph-mon[28247]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:47.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:46 smithi169 ceph-mon[32851]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi028 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:48.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:48.211 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.211 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:48.211 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:48.211 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:48.211 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi028 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-26T01:34:48.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:48.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:48.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:48.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:48.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:48.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi028 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:34:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:49 smithi053 ceph-mon[32593]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:49.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:49 smithi028 ceph-mon[28247]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:49.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:49 smithi169 ceph-mon[32851]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:51 smithi053 ceph-mon[32593]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:51 smithi053 ceph-mon[32593]: from='client.? 172.21.15.53:0/1901570436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]: dispatch 2024-04-26T01:34:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:51 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]: dispatch 2024-04-26T01:34:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:51 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]': finished 2024-04-26T01:34:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:51 smithi053 ceph-mon[32593]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T01:34:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:34:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:51 smithi053 ceph-mon[32593]: from='client.? 172.21.15.53:0/1651845918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:51.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:51 smithi028 ceph-mon[28247]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:51.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:51 smithi028 ceph-mon[28247]: from='client.? 172.21.15.53:0/1901570436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]: dispatch 2024-04-26T01:34:51.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:51 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]: dispatch 2024-04-26T01:34:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:51 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]': finished 2024-04-26T01:34:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:51 smithi028 ceph-mon[28247]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T01:34:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:34:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:51 smithi028 ceph-mon[28247]: from='client.? 172.21.15.53:0/1651845918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:51.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:51 smithi169 ceph-mon[32851]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:51 smithi169 ceph-mon[32851]: from='client.? 172.21.15.53:0/1901570436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]: dispatch 2024-04-26T01:34:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:51 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]: dispatch 2024-04-26T01:34:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:51 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef2c5cbd-b772-4910-b007-c50899067c2a"}]': finished 2024-04-26T01:34:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:51 smithi169 ceph-mon[32851]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T01:34:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:34:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:51 smithi169 ceph-mon[32851]: from='client.? 172.21.15.53:0/1651845918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:53 smithi053 ceph-mon[32593]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:53.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:53 smithi028 ceph-mon[28247]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:53.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:53 smithi169 ceph-mon[32851]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:55.378 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:55 smithi053 ceph-mon[32593]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:55.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:55 smithi028 ceph-mon[28247]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:55.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:55 smithi169 ceph-mon[32851]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:57.465 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:57 smithi053 ceph-mon[32593]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:57.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:57 smithi028 ceph-mon[28247]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:57.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:57 smithi169 ceph-mon[32851]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:58.449 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:58 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:34:58.450 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:58 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:58.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:34:58.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:58 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:58.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:58 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:34:58.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:58 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:59 smithi053 ceph-mon[32593]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:34:59 smithi053 ceph-mon[32593]: Deploying daemon osd.10 on smithi053 2024-04-26T01:34:59.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:59 smithi028 ceph-mon[28247]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:59.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:34:59 smithi028 ceph-mon[28247]: Deploying daemon osd.10 on smithi053 2024-04-26T01:34:59.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:59 smithi169 ceph-mon[32851]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:34:59.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:34:59 smithi169 ceph-mon[32851]: Deploying daemon osd.10 on smithi053 2024-04-26T01:35:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:01 smithi053 ceph-mon[32593]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:01.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:01 smithi028 ceph-mon[28247]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:01.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:01 smithi169 ceph-mon[32851]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:03 smithi053 ceph-mon[32593]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:03.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:03 smithi028 ceph-mon[28247]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:03.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:03 smithi169 ceph-mon[32851]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:05.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:05 smithi053 ceph-mon[32593]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:05.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:05 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:05.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:05 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:05.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:05 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:05.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:05 smithi028 ceph-mon[28247]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:05.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:05 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:05.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:05 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:05.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:05 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:05.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:05 smithi169 ceph-mon[32851]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:05.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:05 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:05.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:05 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:05.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:05 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:07.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:07 smithi028 ceph-mon[28247]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:07.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:07 smithi169 ceph-mon[32851]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:07.419 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:07 smithi053 ceph-mon[32593]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.209+0000 7f787affd700 1 -- 172.21.15.53:0/2138433508 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7884062e80 con 0x7f78700a3c40 2024-04-26T01:35:08.205 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 10 on host 'smithi053' 2024-04-26T01:35:08.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f78700a3c40 msgr2=0x7f78700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:08.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f78700a3c40 0x7f78700a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7884102a20 tx=0x7f7880008040 comp rx=0 tx=0).stop 2024-04-26T01:35:08.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 msgr2=0x7f78841a00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:08.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 0x7f78841a00b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f787c004d50 tx=0x7f787c004e30 comp rx=0 tx=0).stop 2024-04-26T01:35:08.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 shutdown_connections 2024-04-26T01:35:08.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f78700a3c40 0x7f78700a6100 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884103560 0x7f78841a00b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78841019c0 0x7f788406bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 --2- 172.21.15.53:0/2138433508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7884102bc0 0x7f788419da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 >> 172.21.15.53:0/2138433508 conn(0x7f78840fcf60 msgr2=0x7f7884105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:08.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 shutdown_connections 2024-04-26T01:35:08.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-26T01:35:08.211+0000 7f788be18700 1 -- 172.21.15.53:0/2138433508 wait complete. 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:08.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:08.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:08.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:08.921 DEBUG:teuthology.orchestra.run.smithi053:osd.10> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T01:35:08.924 INFO:tasks.cephadm:Deploying osd.11 on smithi169 with /dev/vg_nvme/lv_1... 2024-04-26T01:35:08.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:35:09.175 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:35:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:09 smithi028 ceph-mon[28247]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:09 smithi028 ceph-mon[28247]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:35:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:09 smithi028 ceph-mon[28247]: from='osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:35:09.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:09 smithi028 ceph-mon[28247]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T01:35:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:09 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:09.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:09 smithi028 ceph-mon[28247]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:35:09.532 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:09 smithi169 ceph-mon[32851]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:09.532 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:09 smithi169 ceph-mon[32851]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:35:09.532 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:09 smithi169 ceph-mon[32851]: from='osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:35:09.532 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:09 smithi169 ceph-mon[32851]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T01:35:09.532 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:09 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:09.532 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:09 smithi169 ceph-mon[32851]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:35:09.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:09 smithi053 ceph-mon[32593]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:09.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:09 smithi053 ceph-mon[32593]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:35:09.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:09 smithi053 ceph-mon[32593]: from='osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:35:09.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:09 smithi053 ceph-mon[32593]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T01:35:09.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:09 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:09.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:09 smithi053 ceph-mon[32593]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-26T01:35:09.967 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:35:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:35:09.762+0000 7f905402e700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T01:35:09.967 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:35:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:35:09.776+0000 7f904b618700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:35:10.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:10 smithi053 ceph-mon[32593]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:35:10.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:10 smithi053 ceph-mon[32593]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T01:35:10.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:10 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:10.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:10 smithi053 ceph-mon[32593]: from='osd.10 ' entity='osd.10' 2024-04-26T01:35:10.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:10 smithi053 ceph-mon[32593]: osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] boot 2024-04-26T01:35:10.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:10 smithi053 ceph-mon[32593]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T01:35:10.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:10 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:10.572 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:10 smithi169 ceph-mon[32851]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:35:10.572 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:10 smithi169 ceph-mon[32851]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T01:35:10.572 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:10 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:10.572 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:10 smithi169 ceph-mon[32851]: from='osd.10 ' entity='osd.10' 2024-04-26T01:35:10.572 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:10 smithi169 ceph-mon[32851]: osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] boot 2024-04-26T01:35:10.572 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:10 smithi169 ceph-mon[32851]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T01:35:10.572 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:10 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:10.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:10 smithi028 ceph-mon[28247]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-26T01:35:10.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:10 smithi028 ceph-mon[28247]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T01:35:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:10 smithi028 ceph-mon[28247]: from='osd.10 ' entity='osd.10' 2024-04-26T01:35:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:10 smithi028 ceph-mon[28247]: osd.10 [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] boot 2024-04-26T01:35:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:10 smithi028 ceph-mon[28247]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T01:35:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:10 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:35:11.199 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:35:11.199 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:35:11.199 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-04-26T01:35:11.199 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-04-26T01:35:11.199 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011976 s, 876 MB/s 2024-04-26T01:35:11.199 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-04-26T01:35:11.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:11 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:35:11.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:11 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:35:11.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:11 smithi169 ceph-mon[32851]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:11.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:11 smithi169 ceph-mon[32851]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T01:35:11.671 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:11 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:35:11.671 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:11 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:35:11.671 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:11 smithi053 ceph-mon[32593]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:11.671 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:11 smithi053 ceph-mon[32593]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T01:35:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:11 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:35:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:11 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:35:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:11 smithi028 ceph-mon[28247]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:35:11.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:11 smithi028 ceph-mon[28247]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T01:35:11.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi169:vg_nvme/lv_1 2024-04-26T01:35:12.085 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi053 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi053 to 4273M 2024-04-26T01:35:13.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:13.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:13.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:13 smithi169 ceph-mon[32851]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi053 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi053 to 4273M 2024-04-26T01:35:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:13 smithi028 ceph-mon[28247]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T01:35:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T01:35:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi053 2024-04-26T01:35:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi053 to 4273M 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:13 smithi053 ceph-mon[32593]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T01:35:13.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 -- 172.21.15.169:0/3008258444 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74100be0 msgr2=0x7f3f74101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/3008258444 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74100be0 0x7f3f74101040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009990 tx=0x7f3f60009ca0 comp rx=0 tx=0).stop 2024-04-26T01:35:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 -- 172.21.15.169:0/3008258444 shutdown_connections 2024-04-26T01:35:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/3008258444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f74101580 0x7f3f7410b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/3008258444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74104160 0x7f3f74104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/3008258444 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74100be0 0x7f3f74101040 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 -- 172.21.15.169:0/3008258444 >> 172.21.15.169:0/3008258444 conn(0x7f3f740fc7a0 msgr2=0x7f3f740febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 -- 172.21.15.169:0/3008258444 shutdown_connections 2024-04-26T01:35:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.734+0000 7f3f7b15c700 1 -- 172.21.15.169:0/3008258444 wait complete. 2024-04-26T01:35:13.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f7b15c700 1 Processor -- start 2024-04-26T01:35:13.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f7b15c700 1 -- start start 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f7b15c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f74100be0 0x7f3f74198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f7b15c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74101580 0x7f3f741991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f7b15c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74104160 0x7f3f741a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f7b15c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7410e630 con 0x7f3f74104160 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f7b15c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7410e4b0 con 0x7f3f74100be0 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f7b15c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7410e7b0 con 0x7f3f74101580 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74101580 0x7f3f741991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74101580 0x7f3f741991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42968/0 (socket says 172.21.15.169:42968) 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 -- 172.21.15.169:0/1053275218 learned_addr learned my addr 172.21.15.169:0/1053275218 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:35:13.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f78ef8700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f74100be0 0x7f3f74198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 -- 172.21.15.169:0/1053275218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f74100be0 msgr2=0x7f3f74198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f74100be0 0x7f3f74198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 -- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74104160 msgr2=0x7f3f741a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74104160 0x7f3f741a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 -- 172.21.15.169:0/1053275218 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f741a08f0 con 0x7f3f74101580 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f78ef8700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f74100be0 0x7f3f74198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.735+0000 7f3f73fff700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74101580 0x7f3f741991b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006010 tx=0x7f3f60005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.736+0000 7f3f71ffb700 1 -- 172.21.15.169:0/1053275218 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60027070 con 0x7f3f74101580 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.736+0000 7f3f71ffb700 1 -- 172.21.15.169:0/1053275218 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f60004100 con 0x7f3f74101580 2024-04-26T01:35:13.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.736+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f741a0b80 con 0x7f3f74101580 2024-04-26T01:35:13.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.736+0000 7f3f71ffb700 1 -- 172.21.15.169:0/1053275218 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000f460 con 0x7f3f74101580 2024-04-26T01:35:13.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.736+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f741a1030 con 0x7f3f74101580 2024-04-26T01:35:13.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.737+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f74069730 con 0x7f3f74101580 2024-04-26T01:35:13.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.738+0000 7f3f71ffb700 1 -- 172.21.15.169:0/1053275218 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3f6002a070 con 0x7f3f74101580 2024-04-26T01:35:13.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.738+0000 7f3f71ffb700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f640a3c40 0x7f3f640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:13.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.739+0000 7f3f71ffb700 1 -- 172.21.15.169:0/1053275218 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f3f600d10f0 con 0x7f3f74101580 2024-04-26T01:35:13.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.739+0000 7f3f78ef8700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f640a3c40 0x7f3f640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:13.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.740+0000 7f3f78ef8700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f640a3c40 0x7f3f640a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f74199b50 tx=0x7f3f68008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:13.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.742+0000 7f3f71ffb700 1 -- 172.21.15.169:0/1053275218 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f60013070 con 0x7f3f74101580 2024-04-26T01:35:13.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:13.950+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3f74062e80 con 0x7f3f640a3c40 2024-04-26T01:35:15.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:14 smithi169 ceph-mon[32851]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:15.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:14 smithi169 ceph-mon[32851]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:35:15.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:14 smithi169 ceph-mon[32851]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:15.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:15.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:15.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:14 smithi053 ceph-mon[32593]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:14 smithi053 ceph-mon[32593]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:35:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:14 smithi053 ceph-mon[32593]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:15.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:14 smithi028 ceph-mon[28247]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:15.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:14 smithi028 ceph-mon[28247]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:35:15.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:14 smithi028 ceph-mon[28247]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:15.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:15.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:15.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:17.194 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:16 smithi169 ceph-mon[32851]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-26T01:35:17.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:16 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/83065973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]: dispatch 2024-04-26T01:35:17.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:16 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]: dispatch 2024-04-26T01:35:17.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:16 smithi169 ceph-mon[32851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]': finished 2024-04-26T01:35:17.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:16 smithi169 ceph-mon[32851]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T01:35:17.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:16 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:16 smithi053 ceph-mon[32593]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-26T01:35:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:16 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/83065973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]: dispatch 2024-04-26T01:35:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:16 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]: dispatch 2024-04-26T01:35:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:16 smithi053 ceph-mon[32593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]': finished 2024-04-26T01:35:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:16 smithi053 ceph-mon[32593]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T01:35:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:16 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:17.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:16 smithi028 ceph-mon[28247]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-26T01:35:17.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:16 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/83065973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]: dispatch 2024-04-26T01:35:17.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:16 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]: dispatch 2024-04-26T01:35:17.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:16 smithi028 ceph-mon[28247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "673532a1-0742-4a52-82bf-8dd61d64a955"}]': finished 2024-04-26T01:35:17.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:16 smithi028 ceph-mon[28247]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T01:35:17.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:16 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:18.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:17 smithi053 ceph-mon[32593]: from='client.? 172.21.15.169:0/3132347689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:18.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:17 smithi028 ceph-mon[28247]: from='client.? 172.21.15.169:0/3132347689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:18.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:17 smithi169 ceph-mon[32851]: from='client.? 172.21.15.169:0/3132347689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:18 smithi053 ceph-mon[32593]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T01:35:19.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:18 smithi028 ceph-mon[28247]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T01:35:19.408 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:18 smithi169 ceph-mon[32851]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T01:35:21.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:20 smithi169 ceph-mon[32851]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:35:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:20 smithi053 ceph-mon[32593]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:35:21.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:20 smithi028 ceph-mon[28247]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:35:23.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:22 smithi169 ceph-mon[32851]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:35:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:22 smithi053 ceph-mon[32593]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:35:23.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:22 smithi028 ceph-mon[28247]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:35:25.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:24 smithi053 ceph-mon[32593]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T01:35:25.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:35:25.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:25.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:24 smithi053 ceph-mon[32593]: Deploying daemon osd.11 on smithi169 2024-04-26T01:35:25.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:24 smithi169 ceph-mon[32851]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T01:35:25.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:35:25.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:25.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:24 smithi169 ceph-mon[32851]: Deploying daemon osd.11 on smithi169 2024-04-26T01:35:25.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:24 smithi028 ceph-mon[28247]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T01:35:25.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:35:25.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:25.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:24 smithi028 ceph-mon[28247]: Deploying daemon osd.11 on smithi169 2024-04-26T01:35:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:26 smithi053 ceph-mon[32593]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:27.227 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:26 smithi169 ceph-mon[32851]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:27.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:26 smithi028 ceph-mon[28247]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:29.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:28 smithi053 ceph-mon[32593]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:29.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:28 smithi028 ceph-mon[28247]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:29.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:28 smithi169 ceph-mon[32851]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:31.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:31 smithi169 ceph-mon[32851]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:31.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:31 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:31.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:31 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:31.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:31 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:31 smithi053 ceph-mon[32593]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:31 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:31 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:31 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:31 smithi028 ceph-mon[28247]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:31 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:31.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:31 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:31.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:31 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:33.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:33 smithi169 ceph-mon[32851]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:33 smithi053 ceph-mon[32593]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:33.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:33 smithi028 ceph-mon[28247]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:34.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.340+0000 7f3f71ffb700 1 -- 172.21.15.169:0/1053275218 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3f74062e80 con 0x7f3f640a3c40 2024-04-26T01:35:34.336 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 11 on host 'smithi169' 2024-04-26T01:35:34.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f640a3c40 msgr2=0x7f3f640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:34.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f640a3c40 0x7f3f640a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f74199b50 tx=0x7f3f68008040 comp rx=0 tx=0).stop 2024-04-26T01:35:34.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74101580 msgr2=0x7f3f741991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:34.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74101580 0x7f3f741991b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006010 tx=0x7f3f60005e80 comp rx=0 tx=0).stop 2024-04-26T01:35:34.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 shutdown_connections 2024-04-26T01:35:34.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f640a3c40 0x7f3f640a6100 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:34.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74104160 0x7f3f741a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:34.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f74100be0 0x7f3f74198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:34.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 --2- 172.21.15.169:0/1053275218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f74101580 0x7f3f741991b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:34.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 >> 172.21.15.169:0/1053275218 conn(0x7f3f740fc7a0 msgr2=0x7f3f74107200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:34.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 shutdown_connections 2024-04-26T01:35:34.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:34.342+0000 7f3f7b15c700 1 -- 172.21.15.169:0/1053275218 wait complete. 2024-04-26T01:35:34.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:34.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:34.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:34 smithi169 ceph-mon[32851]: from='osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:35:34.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:34 smithi169 ceph-mon[32851]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:35:34.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:34.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:34 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:34 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:34 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:34 smithi053 ceph-mon[32593]: from='osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:35:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:34 smithi053 ceph-mon[32593]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:35:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:34 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:34 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:35.022 DEBUG:teuthology.orchestra.run.smithi169:osd.11> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T01:35:35.024 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T01:35:35.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T01:35:35.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:35.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:35.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:34 smithi028 ceph-mon[28247]: from='osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:35:35.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:34 smithi028 ceph-mon[28247]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:35:35.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:35.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:34 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:35.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:35 smithi169 ceph-mon[32851]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:35.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:35 smithi169 ceph-mon[32851]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:35:35.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:35 smithi169 ceph-mon[32851]: from='osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:35:35.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:35 smithi169 ceph-mon[32851]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T01:35:35.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:35 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:35.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:35 smithi169 ceph-mon[32851]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:35:35.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:35 smithi028 ceph-mon[28247]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:35.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:35 smithi028 ceph-mon[28247]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:35:35.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:35 smithi028 ceph-mon[28247]: from='osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:35:35.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:35 smithi028 ceph-mon[28247]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T01:35:35.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:35 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:35.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:35 smithi028 ceph-mon[28247]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:35:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:35 smithi053 ceph-mon[32593]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:35 smithi053 ceph-mon[32593]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:35:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:35 smithi053 ceph-mon[32593]: from='osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:35:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:35 smithi053 ceph-mon[32593]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T01:35:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:35 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:35 smithi053 ceph-mon[32593]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-04-26T01:35:36.159 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:35:35 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:35:35.993+0000 7fcc0d0f8700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T01:35:36.159 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:35:36 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:35:36.008+0000 7fcc04ee3700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:35:36.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:36 smithi169 ceph-mon[32851]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:35:36.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:36 smithi169 ceph-mon[32851]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T01:35:36.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:36.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:36.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:36 smithi169 ceph-mon[32851]: from='osd.11 ' entity='osd.11' 2024-04-26T01:35:36.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:36 smithi053 ceph-mon[32593]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:35:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:36 smithi053 ceph-mon[32593]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T01:35:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:36 smithi053 ceph-mon[32593]: from='osd.11 ' entity='osd.11' 2024-04-26T01:35:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:37.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:36 smithi028 ceph-mon[28247]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-04-26T01:35:37.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:36 smithi028 ceph-mon[28247]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T01:35:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:36 smithi028 ceph-mon[28247]: from='osd.11 ' entity='osd.11' 2024-04-26T01:35:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:37.581 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:37 smithi169 ceph-mon[32851]: purged_snaps scrub starts 2024-04-26T01:35:37.581 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:37 smithi169 ceph-mon[32851]: purged_snaps scrub ok 2024-04-26T01:35:37.581 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:37 smithi169 ceph-mon[32851]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:37.581 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:37 smithi169 ceph-mon[32851]: osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] boot 2024-04-26T01:35:37.582 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:37 smithi169 ceph-mon[32851]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T01:35:37.582 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:37 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:37.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:37 smithi028 ceph-mon[28247]: purged_snaps scrub starts 2024-04-26T01:35:37.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:37 smithi028 ceph-mon[28247]: purged_snaps scrub ok 2024-04-26T01:35:37.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:37 smithi028 ceph-mon[28247]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:37.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:37 smithi028 ceph-mon[28247]: osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] boot 2024-04-26T01:35:37.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:37 smithi028 ceph-mon[28247]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T01:35:37.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:37 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:37 smithi053 ceph-mon[32593]: purged_snaps scrub starts 2024-04-26T01:35:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:37 smithi053 ceph-mon[32593]: purged_snaps scrub ok 2024-04-26T01:35:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:37 smithi053 ceph-mon[32593]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:35:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:37 smithi053 ceph-mon[32593]: osd.11 [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] boot 2024-04-26T01:35:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:37 smithi053 ceph-mon[32593]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T01:35:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:37 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 -- 172.21.15.28:0/4021316958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28105e50 msgr2=0x7f6d28108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 --2- 172.21.15.28:0/4021316958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28105e50 0x7f6d28108240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6d140098f0 tx=0x7f6d14009c00 comp rx=0 tx=0).stop 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 -- 172.21.15.28:0/4021316958 shutdown_connections 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 --2- 172.21.15.28:0/4021316958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d28108780 0x7f6d2810abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 --2- 172.21.15.28:0/4021316958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d28103520 0x7f6d28105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 --2- 172.21.15.28:0/4021316958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28105e50 0x7f6d28108240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 -- 172.21.15.28:0/4021316958 >> 172.21.15.28:0/4021316958 conn(0x7f6d280fcf30 msgr2=0x7f6d280ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 -- 172.21.15.28:0/4021316958 shutdown_connections 2024-04-26T01:35:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.133+0000 7f6d2d200700 1 -- 172.21.15.28:0/4021316958 wait complete. 2024-04-26T01:35:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.135+0000 7f6d2d200700 1 Processor -- start 2024-04-26T01:35:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.135+0000 7f6d2d200700 1 -- start start 2024-04-26T01:35:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.135+0000 7f6d2d200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28103520 0x7f6d28102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.135+0000 7f6d2d200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d28105e50 0x7f6d281014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.135+0000 7f6d2d200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d28108780 0x7f6d28101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.135+0000 7f6d2d200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810e570 con 0x7f6d28103520 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.135+0000 7f6d2d200700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810e3f0 con 0x7f6d28105e50 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d2d200700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810e6f0 con 0x7f6d28108780 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28103520 0x7f6d28102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d28105e50 0x7f6d281014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d28105e50 0x7f6d281014d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37770/0 (socket says 172.21.15.28:37770) 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28103520 0x7f6d28102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46098/0 (socket says 172.21.15.28:46098) 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25400700 1 -- 172.21.15.28:0/1402517968 learned_addr learned my addr 172.21.15.28:0/1402517968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d26800700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d28108780 0x7f6d28101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25400700 1 -- 172.21.15.28:0/1402517968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d28108780 msgr2=0x7f6d28101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25400700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d28108780 0x7f6d28101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25400700 1 -- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28103520 msgr2=0x7f6d28102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:38.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25400700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28103520 0x7f6d28102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d25400700 1 -- 172.21.15.28:0/1402517968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d281022c0 con 0x7f6d28105e50 2024-04-26T01:35:38.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.136+0000 7f6d26800700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d28108780 0x7f6d28101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:35:38.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.137+0000 7f6d25400700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d28105e50 0x7f6d281014d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400c010 tx=0x7f6d14005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:38.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.137+0000 7f6d1ea00700 1 -- 172.21.15.28:0/1402517968 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d14005510 con 0x7f6d28105e50 2024-04-26T01:35:38.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.137+0000 7f6d1ea00700 1 -- 172.21.15.28:0/1402517968 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d140056b0 con 0x7f6d28105e50 2024-04-26T01:35:38.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.137+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d281a2da0 con 0x7f6d28105e50 2024-04-26T01:35:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.137+0000 7f6d1ea00700 1 -- 172.21.15.28:0/1402517968 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d14003c10 con 0x7f6d28105e50 2024-04-26T01:35:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.138+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d281a31a0 con 0x7f6d28105e50 2024-04-26T01:35:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.139+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d2802df80 con 0x7f6d28105e50 2024-04-26T01:35:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.142+0000 7f6d1ea00700 1 -- 172.21.15.28:0/1402517968 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6d1400baa0 con 0x7f6d28105e50 2024-04-26T01:35:38.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.144+0000 7f6d1ea00700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6d0c0a3cc0 0x7f6d0c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:38.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.144+0000 7f6d1ea00700 1 -- 172.21.15.28:0/1402517968 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d140d0fe0 con 0x7f6d28105e50 2024-04-26T01:35:38.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.145+0000 7f6d25e00700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6d0c0a3cc0 0x7f6d0c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:38.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.145+0000 7f6d25e00700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6d0c0a3cc0 0x7f6d0c0a6180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009c60 tx=0x7f6d18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:38.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.148+0000 7f6d1ea00700 1 -- 172.21.15.28:0/1402517968 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d14013030 con 0x7f6d28105e50 2024-04-26T01:35:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.384+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f6d28063650 con 0x7f6d28105e50 2024-04-26T01:35:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.385+0000 7f6d1ea00700 1 -- 172.21.15.28:0/1402517968 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f6d1409de70 con 0x7f6d28105e50 2024-04-26T01:35:38.381 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:35:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.387+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6d0c0a3cc0 msgr2=0x7f6d0c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:38.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6d0c0a3cc0 0x7f6d0c0a6180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009c60 tx=0x7f6d18009260 comp rx=0 tx=0).stop 2024-04-26T01:35:38.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d28105e50 msgr2=0x7f6d281014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:38.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d28105e50 0x7f6d281014d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400c010 tx=0x7f6d14005270 comp rx=0 tx=0).stop 2024-04-26T01:35:38.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 shutdown_connections 2024-04-26T01:35:38.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f6d0c0a3cc0 0x7f6d0c0a6180 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d28108780 0x7f6d28101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d28103520 0x7f6d28102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 --2- 172.21.15.28:0/1402517968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d28105e50 0x7f6d281014d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 >> 172.21.15.28:0/1402517968 conn(0x7f6d280fcf30 msgr2=0x7f6d281053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 shutdown_connections 2024-04-26T01:35:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:38.388+0000 7f6d2d200700 1 -- 172.21.15.28:0/1402517968 wait complete. 2024-04-26T01:35:39.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:39.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: Detected new or changed devices on smithi169 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi169 to 4272M 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:39 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1402517968' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:35:39.398 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714095336,"num_in_osds":12,"osd_in_since":1714095316,"num_remapped_pgs":0} 2024-04-26T01:35:39.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: Detected new or changed devices on smithi169 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi169 to 4272M 2024-04-26T01:35:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:39 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1402517968' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:35:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T01:35:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: Detected new or changed devices on smithi169 2024-04-26T01:35:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi169 to 4272M 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:39.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:39 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1402517968' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:35:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:40.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:40.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:40.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:40 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:40.310 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:40.310 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:40.310 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:40 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:40 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:41.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:41 smithi169 ceph-mon[32851]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:41.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:41 smithi028 ceph-mon[28247]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:41.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:41 smithi053 ceph-mon[32593]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:42.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 -- 172.21.15.28:0/1504982423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 msgr2=0x7ff010102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:42.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 --2- 172.21.15.28:0/1504982423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 0x7ff010102f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff000009990 tx=0x7ff000009ca0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 -- 172.21.15.28:0/1504982423 shutdown_connections 2024-04-26T01:35:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 --2- 172.21.15.28:0/1504982423 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0101034b0 0x7ff01010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 --2- 172.21.15.28:0/1504982423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff010107940 0x7ff010107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 --2- 172.21.15.28:0/1504982423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 0x7ff010102f70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 -- 172.21.15.28:0/1504982423 >> 172.21.15.28:0/1504982423 conn(0x7ff010079ba0 msgr2=0x7ff010079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 -- 172.21.15.28:0/1504982423 shutdown_connections 2024-04-26T01:35:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.340+0000 7ff017800700 1 -- 172.21.15.28:0/1504982423 wait complete. 2024-04-26T01:35:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.341+0000 7ff017800700 1 Processor -- start 2024-04-26T01:35:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.341+0000 7ff017800700 1 -- start start 2024-04-26T01:35:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff017800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 0x7ff010074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff017800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0101034b0 0x7ff010075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff017800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff010107940 0x7ff0100757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff017800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff010112c30 con 0x7ff0101034b0 2024-04-26T01:35:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff017800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff010112ab0 con 0x7ff010107940 2024-04-26T01:35:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff017800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff010112db0 con 0x7ff010102af0 2024-04-26T01:35:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff00fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0101034b0 0x7ff010075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff00fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0101034b0 0x7ff010075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36612/0 (socket says 172.21.15.28:36612) 2024-04-26T01:35:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff00fe00700 1 -- 172.21.15.28:0/3926306390 learned_addr learned my addr 172.21.15.28:0/3926306390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:35:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff014e00700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 0x7ff010074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.342+0000 7ff015800700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff010107940 0x7ff0100757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00fe00700 1 -- 172.21.15.28:0/3926306390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 msgr2=0x7ff010074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00fe00700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 0x7ff010074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00fe00700 1 -- 172.21.15.28:0/3926306390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff010107940 msgr2=0x7ff0100757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00fe00700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff010107940 0x7ff0100757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00fe00700 1 -- 172.21.15.28:0/3926306390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0101a7630 con 0x7ff0101034b0 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff014e00700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 0x7ff010074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00fe00700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0101034b0 0x7ff010075280 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff0000057c0 tx=0x7ff000005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00d600700 1 -- 172.21.15.28:0/3926306390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff000026070 con 0x7ff0101034b0 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0101a78c0 con 0x7ff0101034b0 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00d600700 1 -- 172.21.15.28:0/3926306390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff00000b610 con 0x7ff0101034b0 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.343+0000 7ff00d600700 1 -- 172.21.15.28:0/3926306390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff000021e00 con 0x7ff0101034b0 2024-04-26T01:35:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.344+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0101a7df0 con 0x7ff0101034b0 2024-04-26T01:35:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.345+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff010069730 con 0x7ff0101034b0 2024-04-26T01:35:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.347+0000 7ff00d600700 1 -- 172.21.15.28:0/3926306390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff000029070 con 0x7ff0101034b0 2024-04-26T01:35:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.348+0000 7ff00d600700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feff80a3cc0 0x7feff80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.348+0000 7ff00d600700 1 -- 172.21.15.28:0/3926306390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff0000d0a90 con 0x7ff0101034b0 2024-04-26T01:35:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.348+0000 7ff014e00700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feff80a3cc0 0x7feff80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.349+0000 7ff014e00700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feff80a3cc0 0x7feff80a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff004009fd0 tx=0x7ff004009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:42.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.353+0000 7ff00d600700 1 -- 172.21.15.28:0/3926306390 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff000013070 con 0x7ff0101034b0 2024-04-26T01:35:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.589+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff01002e140 con 0x7ff0101034b0 2024-04-26T01:35:42.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.590+0000 7ff00d600700 1 -- 172.21.15.28:0/3926306390 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18432 (secure 0 0 0) 0x7ff00009d900 con 0x7ff0101034b0 2024-04-26T01:35:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:35:42.587 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:35:38.001339+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feff80a3cc0 msgr2=0x7feff80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feff80a3cc0 0x7feff80a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff004009fd0 tx=0x7ff004009260 comp rx=0 tx=0).stop 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0101034b0 msgr2=0x7ff010075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0101034b0 0x7ff010075280 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff0000057c0 tx=0x7ff000005ef0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 shutdown_connections 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feff80a3cc0 0x7feff80a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff010107940 0x7ff0100757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff010102af0 0x7ff010074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 --2- 172.21.15.28:0/3926306390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0101034b0 0x7ff010075280 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.593+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 >> 172.21.15.28:0/3926306390 conn(0x7ff010079ba0 msgr2=0x7ff010100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.594+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 shutdown_connections 2024-04-26T01:35:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:42.594+0000 7ff017800700 1 -- 172.21.15.28:0/3926306390 wait complete. 2024-04-26T01:35:43.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:43 smithi028 ceph-mon[28247]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:43.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:43 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3926306390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:35:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:43 smithi169 ceph-mon[32851]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:43 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3926306390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:35:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:43 smithi053 ceph-mon[32593]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:43 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3926306390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:35:43.488 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T01:31:11.765874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T01:35:43.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T01:35:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:45.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:45 smithi028 ceph-mon[28247]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:45.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:45 smithi169 ceph-mon[32851]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:45 smithi053 ceph-mon[32593]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:46.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.237+0000 7f1384c00700 1 -- 172.21.15.28:0/2286842287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 msgr2=0x7f1380105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:46.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.237+0000 7f1384c00700 1 --2- 172.21.15.28:0/2286842287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 0x7f1380105190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1370009990 tx=0x7f1370009ca0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.238+0000 7f1384c00700 1 -- 172.21.15.28:0/2286842287 shutdown_connections 2024-04-26T01:35:46.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.238+0000 7f1384c00700 1 --2- 172.21.15.28:0/2286842287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1380109e90 0x7f1380111a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.238+0000 7f1384c00700 1 --2- 172.21.15.28:0/2286842287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 0x7f1380105190 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.238+0000 7f1384c00700 1 --2- 172.21.15.28:0/2286842287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1380105760 0x7f13801097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.238+0000 7f1384c00700 1 -- 172.21.15.28:0/2286842287 >> 172.21.15.28:0/2286842287 conn(0x7f1380100620 msgr2=0x7f1380102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.238+0000 7f1384c00700 1 -- 172.21.15.28:0/2286842287 shutdown_connections 2024-04-26T01:35:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.238+0000 7f1384c00700 1 -- 172.21.15.28:0/2286842287 wait complete. 2024-04-26T01:35:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.239+0000 7f1384c00700 1 Processor -- start 2024-04-26T01:35:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.239+0000 7f1384c00700 1 -- start start 2024-04-26T01:35:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f1384c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 0x7f138019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f1384c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1380105760 0x7f138019b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f1384c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1380109e90 0x7f138019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f1384c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1380114d80 con 0x7f1380105760 2024-04-26T01:35:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f1384c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1380114c00 con 0x7f1380104db0 2024-04-26T01:35:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f1384c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1380114f00 con 0x7f1380109e90 2024-04-26T01:35:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f137ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1380105760 0x7f138019b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f137ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1380105760 0x7f138019b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36628/0 (socket says 172.21.15.28:36628) 2024-04-26T01:35:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f137ea00700 1 -- 172.21.15.28:0/1072425712 learned_addr learned my addr 172.21.15.28:0/1072425712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:35:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f137f400700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 0x7f138019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.240+0000 7f137fe00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1380109e90 0x7f138019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f137ea00700 1 -- 172.21.15.28:0/1072425712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1380109e90 msgr2=0x7f138019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f137ea00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1380109e90 0x7f138019fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f137ea00700 1 -- 172.21.15.28:0/1072425712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 msgr2=0x7f138019b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f137ea00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 0x7f138019b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f137ea00700 1 -- 172.21.15.28:0/1072425712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13801a02a0 con 0x7f1380105760 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f137f400700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 0x7f138019b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f137ea00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1380105760 0x7f138019b7d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f136c00e970 tx=0x7f136c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f1377e00700 1 -- 172.21.15.28:0/1072425712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c004a30 con 0x7f1380105760 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13801a0590 con 0x7f1380105760 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f1377e00700 1 -- 172.21.15.28:0/1072425712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f136c00cc90 con 0x7f1380105760 2024-04-26T01:35:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f1377e00700 1 -- 172.21.15.28:0/1072425712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c005610 con 0x7f1380105760 2024-04-26T01:35:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.241+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13801a7e10 con 0x7f1380105760 2024-04-26T01:35:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.243+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1380069730 con 0x7f1380105760 2024-04-26T01:35:46.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.247+0000 7f1377e00700 1 -- 172.21.15.28:0/1072425712 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f136c00c7e0 con 0x7f1380105760 2024-04-26T01:35:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.248+0000 7f1377e00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f13640a3c40 0x7f13640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.248+0000 7f1377e00700 1 -- 172.21.15.28:0/1072425712 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f136c0d02c0 con 0x7f1380105760 2024-04-26T01:35:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.248+0000 7f137f400700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f13640a3c40 0x7f13640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.249+0000 7f137f400700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f13640a3c40 0x7f13640a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f137000b790 tx=0x7f137000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:46.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.251+0000 7f1377e00700 1 -- 172.21.15.28:0/1072425712 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f136c053ff0 con 0x7f1380105760 2024-04-26T01:35:46.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.486+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f138002df80 con 0x7f1380105760 2024-04-26T01:35:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.487+0000 7f1377e00700 1 -- 172.21.15.28:0/1072425712 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f136c09d140 con 0x7f1380105760 2024-04-26T01:35:46.483 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-26T01:35:46.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.491+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f13640a3c40 msgr2=0x7f13640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:46.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.491+0000 7f1384c00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f13640a3c40 0x7f13640a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f137000b790 tx=0x7f137000b260 comp rx=0 tx=0).stop 2024-04-26T01:35:46.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.491+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1380105760 msgr2=0x7f138019b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:46.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.491+0000 7f1384c00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1380105760 0x7f138019b7d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f136c00e970 tx=0x7f136c00ec80 comp rx=0 tx=0).stop 2024-04-26T01:35:46.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.492+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 shutdown_connections 2024-04-26T01:35:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.492+0000 7f1384c00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f13640a3c40 0x7f13640a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.492+0000 7f1384c00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1380109e90 0x7f138019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.492+0000 7f1384c00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1380104db0 0x7f138019b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.492+0000 7f1384c00700 1 --2- 172.21.15.28:0/1072425712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1380105760 0x7f138019b7d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.492+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 >> 172.21.15.28:0/1072425712 conn(0x7f1380100620 msgr2=0x7f1380109020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.492+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 shutdown_connections 2024-04-26T01:35:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:46.492+0000 7f1384c00700 1 -- 172.21.15.28:0/1072425712 wait complete. 2024-04-26T01:35:47.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:47 smithi028 ceph-mon[28247]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:47.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:47 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1072425712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:35:47.362 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-04-26T01:35:47.362 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.b on smithi053 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.e on smithi053 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.h on smithi053 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.k on smithi053 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.c on smithi169 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.f on smithi169 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.i on smithi169 2024-04-26T01:35:47.363 INFO:tasks.cephadm:Adding mds.l on smithi169 2024-04-26T01:35:47.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l' 2024-04-26T01:35:47.392 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:47 smithi169 ceph-mon[32851]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:47.392 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:47 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1072425712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:35:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:47 smithi053 ceph-mon[32593]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:47 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1072425712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:35:47.618 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.347+0000 7fa24e28c700 1 -- 172.21.15.169:0/1703392614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 msgr2=0x7fa248101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.347+0000 7fa24e28c700 1 --2- 172.21.15.169:0/1703392614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 0x7fa248101dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa238009990 tx=0x7fa238009ca0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 -- 172.21.15.169:0/1703392614 shutdown_connections 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 --2- 172.21.15.169:0/1703392614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa248103540 0x7fa248109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 --2- 172.21.15.169:0/1703392614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 0x7fa248101dc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 --2- 172.21.15.169:0/1703392614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa248102ba0 0x7fa248103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 -- 172.21.15.169:0/1703392614 >> 172.21.15.169:0/1703392614 conn(0x7fa2480fcf60 msgr2=0x7fa2480ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 -- 172.21.15.169:0/1703392614 shutdown_connections 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 -- 172.21.15.169:0/1703392614 wait complete. 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 Processor -- start 2024-04-26T01:35:49.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.348+0000 7fa24e28c700 1 -- start start 2024-04-26T01:35:49.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa24e28c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 0x7fa24806bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:49.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa24e28c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa248102ba0 0x7fa24819d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:49.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa24e28c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa248103540 0x7fa24819ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:49.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa24e28c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa24810d490 con 0x7fa248103540 2024-04-26T01:35:49.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa24e28c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa24810d310 con 0x7fa248102ba0 2024-04-26T01:35:49.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa24e28c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa24810d610 con 0x7fa2481019a0 2024-04-26T01:35:49.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa247fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 0x7fa24806bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:49.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa247fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 0x7fa24806bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49802/0 (socket says 172.21.15.169:49802) 2024-04-26T01:35:49.346 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:49 smithi169 ceph-mon[32851]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa247fff700 1 -- 172.21.15.169:0/2712485390 learned_addr learned my addr 172.21.15.169:0/2712485390 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa2477fe700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa248102ba0 0x7fa24819d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa24c829700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa248103540 0x7fa24819ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa247fff700 1 -- 172.21.15.169:0/2712485390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa248102ba0 msgr2=0x7fa24819d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa247fff700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa248102ba0 0x7fa24819d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa247fff700 1 -- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa248103540 msgr2=0x7fa24819ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa247fff700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa248103540 0x7fa24819ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.349+0000 7fa247fff700 1 -- 172.21.15.169:0/2712485390 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2481a0870 con 0x7fa2481019a0 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.350+0000 7fa247fff700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 0x7fa24806bd30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa2380036f0 tx=0x7fa238003b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.350+0000 7fa2457fa700 1 -- 172.21.15.169:0/2712485390 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238027070 con 0x7fa2481019a0 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.350+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2481a0b00 con 0x7fa2481019a0 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.350+0000 7fa2457fa700 1 -- 172.21.15.169:0/2712485390 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa23800bd90 con 0x7fa2481019a0 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.350+0000 7fa2457fa700 1 -- 172.21.15.169:0/2712485390 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23800f460 con 0x7fa2481019a0 2024-04-26T01:35:49.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.350+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2481a0fb0 con 0x7fa2481019a0 2024-04-26T01:35:49.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.351+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa248069730 con 0x7fa2481019a0 2024-04-26T01:35:49.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.352+0000 7fa2457fa700 1 -- 172.21.15.169:0/2712485390 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa23802a070 con 0x7fa2481019a0 2024-04-26T01:35:49.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.353+0000 7fa2457fa700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa2300a3c40 0x7fa2300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:49.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.353+0000 7fa2457fa700 1 -- 172.21.15.169:0/2712485390 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa2380d1110 con 0x7fa2481019a0 2024-04-26T01:35:49.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.353+0000 7fa2477fe700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa2300a3c40 0x7fa2300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:49.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.354+0000 7fa2477fe700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa2300a3c40 0x7fa2300a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa24806c290 tx=0x7fa23c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:49.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.358+0000 7fa2457fa700 1 -- 172.21.15.169:0/2712485390 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa23805eff0 con 0x7fa2481019a0 2024-04-26T01:35:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:49 smithi053 ceph-mon[32593]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:49.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:49 smithi028 ceph-mon[28247]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:49.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.565+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l", "target": ["mon-mgr", ""]}) -- 0x7fa248063140 con 0x7fa2300a3c40 2024-04-26T01:35:49.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.582+0000 7fa2457fa700 1 -- 172.21.15.169:0/2712485390 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fa248063140 con 0x7fa2300a3c40 2024-04-26T01:35:49.578 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mds.all update... 2024-04-26T01:35:49.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa2300a3c40 msgr2=0x7fa2300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:49.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa2300a3c40 0x7fa2300a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa24806c290 tx=0x7fa23c009190 comp rx=0 tx=0).stop 2024-04-26T01:35:49.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 msgr2=0x7fa24806bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:49.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 0x7fa24806bd30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa2380036f0 tx=0x7fa238003b50 comp rx=0 tx=0).stop 2024-04-26T01:35:49.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 shutdown_connections 2024-04-26T01:35:49.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa2300a3c40 0x7fa2300a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa248103540 0x7fa24819ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2481019a0 0x7fa24806bd30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.584+0000 7fa24e28c700 1 --2- 172.21.15.169:0/2712485390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa248102ba0 0x7fa24819d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:49.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.585+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 >> 172.21.15.169:0/2712485390 conn(0x7fa2480fcf60 msgr2=0x7fa248105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:49.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.585+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 shutdown_connections 2024-04-26T01:35:49.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-26T01:35:49.585+0000 7fa24e28c700 1 -- 172.21.15.169:0/2712485390 wait complete. 2024-04-26T01:35:50.210 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T01:35:50.219 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T01:35:50.266 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T01:35:50.311 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T01:35:50.356 DEBUG:teuthology.orchestra.run.smithi053:mds.b> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T01:35:50.359 DEBUG:teuthology.orchestra.run.smithi053:mds.e> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T01:35:50.403 DEBUG:teuthology.orchestra.run.smithi053:mds.h> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T01:35:50.445 DEBUG:teuthology.orchestra.run.smithi053:mds.k> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T01:35:50.489 DEBUG:teuthology.orchestra.run.smithi169:mds.c> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T01:35:50.492 DEBUG:teuthology.orchestra.run.smithi169:mds.f> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T01:35:50.534 DEBUG:teuthology.orchestra.run.smithi169:mds.i> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T01:35:50.577 DEBUG:teuthology.orchestra.run.smithi169:mds.l> sudo journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T01:35:50.620 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T01:35:50.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:35:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:50 smithi053 ceph-mon[32593]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:50 smithi053 ceph-mon[32593]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l;count:12 2024-04-26T01:35:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:50 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:50 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:50.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:50 smithi169 ceph-mon[32851]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:50.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:50 smithi169 ceph-mon[32851]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l;count:12 2024-04-26T01:35:50.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:50 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:50.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:50 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:50 smithi028 ceph-mon[28247]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:50 smithi028 ceph-mon[28247]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l;count:12 2024-04-26T01:35:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:50 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:51.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:50 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:51.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:51 smithi169 ceph-mon[32851]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:51.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:51.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:51.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:51.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:51.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:51.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:51.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:51 smithi053 ceph-mon[32593]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:51.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:51.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:51.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:51.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:51.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:51.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:52.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:51 smithi028 ceph-mon[28247]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:52.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:52.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:52.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:52.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:52.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:52.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:52.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:52 smithi028 ceph-mon[28247]: Deploying daemon mds.l on smithi169 2024-04-26T01:35:52.847 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:52 smithi169 ceph-mon[32851]: Deploying daemon mds.l on smithi169 2024-04-26T01:35:52.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:52 smithi053 ceph-mon[32593]: Deploying daemon mds.l on smithi169 2024-04-26T01:35:53.159 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 systemd[1]: Starting Ceph mds.l for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:35:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 -- 172.21.15.28:0/2231549804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c109c80 msgr2=0x7f1d4c10dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 --2- 172.21.15.28:0/2231549804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c109c80 0x7f1d4c10dcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d340099c0 tx=0x7f1d34009cd0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 -- 172.21.15.28:0/2231549804 shutdown_connections 2024-04-26T01:35:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 --2- 172.21.15.28:0/2231549804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d4c10e3b0 0x7f1d4c115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 --2- 172.21.15.28:0/2231549804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c1092d0 0x7f1d4c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 --2- 172.21.15.28:0/2231549804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c109c80 0x7f1d4c10dcd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 -- 172.21.15.28:0/2231549804 >> 172.21.15.28:0/2231549804 conn(0x7f1d4c076210 msgr2=0x7f1d4c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:53.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 -- 172.21.15.28:0/2231549804 shutdown_connections 2024-04-26T01:35:53.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.417+0000 7f1d53200700 1 -- 172.21.15.28:0/2231549804 wait complete. 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.419+0000 7f1d53200700 1 Processor -- start 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.419+0000 7f1d53200700 1 -- start start 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d53200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c1092d0 0x7f1d4c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d53200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d4c109c80 0x7f1d4c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d53200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c10e3b0 0x7f1d4c1a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d53200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c119290 con 0x7f1d4c1092d0 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d53200700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c119110 con 0x7f1d4c109c80 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d53200700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c119410 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d51200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c10e3b0 0x7f1d4c1a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d4be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c1092d0 0x7f1d4c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d51200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c10e3b0 0x7f1d4c1a3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:47934/0 (socket says 172.21.15.28:47934) 2024-04-26T01:35:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d51200700 1 -- 172.21.15.28:0/4291966198 learned_addr learned my addr 172.21.15.28:0/4291966198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:35:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d43400700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d4c109c80 0x7f1d4c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d51200700 1 -- 172.21.15.28:0/4291966198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d4c109c80 msgr2=0x7f1d4c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d51200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d4c109c80 0x7f1d4c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d51200700 1 -- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c1092d0 msgr2=0x7f1d4c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d51200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c1092d0 0x7f1d4c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.420+0000 7f1d51200700 1 -- 172.21.15.28:0/4291966198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d4c1a4730 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.421+0000 7f1d51200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c10e3b0 0x7f1d4c1a3ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400ef30 tx=0x7f1d4400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.421+0000 7f1d49600700 1 -- 172.21.15.28:0/4291966198 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d440092e0 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.421+0000 7f1d49600700 1 -- 172.21.15.28:0/4291966198 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d44004bb0 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.421+0000 7f1d49600700 1 -- 172.21.15.28:0/4291966198 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d44004020 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.421+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d4c1a49c0 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.421+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d4c1ac2a0 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.422+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d4c069730 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.425+0000 7f1d49600700 1 -- 172.21.15.28:0/4291966198 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1d4401b030 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.426+0000 7f1d49600700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d300a3cc0 0x7f1d300a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:53.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.426+0000 7f1d49600700 1 -- 172.21.15.28:0/4291966198 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1d440cf9b0 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.426+0000 7f1d4be00700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d300a3cc0 0x7f1d300a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.427+0000 7f1d4be00700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d300a3cc0 0x7f1d300a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009c60 tx=0x7f1d3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.430+0000 7f1d49600700 1 -- 172.21.15.28:0/4291966198 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1d44022020 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.523 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 podman[63124]: 2024-04-26 01:35:53.156759119 +0000 UTC m=+0.018042287 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:35:53.523 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 podman[63124]: 2024-04-26 01:35:53.283912503 +0000 UTC m=+0.145195714 container create 5dc46c50602caa0a5fab21da8be9c037ce7224623950ab1bd75018b71caddefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:35:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.717+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d4c02df80 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.717+0000 7f1d49600700 1 -- 172.21.15.28:0/4291966198 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f1d44010040 con 0x7f1d4c10e3b0 2024-04-26T01:35:53.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:53 smithi028 ceph-mon[28247]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:53.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:53 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6832/451902326,v1:172.21.15.169:6833/451902326] up:boot 2024-04-26T01:35:53.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:53 smithi028 ceph-mon[28247]: fsmap 1 up:standby 2024-04-26T01:35:53.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:53 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:35:53.714 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:35:53.714 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T01:35:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.721+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d300a3cc0 msgr2=0x7f1d300a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.721+0000 7f1d53200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d300a3cc0 0x7f1d300a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009c60 tx=0x7f1d3c009260 comp rx=0 tx=0).stop 2024-04-26T01:35:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.721+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c10e3b0 msgr2=0x7f1d4c1a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.721+0000 7f1d53200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c10e3b0 0x7f1d4c1a3ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400ef30 tx=0x7f1d4400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.722+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 shutdown_connections 2024-04-26T01:35:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.722+0000 7f1d53200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1d300a3cc0 0x7f1d300a6180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.722+0000 7f1d53200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d4c10e3b0 0x7f1d4c1a3ff0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.722+0000 7f1d53200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c1092d0 0x7f1d4c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.722+0000 7f1d53200700 1 --2- 172.21.15.28:0/4291966198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d4c109c80 0x7f1d4c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.722+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 >> 172.21.15.28:0/4291966198 conn(0x7f1d4c076210 msgr2=0x7f1d4c076830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.722+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 shutdown_connections 2024-04-26T01:35:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:53.722+0000 7f1d53200700 1 -- 172.21.15.28:0/4291966198 wait complete. 2024-04-26T01:35:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-04-26T01:35:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:53 smithi169 ceph-mon[32851]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:53 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6832/451902326,v1:172.21.15.169:6833/451902326] up:boot 2024-04-26T01:35:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:53 smithi169 ceph-mon[32851]: fsmap 1 up:standby 2024-04-26T01:35:53.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:53 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:35:53.909 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 podman[63124]: 2024-04-26 01:35:53.528581752 +0000 UTC m=+0.389864908 container init 5dc46c50602caa0a5fab21da8be9c037ce7224623950ab1bd75018b71caddefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T01:35:53.909 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 podman[63124]: 2024-04-26 01:35:53.531482386 +0000 UTC m=+0.392765555 container start 5dc46c50602caa0a5fab21da8be9c037ce7224623950ab1bd75018b71caddefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T01:35:53.910 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 ceph-mds[63143]: -- 172.21.15.169:0/4240526312 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556559b11a40 con 0x556559ae4c00 2024-04-26T01:35:53.910 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l[63139]: starting mds.l at 2024-04-26T01:35:53.910 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 bash[63124]: 5dc46c50602caa0a5fab21da8be9c037ce7224623950ab1bd75018b71caddefc 2024-04-26T01:35:53.910 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 01:35:53 smithi169 systemd[1]: Started Ceph mds.l for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:35:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:53 smithi053 ceph-mon[32593]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:53 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6832/451902326,v1:172.21.15.169:6833/451902326] up:boot 2024-04-26T01:35:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:53 smithi053 ceph-mon[32593]: fsmap 1 up:standby 2024-04-26T01:35:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:53 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/4291966198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: Deploying daemon mds.k on smithi053 2024-04-26T01:35:54.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:54 smithi028 ceph-mon[28247]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:54.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:54.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/4291966198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:35:54.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:54.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:54.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: Deploying daemon mds.k on smithi053 2024-04-26T01:35:54.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:54 smithi053 ceph-mon[32593]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:55.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:55.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:55.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:55.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:55.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/4291966198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:35:55.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:55.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:55.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: Deploying daemon mds.k on smithi053 2024-04-26T01:35:55.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:54 smithi169 ceph-mon[32851]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:56.217 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 systemd[1]: Starting Ceph mds.k for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:35:56.493 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 podman[61700]: 2024-04-26 01:35:56.125834038 +0000 UTC m=+0.018078789 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:35:56.493 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 podman[61700]: 2024-04-26 01:35:56.253821075 +0000 UTC m=+0.146065819 container create 7498e416a85af0e8868acabd2f2675d2ea35014c7ff6f6e57a9fd624685f1dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-26T01:35:56.493 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 podman[61700]: 2024-04-26 01:35:56.498159438 +0000 UTC m=+0.390404180 container init 7498e416a85af0e8868acabd2f2675d2ea35014c7ff6f6e57a9fd624685f1dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T01:35:56.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:35:56.784 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 podman[61700]: 2024-04-26 01:35:56.501013925 +0000 UTC m=+0.393258679 container start 7498e416a85af0e8868acabd2f2675d2ea35014c7ff6f6e57a9fd624685f1dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:35:56.784 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-mds[61719]: -- 172.21.15.53:0/2785329050 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f494cf3a40 con 0x55f494cc6800 2024-04-26T01:35:56.784 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k[61715]: starting mds.k at 2024-04-26T01:35:56.784 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 bash[61700]: 7498e416a85af0e8868acabd2f2675d2ea35014c7ff6f6e57a9fd624685f1dfa 2024-04-26T01:35:56.784 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 01:35:56 smithi053 systemd[1]: Started Ceph mds.k for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:35:57.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:56 smithi028 ceph-mon[28247]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:57.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:56 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:56 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:56 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:56 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:56 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:56 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:57.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:56 smithi169 ceph-mon[32851]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:57.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:56 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:56 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:56 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:56 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:57.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:56 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:57.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:56 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-mon[32593]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:35:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:35:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:35:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:56 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:57.900 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:57 smithi053 ceph-mon[32593]: Deploying daemon mds.j on smithi028 2024-04-26T01:35:57.900 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:57 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.53:6832/332324514,v1:172.21.15.53:6833/332324514] up:boot 2024-04-26T01:35:57.900 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:57 smithi053 ceph-mon[32593]: fsmap 2 up:standby 2024-04-26T01:35:57.900 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:57 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:35:58.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:57 smithi028 ceph-mon[28247]: Deploying daemon mds.j on smithi028 2024-04-26T01:35:58.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:57 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.53:6832/332324514,v1:172.21.15.53:6833/332324514] up:boot 2024-04-26T01:35:58.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:57 smithi028 ceph-mon[28247]: fsmap 2 up:standby 2024-04-26T01:35:58.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:57 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:35:58.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:57 smithi169 ceph-mon[32851]: Deploying daemon mds.j on smithi028 2024-04-26T01:35:58.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:57 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.53:6832/332324514,v1:172.21.15.53:6833/332324514] up:boot 2024-04-26T01:35:58.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:57 smithi169 ceph-mon[32851]: fsmap 2 up:standby 2024-04-26T01:35:58.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:57 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:35:58.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:35:58 smithi028 ceph-mon[28247]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:59.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:35:58 smithi169 ceph-mon[32851]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:59.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:35:58 smithi053 ceph-mon[32593]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:35:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 -- 172.21.15.28:0/1478347094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940102300 msgr2=0x7f294010e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 --2- 172.21.15.28:0/1478347094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940102300 0x7f294010e7f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f293800b0a0 tx=0x7f293800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:35:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 -- 172.21.15.28:0/1478347094 shutdown_connections 2024-04-26T01:35:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 --2- 172.21.15.28:0/1478347094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940102300 0x7f294010e7f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 --2- 172.21.15.28:0/1478347094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2940107990 0x7f2940107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 --2- 172.21.15.28:0/1478347094 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2940101940 0x7f2940101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 -- 172.21.15.28:0/1478347094 >> 172.21.15.28:0/1478347094 conn(0x7f29400788f0 msgr2=0x7f2940078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 -- 172.21.15.28:0/1478347094 shutdown_connections 2024-04-26T01:35:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.953+0000 7f2945000700 1 -- 172.21.15.28:0/1478347094 wait complete. 2024-04-26T01:35:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.954+0000 7f2945000700 1 Processor -- start 2024-04-26T01:35:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.954+0000 7f2945000700 1 -- start start 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f2945000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940101940 0x7f294019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f2945000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2940102300 0x7f294019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f2945000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2940107990 0x7f294019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f2945000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2940111b40 con 0x7f2940101940 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f2945000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f29401119c0 con 0x7f2940102300 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f2945000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2940111cc0 con 0x7f2940107990 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f293de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940101940 0x7f294019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f293de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940101940 0x7f294019b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34944/0 (socket says 172.21.15.28:34944) 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f293de00700 1 -- 172.21.15.28:0/23933947 learned_addr learned my addr 172.21.15.28:0/23933947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f293d400700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2940102300 0x7f294019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.955+0000 7f293e800700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2940107990 0x7f294019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f293de00700 1 -- 172.21.15.28:0/23933947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2940107990 msgr2=0x7f294019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f293de00700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2940107990 0x7f294019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f293de00700 1 -- 172.21.15.28:0/23933947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2940102300 msgr2=0x7f294019b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:59.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f293de00700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2940102300 0x7f294019b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:59.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f293de00700 1 -- 172.21.15.28:0/23933947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29401a0300 con 0x7f2940101940 2024-04-26T01:35:59.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f293de00700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940101940 0x7f294019b2f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f293000e9a0 tx=0x7f293000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:59.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f2936a00700 1 -- 172.21.15.28:0/23933947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293000ca10 con 0x7f2940101940 2024-04-26T01:35:59.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f2936a00700 1 -- 172.21.15.28:0/23933947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f293000cbb0 con 0x7f2940101940 2024-04-26T01:35:59.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.956+0000 7f2936a00700 1 -- 172.21.15.28:0/23933947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930018680 con 0x7f2940101940 2024-04-26T01:35:59.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.957+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29401a05f0 con 0x7f2940101940 2024-04-26T01:35:59.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.957+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29401a7e70 con 0x7f2940101940 2024-04-26T01:35:59.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.958+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2940103060 con 0x7f2940101940 2024-04-26T01:35:59.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.961+0000 7f2936a00700 1 -- 172.21.15.28:0/23933947 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2930010430 con 0x7f2940101940 2024-04-26T01:35:59.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.962+0000 7f2936a00700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f29240a3c40 0x7f29240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:59.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.962+0000 7f2936a00700 1 -- 172.21.15.28:0/23933947 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29300d0ba0 con 0x7f2940101940 2024-04-26T01:35:59.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.962+0000 7f293d400700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f29240a3c40 0x7f29240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.963+0000 7f293d400700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f29240a3c40 0x7f29240a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f29401018f0 tx=0x7f292c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:35:59.965+0000 7f2936a00700 1 -- 172.21.15.28:0/23933947 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2930026070 con 0x7f2940101940 2024-04-26T01:36:00.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.248+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2940069730 con 0x7f2940101940 2024-04-26T01:36:00.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.249+0000 7f2936a00700 1 -- 172.21.15.28:0/23933947 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2215 (secure 0 0 0) 0x7f293009d980 con 0x7f2940101940 2024-04-26T01:36:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:36:00.246 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T01:36:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f29240a3c40 msgr2=0x7f29240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f29240a3c40 0x7f29240a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f29401018f0 tx=0x7f292c009260 comp rx=0 tx=0).stop 2024-04-26T01:36:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940101940 msgr2=0x7f294019b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940101940 0x7f294019b2f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f293000e9a0 tx=0x7f293000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:36:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 shutdown_connections 2024-04-26T01:36:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f29240a3c40 0x7f29240a6100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2940107990 0x7f294019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2940101940 0x7f294019b2f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 --2- 172.21.15.28:0/23933947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2940102300 0x7f294019b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 >> 172.21.15.28:0/23933947 conn(0x7f29400788f0 msgr2=0x7f294006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 shutdown_connections 2024-04-26T01:36:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:00.253+0000 7f2945000700 1 -- 172.21.15.28:0/23933947 wait complete. 2024-04-26T01:36:00.251 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-04-26T01:36:01.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:00 smithi028 ceph-mon[28247]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:01.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:00 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/23933947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:01.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:00 smithi169 ceph-mon[32851]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:01.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:00 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/23933947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:01.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:00 smithi053 ceph-mon[32593]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:01.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:00 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/23933947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:02.031 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:01 smithi028 systemd[1]: Starting Ceph mds.j for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:36:02.394 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:02 smithi028 podman[66510]: 2024-04-26 01:36:01.979678786 +0000 UTC m=+0.037566272 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:36:02.394 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:02 smithi028 podman[66510]: 2024-04-26 01:36:02.101469247 +0000 UTC m=+0.159356763 container create 2492a7c9273af25763b1028870af812524a614b776c14e623bfd895ab0f4e742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T01:36:02.394 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:02 smithi028 podman[66510]: 2024-04-26 01:36:02.39709317 +0000 UTC m=+0.454980693 container init 2492a7c9273af25763b1028870af812524a614b776c14e623bfd895ab0f4e742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T01:36:02.781 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:02 smithi028 podman[66510]: 2024-04-26 01:36:02.40872147 +0000 UTC m=+0.466608964 container start 2492a7c9273af25763b1028870af812524a614b776c14e623bfd895ab0f4e742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T01:36:02.781 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-mds[66530]: -- 172.21.15.28:0/948684457 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558f3bf79a40 con 0x558f3bf4d000 2024-04-26T01:36:02.781 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j[66526]: starting mds.j at 2024-04-26T01:36:02.781 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:02 smithi028 bash[66510]: 2492a7c9273af25763b1028870af812524a614b776c14e623bfd895ab0f4e742 2024-04-26T01:36:02.781 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 01:36:02 smithi028 systemd[1]: Started Ceph mds.j for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:03.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:02 smithi169 ceph-mon[32851]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:03.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:02 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:02 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:02 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:02 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:03.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:02 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:03.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:02 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:02 smithi053 ceph-mon[32593]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:02 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:02 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:02 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:02 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:03.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:02 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:03.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:02 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:03.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-mon[28247]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:03.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:03.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:03.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:03.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:02 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:03.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:36:04.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:03 smithi028 ceph-mon[28247]: Deploying daemon mds.d on smithi028 2024-04-26T01:36:04.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:03 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.28:6834/2607969943,v1:172.21.15.28:6835/2607969943] up:boot 2024-04-26T01:36:04.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:03 smithi028 ceph-mon[28247]: fsmap 3 up:standby 2024-04-26T01:36:04.130 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:03 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:36:04.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:03 smithi169 ceph-mon[32851]: Deploying daemon mds.d on smithi028 2024-04-26T01:36:04.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:03 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.28:6834/2607969943,v1:172.21.15.28:6835/2607969943] up:boot 2024-04-26T01:36:04.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:03 smithi169 ceph-mon[32851]: fsmap 3 up:standby 2024-04-26T01:36:04.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:03 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:36:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:03 smithi053 ceph-mon[32593]: Deploying daemon mds.d on smithi028 2024-04-26T01:36:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:03 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.28:6834/2607969943,v1:172.21.15.28:6835/2607969943] up:boot 2024-04-26T01:36:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:03 smithi053 ceph-mon[32593]: fsmap 3 up:standby 2024-04-26T01:36:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:03 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:36:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:05.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:04 smithi028 ceph-mon[28247]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:05.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:04 smithi169 ceph-mon[32851]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:05.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:04 smithi053 ceph-mon[32593]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:06.032 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:05 smithi028 systemd[1]: Starting Ceph mds.d for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:36:07.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:06 smithi028 ceph-mon[28247]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:07.024 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:06 smithi028 podman[66911]: 2024-04-26 01:36:06.605380269 +0000 UTC m=+0.039884409 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:36:07.025 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:06 smithi028 podman[66911]: 2024-04-26 01:36:06.882118431 +0000 UTC m=+0.316622627 container create 83f859f873459a737475812aa86d9743bc2251216fa4678613cc5b7af5a1d076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T01:36:07.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:06 smithi169 ceph-mon[32851]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:06 smithi053 ceph-mon[32593]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.398+0000 7f3207e00700 1 -- 172.21.15.28:0/1677578553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f320006a750 msgr2=0x7f320006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.398+0000 7f3207e00700 1 --2- 172.21.15.28:0/1677578553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f320006a750 0x7f320006abb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009990 tx=0x7f31ec009ca0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.399+0000 7f3207e00700 1 -- 172.21.15.28:0/1677578553 shutdown_connections 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.399+0000 7f3207e00700 1 --2- 172.21.15.28:0/1677578553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f320006b0f0 0x7f320010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.399+0000 7f3207e00700 1 --2- 172.21.15.28:0/1677578553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3200108a30 0x7f3200108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.399+0000 7f3207e00700 1 --2- 172.21.15.28:0/1677578553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f320006a750 0x7f320006abb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.399+0000 7f3207e00700 1 -- 172.21.15.28:0/1677578553 >> 172.21.15.28:0/1677578553 conn(0x7f32000788f0 msgr2=0x7f3200078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.399+0000 7f3207e00700 1 -- 172.21.15.28:0/1677578553 shutdown_connections 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.399+0000 7f3207e00700 1 -- 172.21.15.28:0/1677578553 wait complete. 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.400+0000 7f3207e00700 1 Processor -- start 2024-04-26T01:36:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.400+0000 7f3207e00700 1 -- start start 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.400+0000 7f3207e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f320006a750 0x7f320019ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.400+0000 7f3207e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f320006b0f0 0x7f32001a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.400+0000 7f3207e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3200108a30 0x7f320019b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.400+0000 7f3207e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3200101f10 con 0x7f320006a750 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.400+0000 7f3207e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3200101d90 con 0x7f3200108a30 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.400+0000 7f3207e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3200102090 con 0x7f320006b0f0 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f320006a750 0x7f320019ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3200108a30 0x7f320019b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3200108a30 0x7f320019b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39252/0 (socket says 172.21.15.28:39252) 2024-04-26T01:36:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 -- 172.21.15.28:0/1150690417 learned_addr learned my addr 172.21.15.28:0/1150690417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3204a00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f320006b0f0 0x7f32001a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 -- 172.21.15.28:0/1150690417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f320006b0f0 msgr2=0x7f32001a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f320006b0f0 0x7f32001a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 -- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f320006a750 msgr2=0x7f320019ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f320006a750 0x7f320019ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 -- 172.21.15.28:0/1150690417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320019bfb0 con 0x7f3200108a30 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.401+0000 7f3205e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3200108a30 0x7f320019b770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f31f800c370 tx=0x7f31f800c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.402+0000 7f31fe000700 1 -- 172.21.15.28:0/1150690417 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f80092e0 con 0x7f3200108a30 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.402+0000 7f31fe000700 1 -- 172.21.15.28:0/1150690417 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31f8003a00 con 0x7f3200108a30 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.402+0000 7f31fe000700 1 -- 172.21.15.28:0/1150690417 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f80046f0 con 0x7f3200108a30 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.402+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f320019c240 con 0x7f3200108a30 2024-04-26T01:36:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.402+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32001a7e60 con 0x7f3200108a30 2024-04-26T01:36:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.403+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3200104170 con 0x7f3200108a30 2024-04-26T01:36:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.405+0000 7f31fe000700 1 -- 172.21.15.28:0/1150690417 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f31f801a020 con 0x7f3200108a30 2024-04-26T01:36:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.406+0000 7f31fe000700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f31e80a3c40 0x7f31e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.407+0000 7f31fe000700 1 -- 172.21.15.28:0/1150690417 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f31f80cf870 con 0x7f3200108a30 2024-04-26T01:36:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.407+0000 7f3205400700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f31e80a3c40 0x7f31e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.407+0000 7f3205400700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f31e80a3c40 0x7f31e80a6100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f31f4009be0 tx=0x7f31f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.411+0000 7f31fe000700 1 -- 172.21.15.28:0/1150690417 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31f8010040 con 0x7f3200108a30 2024-04-26T01:36:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.691+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32001a8060 con 0x7f3200108a30 2024-04-26T01:36:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.692+0000 7f31fe000700 1 -- 172.21.15.28:0/1150690417 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3040 (secure 0 0 0) 0x7f31f809c650 con 0x7f3200108a30 2024-04-26T01:36:07.688 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:36:07.688 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T01:36:07.688 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:07 smithi028 podman[66911]: 2024-04-26 01:36:07.463838079 +0000 UTC m=+0.898342249 container init 83f859f873459a737475812aa86d9743bc2251216fa4678613cc5b7af5a1d076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T01:36:07.688 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:07 smithi028 podman[66911]: 2024-04-26 01:36:07.472031623 +0000 UTC m=+0.906535771 container start 83f859f873459a737475812aa86d9743bc2251216fa4678613cc5b7af5a1d076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-04-26T01:36:07.688 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:07 smithi028 ceph-mds[66977]: -- 172.21.15.28:0/4208943081 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f9fdda5a40 con 0x55f9fdd78c00 2024-04-26T01:36:07.688 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:07 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d[66943]: starting mds.d at 2024-04-26T01:36:07.689 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:07 smithi028 bash[66911]: 83f859f873459a737475812aa86d9743bc2251216fa4678613cc5b7af5a1d076 2024-04-26T01:36:07.689 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 01:36:07 smithi028 systemd[1]: Started Ceph mds.d for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.695+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f31e80a3c40 msgr2=0x7f31e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.695+0000 7f3207e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f31e80a3c40 0x7f31e80a6100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f31f4009be0 tx=0x7f31f4009260 comp rx=0 tx=0).stop 2024-04-26T01:36:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.695+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3200108a30 msgr2=0x7f320019b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.695+0000 7f3207e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3200108a30 0x7f320019b770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f31f800c370 tx=0x7f31f800c680 comp rx=0 tx=0).stop 2024-04-26T01:36:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.696+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 shutdown_connections 2024-04-26T01:36:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.696+0000 7f3207e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f31e80a3c40 0x7f31e80a6100 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.696+0000 7f3207e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3200108a30 0x7f320019b770 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.696+0000 7f3207e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f320006a750 0x7f320019ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.696+0000 7f3207e00700 1 --2- 172.21.15.28:0/1150690417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f320006b0f0 0x7f32001a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.696+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 >> 172.21.15.28:0/1150690417 conn(0x7f32000788f0 msgr2=0x7f3200104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.696+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 shutdown_connections 2024-04-26T01:36:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:07.696+0000 7f3207e00700 1 -- 172.21.15.28:0/1150690417 wait complete. 2024-04-26T01:36:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-04-26T01:36:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:08.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:08.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:08.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:08.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:08.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:08.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: Deploying daemon mds.e on smithi053 2024-04-26T01:36:08.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1150690417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:08.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:08.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.28:6836/447949088,v1:172.21.15.28:6837/447949088] up:boot 2024-04-26T01:36:08.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: fsmap 4 up:standby 2024-04-26T01:36:08.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:08 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:36:08.932 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:08.932 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:08.932 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:08.932 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:08.933 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:08.933 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:08.933 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: Deploying daemon mds.e on smithi053 2024-04-26T01:36:08.933 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1150690417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:08.933 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:08.933 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.28:6836/447949088,v1:172.21.15.28:6837/447949088] up:boot 2024-04-26T01:36:08.934 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: fsmap 4 up:standby 2024-04-26T01:36:08.934 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:08 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: Deploying daemon mds.e on smithi053 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1150690417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:09.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:09.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.28:6836/447949088,v1:172.21.15.28:6837/447949088] up:boot 2024-04-26T01:36:09.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: fsmap 4 up:standby 2024-04-26T01:36:09.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:08 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:36:09.967 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:09 smithi053 systemd[1]: Starting Ceph mds.e for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:36:10.399 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:10 smithi053 podman[62024]: 2024-04-26 01:36:09.941517593 +0000 UTC m=+0.018475380 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:36:10.399 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:10 smithi053 podman[62024]: 2024-04-26 01:36:10.042338871 +0000 UTC m=+0.119296682 container create 998214597ce2f89b16d390535398e8b7defa799ca2111d2174aa95b9272bd75c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T01:36:10.399 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:10 smithi053 podman[62024]: 2024-04-26 01:36:10.403519119 +0000 UTC m=+0.480476885 container init 998214597ce2f89b16d390535398e8b7defa799ca2111d2174aa95b9272bd75c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True) 2024-04-26T01:36:10.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:36:10.717 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:10 smithi053 podman[62024]: 2024-04-26 01:36:10.406461918 +0000 UTC m=+0.483419676 container start 998214597ce2f89b16d390535398e8b7defa799ca2111d2174aa95b9272bd75c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:36:10.717 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:10 smithi053 ceph-mds[62043]: -- 172.21.15.53:0/3910185962 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f66af2ba40 con 0x55f66aefe800 2024-04-26T01:36:10.718 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e[62039]: starting mds.e at 2024-04-26T01:36:10.718 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:10 smithi053 bash[62024]: 998214597ce2f89b16d390535398e8b7defa799ca2111d2174aa95b9272bd75c 2024-04-26T01:36:10.718 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 01:36:10 smithi053 systemd[1]: Started Ceph mds.e for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:10.978 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.53:6834/2175861856,v1:172.21.15.53:6835/2175861856] up:boot 2024-04-26T01:36:11.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: fsmap 5 up:standby 2024-04-26T01:36:11.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:11 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:36:11.804 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:11.804 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:11.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:11.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:11.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.53:6834/2175861856,v1:172.21.15.53:6835/2175861856] up:boot 2024-04-26T01:36:11.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: fsmap 5 up:standby 2024-04-26T01:36:11.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:11 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:36:11.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:11.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:11.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:11.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:11.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:11.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.53:6834/2175861856,v1:172.21.15.53:6835/2175861856] up:boot 2024-04-26T01:36:11.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: fsmap 5 up:standby 2024-04-26T01:36:11.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:11 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:36:12.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:12 smithi028 ceph-mon[28247]: Deploying daemon mds.f on smithi169 2024-04-26T01:36:12.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:12 smithi169 ceph-mon[32851]: Deploying daemon mds.f on smithi169 2024-04-26T01:36:12.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:12 smithi053 ceph-mon[32593]: Deploying daemon mds.f on smithi169 2024-04-26T01:36:13.159 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:36:13 smithi169 podman[63446]: 2024-04-26 01:36:13.113901352 +0000 UTC m=+0.134996674 container create a2f9a35ac466d75335266c47bc8b24f8712d0462cabf409077c8d76462ad6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:36:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 -- 172.21.15.28:0/1113991375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0101940 msgr2=0x7f1fd0101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 --2- 172.21.15.28:0/1113991375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0101940 0x7f1fd0101dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0098f0 tx=0x7f1fbc009c00 comp rx=0 tx=0).stop 2024-04-26T01:36:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 -- 172.21.15.28:0/1113991375 shutdown_connections 2024-04-26T01:36:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 --2- 172.21.15.28:0/1113991375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1fd0102300 0x7f1fd010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 --2- 172.21.15.28:0/1113991375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd0107990 0x7f1fd0107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 --2- 172.21.15.28:0/1113991375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0101940 0x7f1fd0101dc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 -- 172.21.15.28:0/1113991375 >> 172.21.15.28:0/1113991375 conn(0x7f1fd00788f0 msgr2=0x7f1fd0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 -- 172.21.15.28:0/1113991375 shutdown_connections 2024-04-26T01:36:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.180+0000 7f1fd8400700 1 -- 172.21.15.28:0/1113991375 wait complete. 2024-04-26T01:36:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.181+0000 7f1fd8400700 1 Processor -- start 2024-04-26T01:36:13.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.182+0000 7f1fd8400700 1 -- start start 2024-04-26T01:36:13.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.182+0000 7f1fd8400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd0101940 0x7f1fd019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.182+0000 7f1fd8400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0102300 0x7f1fd019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.182+0000 7f1fd8400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1fd0107990 0x7f1fd019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.182+0000 7f1fd8400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd0110a50 con 0x7f1fd0101940 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.182+0000 7f1fd8400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd01108d0 con 0x7f1fd0102300 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.182+0000 7f1fd8400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd0110bd0 con 0x7f1fd0107990 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd0101940 0x7f1fd019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0102300 0x7f1fd019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0102300 0x7f1fd019b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38112/0 (socket says 172.21.15.28:38112) 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 -- 172.21.15.28:0/3211096333 learned_addr learned my addr 172.21.15.28:0/3211096333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd6400700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1fd0107990 0x7f1fd019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 -- 172.21.15.28:0/3211096333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1fd0107990 msgr2=0x7f1fd019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1fd0107990 0x7f1fd019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 -- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd0101940 msgr2=0x7f1fd019b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd0101940 0x7f1fd019b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 -- 172.21.15.28:0/3211096333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd01a0380 con 0x7f1fd0102300 2024-04-26T01:36:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.183+0000 7f1fd5000700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0102300 0x7f1fd019b8b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009230 tx=0x7f1fbc00fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.184+0000 7f1fce000700 1 -- 172.21.15.28:0/3211096333 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc0050c0 con 0x7f1fd0102300 2024-04-26T01:36:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.184+0000 7f1fce000700 1 -- 172.21.15.28:0/3211096333 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fbc00b560 con 0x7f1fd0102300 2024-04-26T01:36:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.184+0000 7f1fce000700 1 -- 172.21.15.28:0/3211096333 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc003a00 con 0x7f1fd0102300 2024-04-26T01:36:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.184+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd01a0610 con 0x7f1fd0102300 2024-04-26T01:36:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.184+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd01a7ef0 con 0x7f1fd0102300 2024-04-26T01:36:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.185+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd0103060 con 0x7f1fd0102300 2024-04-26T01:36:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.187+0000 7f1fce000700 1 -- 172.21.15.28:0/3211096333 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1fbc00b710 con 0x7f1fd0102300 2024-04-26T01:36:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.189+0000 7f1fce000700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1fb80a3ca0 0x7f1fb80a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.189+0000 7f1fce000700 1 -- 172.21.15.28:0/3211096333 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1fbc0d0e90 con 0x7f1fd0102300 2024-04-26T01:36:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.189+0000 7f1fd5a00700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1fb80a3ca0 0x7f1fb80a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.190+0000 7f1fd5a00700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1fb80a3ca0 0x7f1fb80a6160 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009be0 tx=0x7f1fc4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.191+0000 7f1fce000700 1 -- 172.21.15.28:0/3211096333 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1fbc013030 con 0x7f1fd0102300 2024-04-26T01:36:13.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.473+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fd019c6d0 con 0x7f1fd0102300 2024-04-26T01:36:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.477+0000 7f1fce000700 1 -- 172.21.15.28:0/3211096333 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4687 (secure 0 0 0) 0x7f1fbc022e50 con 0x7f1fd0102300 2024-04-26T01:36:13.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:36:13.474 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T01:36:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1fb80a3ca0 msgr2=0x7f1fb80a6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1fb80a3ca0 0x7f1fb80a6160 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009be0 tx=0x7f1fc4009260 comp rx=0 tx=0).stop 2024-04-26T01:36:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0102300 msgr2=0x7f1fd019b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0102300 0x7f1fd019b8b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009230 tx=0x7f1fbc00fa70 comp rx=0 tx=0).stop 2024-04-26T01:36:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 shutdown_connections 2024-04-26T01:36:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1fb80a3ca0 0x7f1fb80a6160 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1fd0107990 0x7f1fd019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd0101940 0x7f1fd019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 --2- 172.21.15.28:0/3211096333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fd0102300 0x7f1fd019b8b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:13.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.480+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 >> 172.21.15.28:0/3211096333 conn(0x7f1fd00788f0 msgr2=0x7f1fd00ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:13.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.481+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 shutdown_connections 2024-04-26T01:36:13.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:13.481+0000 7f1fd8400700 1 -- 172.21.15.28:0/3211096333 wait complete. 2024-04-26T01:36:13.478 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-04-26T01:36:13.570 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:36:13 smithi169 podman[63446]: 2024-04-26 01:36:13.350515352 +0000 UTC m=+0.371610682 container init a2f9a35ac466d75335266c47bc8b24f8712d0462cabf409077c8d76462ad6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True) 2024-04-26T01:36:13.570 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:36:13 smithi169 podman[63446]: 2024-04-26 01:36:13.353655893 +0000 UTC m=+0.374751225 container start a2f9a35ac466d75335266c47bc8b24f8712d0462cabf409077c8d76462ad6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T01:36:13.570 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mds[63465]: -- 172.21.15.169:0/1536010125 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e8bb079a40 con 0x55e8bb04cc00 2024-04-26T01:36:13.571 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f[63461]: starting mds.f at 2024-04-26T01:36:13.571 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:36:13 smithi169 bash[63446]: a2f9a35ac466d75335266c47bc8b24f8712d0462cabf409077c8d76462ad6a6c 2024-04-26T01:36:13.571 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:36:13 smithi169 systemd[1]: Started Ceph mds.f for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:13 smithi028 ceph-mon[28247]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:13 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3211096333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:13 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:13.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mon[32851]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:13.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3211096333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:13.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:13.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:13.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:13 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:13 smithi053 ceph-mon[32593]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:13 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3211096333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:13.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:13 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:14.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:14 smithi028 ceph-mon[28247]: Deploying daemon mds.c on smithi169 2024-04-26T01:36:14.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:14 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6834/1802608598,v1:172.21.15.169:6835/1802608598] up:boot 2024-04-26T01:36:14.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:14 smithi028 ceph-mon[28247]: fsmap 6 up:standby 2024-04-26T01:36:14.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:14 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:36:14.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:14 smithi169 ceph-mon[32851]: Deploying daemon mds.c on smithi169 2024-04-26T01:36:14.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:14 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6834/1802608598,v1:172.21.15.169:6835/1802608598] up:boot 2024-04-26T01:36:14.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:14 smithi169 ceph-mon[32851]: fsmap 6 up:standby 2024-04-26T01:36:14.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:14 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:36:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:14 smithi053 ceph-mon[32593]: Deploying daemon mds.c on smithi169 2024-04-26T01:36:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:14 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6834/1802608598,v1:172.21.15.169:6835/1802608598] up:boot 2024-04-26T01:36:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:14 smithi053 ceph-mon[32593]: fsmap 6 up:standby 2024-04-26T01:36:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:14 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:36:15.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:15 smithi169 ceph-mon[32851]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:15.882 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:15 smithi169 systemd[1]: Starting Ceph mds.c for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:36:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:15 smithi053 ceph-mon[32593]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:16.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:15 smithi028 ceph-mon[28247]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:16.159 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:15 smithi169 podman[63755]: 2024-04-26 01:36:15.789713558 +0000 UTC m=+0.018022974 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:36:16.159 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:15 smithi169 podman[63755]: 2024-04-26 01:36:15.92119209 +0000 UTC m=+0.149501553 container create 813d4dceafefe51e333d6517c5ac0bc542f7ea0f0125856ce819b3a029a83f3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T01:36:16.159 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:16 smithi169 podman[63755]: 2024-04-26 01:36:16.164126398 +0000 UTC m=+0.392435816 container init 813d4dceafefe51e333d6517c5ac0bc542f7ea0f0125856ce819b3a029a83f3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:36:16.409 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:16 smithi169 podman[63755]: 2024-04-26 01:36:16.166811559 +0000 UTC m=+0.395120984 container start 813d4dceafefe51e333d6517c5ac0bc542f7ea0f0125856ce819b3a029a83f3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:36:16.409 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:16 smithi169 ceph-mds[63774]: -- 172.21.15.169:0/4278670425 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635ca90da40 con 0x5635ca8e0800 2024-04-26T01:36:16.409 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c[63770]: starting mds.c at 2024-04-26T01:36:16.409 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:16 smithi169 bash[63755]: 813d4dceafefe51e333d6517c5ac0bc542f7ea0f0125856ce819b3a029a83f3a 2024-04-26T01:36:16.409 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:36:16 smithi169 systemd[1]: Started Ceph mds.c for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:16.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:36:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:17.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:17.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:17.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:17.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:17.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: Deploying daemon mds.h on smithi053 2024-04-26T01:36:17.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] up:boot 2024-04-26T01:36:17.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: fsmap 7 up:standby 2024-04-26T01:36:17.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:17 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:36:17.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:17.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:17.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:17.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:17.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: Deploying daemon mds.h on smithi053 2024-04-26T01:36:17.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] up:boot 2024-04-26T01:36:17.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: fsmap 7 up:standby 2024-04-26T01:36:17.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:17 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:36:17.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:17.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:17.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: Deploying daemon mds.h on smithi053 2024-04-26T01:36:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] up:boot 2024-04-26T01:36:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: fsmap 7 up:standby 2024-04-26T01:36:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:17 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:36:18.941 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:18 smithi053 systemd[1]: Starting Ceph mds.h for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:36:19.217 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:18 smithi053 podman[62339]: 2024-04-26 01:36:18.849398057 +0000 UTC m=+0.018841017 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:36:19.218 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:18 smithi053 podman[62339]: 2024-04-26 01:36:18.966314055 +0000 UTC m=+0.135757006 container create 57bcc154dc479d8ce882e006502be1226f040bf840187c0198afcccfc8e5fe6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 -- 172.21.15.28:0/3597481049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f2410f030 msgr2=0x7f8f2410f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 --2- 172.21.15.28:0/3597481049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f2410f030 0x7f8f2410f410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009990 tx=0x7f8f14009ca0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 -- 172.21.15.28:0/3597481049 shutdown_connections 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 --2- 172.21.15.28:0/3597481049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24109940 0x7f8f24115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 --2- 172.21.15.28:0/3597481049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f2410f030 0x7f8f2410f410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 --2- 172.21.15.28:0/3597481049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24108f80 0x7f8f24109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 -- 172.21.15.28:0/3597481049 >> 172.21.15.28:0/3597481049 conn(0x7f8f240787d0 msgr2=0x7f8f24107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 -- 172.21.15.28:0/3597481049 shutdown_connections 2024-04-26T01:36:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.466+0000 7f8f23e00700 1 -- 172.21.15.28:0/3597481049 wait complete. 2024-04-26T01:36:19.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:19 smithi028 ceph-mon[28247]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.468+0000 7f8f23e00700 1 Processor -- start 2024-04-26T01:36:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.468+0000 7f8f23e00700 1 -- start start 2024-04-26T01:36:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f23e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24108f80 0x7f8f2419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f23e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24109940 0x7f8f2419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f23e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f2410f030 0x7f8f241a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f23e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24119140 con 0x7f8f24109940 2024-04-26T01:36:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f23e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24118fc0 con 0x7f8f24108f80 2024-04-26T01:36:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f23e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8f241192c0 con 0x7f8f2410f030 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f22000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24109940 0x7f8f2419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f22a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24108f80 0x7f8f2419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f23400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f2410f030 0x7f8f241a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f22000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24109940 0x7f8f2419fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53034/0 (socket says 172.21.15.28:53034) 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.469+0000 7f8f22000700 1 -- 172.21.15.28:0/1832566236 learned_addr learned my addr 172.21.15.28:0/1832566236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f22a00700 1 -- 172.21.15.28:0/1832566236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f2410f030 msgr2=0x7f8f241a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f22a00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f2410f030 0x7f8f241a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f22a00700 1 -- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24109940 msgr2=0x7f8f2419fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f22a00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24109940 0x7f8f2419fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f22a00700 1 -- 172.21.15.28:0/1832566236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f241a45a0 con 0x7f8f24108f80 2024-04-26T01:36:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f22000700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24109940 0x7f8f2419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:36:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f22a00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24108f80 0x7f8f2419f590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8f140098e0 tx=0x7f8f140057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f1b400700 1 -- 172.21.15.28:0/1832566236 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14027070 con 0x7f8f24108f80 2024-04-26T01:36:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f1b400700 1 -- 172.21.15.28:0/1832566236 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f14022470 con 0x7f8f24108f80 2024-04-26T01:36:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f241a4830 con 0x7f8f24108f80 2024-04-26T01:36:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.470+0000 7f8f1b400700 1 -- 172.21.15.28:0/1832566236 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14017650 con 0x7f8f24108f80 2024-04-26T01:36:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.471+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f241ac110 con 0x7f8f24108f80 2024-04-26T01:36:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.472+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f24069730 con 0x7f8f24108f80 2024-04-26T01:36:19.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.473+0000 7f8f1b400700 1 -- 172.21.15.28:0/1832566236 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8f1402a070 con 0x7f8f24108f80 2024-04-26T01:36:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.476+0000 7f8f1b400700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8f080a3cc0 0x7f8f080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.476+0000 7f8f1b400700 1 -- 172.21.15.28:0/1832566236 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8f140d0c20 con 0x7f8f24108f80 2024-04-26T01:36:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.476+0000 7f8f22000700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8f080a3cc0 0x7f8f080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.477+0000 7f8f22000700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8f080a3cc0 0x7f8f080a6180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8f2410e710 tx=0x7f8f10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.479+0000 7f8f1b400700 1 -- 172.21.15.28:0/1832566236 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f1405eff0 con 0x7f8f24108f80 2024-04-26T01:36:19.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:19 smithi169 ceph-mon[32851]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:19 smithi053 ceph-mon[32593]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:19.717 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:19 smithi053 podman[62339]: 2024-04-26 01:36:19.234175484 +0000 UTC m=+0.403618435 container init 57bcc154dc479d8ce882e006502be1226f040bf840187c0198afcccfc8e5fe6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:36:19.717 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:19 smithi053 podman[62339]: 2024-04-26 01:36:19.237282542 +0000 UTC m=+0.406725490 container start 57bcc154dc479d8ce882e006502be1226f040bf840187c0198afcccfc8e5fe6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-04-26T01:36:19.718 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:19 smithi053 ceph-mds[62358]: -- 172.21.15.53:0/889508869 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607ad517a40 con 0x5607ad4eac00 2024-04-26T01:36:19.718 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:19 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h[62354]: starting mds.h at 2024-04-26T01:36:19.718 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:19 smithi053 bash[62339]: 57bcc154dc479d8ce882e006502be1226f040bf840187c0198afcccfc8e5fe6d 2024-04-26T01:36:19.718 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 01:36:19 smithi053 systemd[1]: Started Ceph mds.h for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.770+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f241a08f0 con 0x7f8f24108f80 2024-04-26T01:36:19.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.772+0000 7f8f1b400700 1 -- 172.21.15.28:0/1832566236 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7168 (secure 0 0 0) 0x7f8f14013070 con 0x7f8f24108f80 2024-04-26T01:36:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:36:19.769 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8f080a3cc0 msgr2=0x7f8f080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8f080a3cc0 0x7f8f080a6180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8f2410e710 tx=0x7f8f10009260 comp rx=0 tx=0).stop 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24108f80 msgr2=0x7f8f2419f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24108f80 0x7f8f2419f590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8f140098e0 tx=0x7f8f140057c0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 shutdown_connections 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8f080a3cc0 0x7f8f080a6180 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f2410f030 0x7f8f241a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24108f80 0x7f8f2419f590 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 --2- 172.21.15.28:0/1832566236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24109940 0x7f8f2419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.775+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 >> 172.21.15.28:0/1832566236 conn(0x7f8f240787d0 msgr2=0x7f8f240798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.776+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 shutdown_connections 2024-04-26T01:36:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:19.776+0000 7f8f23e00700 1 -- 172.21.15.28:0/1832566236 wait complete. 2024-04-26T01:36:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-26T01:36:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.53:6836/2195700399,v1:172.21.15.53:6837/2195700399] up:boot 2024-04-26T01:36:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: fsmap 8 up:standby 2024-04-26T01:36:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:36:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:36:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:20.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:20.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:20.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: Deploying daemon mds.i on smithi169 2024-04-26T01:36:20.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:20 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1832566236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.53:6836/2195700399,v1:172.21.15.53:6837/2195700399] up:boot 2024-04-26T01:36:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: fsmap 8 up:standby 2024-04-26T01:36:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:36:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:36:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:20.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:20.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:20.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: Deploying daemon mds.i on smithi169 2024-04-26T01:36:20.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:20 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1832566236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:20.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.53:6836/2195700399,v1:172.21.15.53:6837/2195700399] up:boot 2024-04-26T01:36:20.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: fsmap 8 up:standby 2024-04-26T01:36:20.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: Deploying daemon mds.i on smithi169 2024-04-26T01:36:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:20 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1832566236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:21.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:21 smithi169 ceph-mon[32851]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:21.659 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:21 smithi169 systemd[1]: Starting Ceph mds.i for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:36:21.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:21 smithi053 ceph-mon[32593]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:21.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:21 smithi028 ceph-mon[28247]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:22.050 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:21 smithi169 podman[64065]: 2024-04-26 01:36:21.698253699 +0000 UTC m=+0.017525559 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:36:22.050 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:21 smithi169 podman[64065]: 2024-04-26 01:36:21.810355223 +0000 UTC m=+0.129627075 container create 36c61d61a6700436930f17f4cdb0757369d037fd27b72677cacde837364dcf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T01:36:22.051 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:22 smithi169 podman[64065]: 2024-04-26 01:36:22.055621483 +0000 UTC m=+0.374893334 container init 36c61d61a6700436930f17f4cdb0757369d037fd27b72677cacde837364dcf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:36:22.409 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:22 smithi169 podman[64065]: 2024-04-26 01:36:22.058851835 +0000 UTC m=+0.378123695 container start 36c61d61a6700436930f17f4cdb0757369d037fd27b72677cacde837364dcf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T01:36:22.409 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:22 smithi169 ceph-mds[64084]: -- 172.21.15.169:0/2824144097 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561a40e73a40 con 0x561a40e46c00 2024-04-26T01:36:22.409 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: starting mds.i at 2024-04-26T01:36:22.409 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:22 smithi169 bash[64065]: 36c61d61a6700436930f17f4cdb0757369d037fd27b72677cacde837364dcf6c 2024-04-26T01:36:22.409 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:36:22 smithi169 systemd[1]: Started Ceph mds.i for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:22.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:36:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:23.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:23.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: Deploying daemon mds.g on smithi028 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] up:boot 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: fsmap 9 up:standby 2024-04-26T01:36:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:23 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: Deploying daemon mds.g on smithi028 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] up:boot 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: fsmap 9 up:standby 2024-04-26T01:36:23.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:23 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:36:23.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:23.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: Deploying daemon mds.g on smithi028 2024-04-26T01:36:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] up:boot 2024-04-26T01:36:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: fsmap 9 up:standby 2024-04-26T01:36:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:23 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:36:25.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:25 smithi169 ceph-mon[32851]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:25.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:25 smithi028 ceph-mon[28247]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:25 smithi053 ceph-mon[32593]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 -- 172.21.15.28:0/2910789739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcebc074c70 msgr2=0x7fcebc075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 --2- 172.21.15.28:0/2910789739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcebc074c70 0x7fcebc075050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fceac009990 tx=0x7fceac009ca0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 -- 172.21.15.28:0/2910789739 shutdown_connections 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 --2- 172.21.15.28:0/2910789739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc113c10 0x7fcebc116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 --2- 172.21.15.28:0/2910789739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcebc074c70 0x7fcebc075050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 --2- 172.21.15.28:0/2910789739 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcebc075590 0x7fcebc1136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 -- 172.21.15.28:0/2910789739 >> 172.21.15.28:0/2910789739 conn(0x7fcebc079ba0 msgr2=0x7fcebc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 -- 172.21.15.28:0/2910789739 shutdown_connections 2024-04-26T01:36:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.619+0000 7fcec2a00700 1 -- 172.21.15.28:0/2910789739 wait complete. 2024-04-26T01:36:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.620+0000 7fcec2a00700 1 Processor -- start 2024-04-26T01:36:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.620+0000 7fcec2a00700 1 -- start start 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcec2a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc074c70 0x7fcebc1a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcec2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcebc075590 0x7fcebc1a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcec2a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcebc113c10 0x7fcebc19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcec2a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc119240 con 0x7fcebc075590 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcec2a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1190c0 con 0x7fcebc074c70 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcec2a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1193c0 con 0x7fcebc113c10 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcebbe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc074c70 0x7fcebc1a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcebbe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc074c70 0x7fcebc1a53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43726/0 (socket says 172.21.15.28:43726) 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcebbe00700 1 -- 172.21.15.28:0/3451106526 learned_addr learned my addr 172.21.15.28:0/3451106526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcebb400700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcebc075590 0x7fcebc1a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.621+0000 7fcec0a00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcebc113c10 0x7fcebc19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fcebbe00700 1 -- 172.21.15.28:0/3451106526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcebc113c10 msgr2=0x7fcebc19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fcebbe00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcebc113c10 0x7fcebc19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fcebbe00700 1 -- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcebc075590 msgr2=0x7fcebc1a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fcebbe00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcebc075590 0x7fcebc1a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fcebbe00700 1 -- 172.21.15.28:0/3451106526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc19fd00 con 0x7fcebc074c70 2024-04-26T01:36:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fcec0a00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcebc113c10 0x7fcebc19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:36:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fcebbe00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc074c70 0x7fcebc1a53e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fceac000c00 tx=0x7fceac0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fceb8c00700 1 -- 172.21.15.28:0/3451106526 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac027070 con 0x7fcebc074c70 2024-04-26T01:36:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fceb8c00700 1 -- 172.21.15.28:0/3451106526 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceac022470 con 0x7fcebc074c70 2024-04-26T01:36:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.622+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcebc19ff90 con 0x7fcebc074c70 2024-04-26T01:36:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.623+0000 7fceb8c00700 1 -- 172.21.15.28:0/3451106526 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac017650 con 0x7fcebc074c70 2024-04-26T01:36:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.623+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcebc1ac2b0 con 0x7fcebc074c70 2024-04-26T01:36:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.624+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcebc069730 con 0x7fcebc074c70 2024-04-26T01:36:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.628+0000 7fceb8c00700 1 -- 172.21.15.28:0/3451106526 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fceac02a070 con 0x7fcebc074c70 2024-04-26T01:36:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.629+0000 7fceb8c00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcea00a3c40 0x7fcea00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.629+0000 7fceb8c00700 1 -- 172.21.15.28:0/3451106526 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fceac0d0d30 con 0x7fcebc074c70 2024-04-26T01:36:26.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.629+0000 7fcebb400700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcea00a3c40 0x7fcea00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:26.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.630+0000 7fceb8c00700 1 -- 172.21.15.28:0/3451106526 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fceac013070 con 0x7fcebc074c70 2024-04-26T01:36:26.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.630+0000 7fcebb400700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcea00a3c40 0x7fcea00a6100 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcebc110580 tx=0x7fcea8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:26.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.922+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcebc1ac480 con 0x7fcebc074c70 2024-04-26T01:36:26.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.923+0000 7fceb8c00700 1 -- 172.21.15.28:0/3451106526 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7998 (secure 0 0 0) 0x7fceac09dba0 con 0x7fcebc074c70 2024-04-26T01:36:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:36:26.920 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T01:36:26.920 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:26 smithi028 systemd[1]: Starting Ceph mds.g for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:36:26.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcea00a3c40 msgr2=0x7fcea00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:26.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcea00a3c40 0x7fcea00a6100 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcebc110580 tx=0x7fcea8009260 comp rx=0 tx=0).stop 2024-04-26T01:36:26.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc074c70 msgr2=0x7fcebc1a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:26.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc074c70 0x7fcebc1a53e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fceac000c00 tx=0x7fceac0057c0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 shutdown_connections 2024-04-26T01:36:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcea00a3c40 0x7fcea00a6100 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcebc113c10 0x7fcebc19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc074c70 0x7fcebc1a53e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 --2- 172.21.15.28:0/3451106526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcebc075590 0x7fcebc1a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.927+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 >> 172.21.15.28:0/3451106526 conn(0x7fcebc079ba0 msgr2=0x7fcebc112ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.928+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 shutdown_connections 2024-04-26T01:36:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:26.928+0000 7fcec2a00700 1 -- 172.21.15.28:0/3451106526 wait complete. 2024-04-26T01:36:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-04-26T01:36:27.208 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:27 smithi028 podman[67733]: 2024-04-26 01:36:26.93671926 +0000 UTC m=+0.034703368 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:36:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:27 smithi053 ceph-mon[32593]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:27 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3451106526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:27 smithi028 ceph-mon[28247]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:27.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:27 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3451106526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:27.532 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:27 smithi028 podman[67733]: 2024-04-26 01:36:27.274769875 +0000 UTC m=+0.372754010 container create 5127c778750182e1025f0b05561b3da4a7a0e3cc8700dd8e295a5fd731aa7dcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:36:27.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:27 smithi169 ceph-mon[32851]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:27.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:27 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3451106526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:27.785 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:27 smithi028 podman[67733]: 2024-04-26 01:36:27.537307981 +0000 UTC m=+0.635292166 container init 5127c778750182e1025f0b05561b3da4a7a0e3cc8700dd8e295a5fd731aa7dcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:36:27.785 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:27 smithi028 podman[67733]: 2024-04-26 01:36:27.547755756 +0000 UTC m=+0.645739860 container start 5127c778750182e1025f0b05561b3da4a7a0e3cc8700dd8e295a5fd731aa7dcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T01:36:27.785 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:27 smithi028 ceph-mds[67768]: -- 172.21.15.28:0/1837106759 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b0cc51a40 con 0x564b0cc24800 2024-04-26T01:36:27.785 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:27 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g[67762]: starting mds.g at 2024-04-26T01:36:27.786 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:27 smithi028 bash[67733]: 5127c778750182e1025f0b05561b3da4a7a0e3cc8700dd8e295a5fd731aa7dcd 2024-04-26T01:36:28.228 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:36:27 smithi028 systemd[1]: Started Ceph mds.g for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: Deploying daemon mds.a on smithi028 2024-04-26T01:36:29.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.28:6838/2114874192,v1:172.21.15.28:6839/2114874192] up:boot 2024-04-26T01:36:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: fsmap 10 up:standby 2024-04-26T01:36:29.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:28 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:36:29.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:29.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:29.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:29.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:29.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: Deploying daemon mds.a on smithi028 2024-04-26T01:36:29.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.28:6838/2114874192,v1:172.21.15.28:6839/2114874192] up:boot 2024-04-26T01:36:29.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: fsmap 10 up:standby 2024-04-26T01:36:29.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:28 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:36:29.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:29.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: Deploying daemon mds.a on smithi028 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.28:6838/2114874192,v1:172.21.15.28:6839/2114874192] up:boot 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: fsmap 10 up:standby 2024-04-26T01:36:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:28 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:36:30.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:36:30.788 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:31.160 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:30 smithi028 ceph-mon[28247]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:31.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:30 smithi053 ceph-mon[32593]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:31.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:30 smithi169 ceph-mon[32851]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:32.764 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:36:32 smithi028 podman[68143]: 2024-04-26 01:36:32.525674294 +0000 UTC m=+0.153942182 container create 984a7114bd058e87596344d50f76734e525bdca4ff87647edee5a445c454c1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-26T01:36:33.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:32 smithi028 ceph-mon[28247]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:33.031 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:36:32 smithi028 podman[68143]: 2024-04-26 01:36:32.813565199 +0000 UTC m=+0.441833096 container init 984a7114bd058e87596344d50f76734e525bdca4ff87647edee5a445c454c1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T01:36:33.031 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:36:32 smithi028 podman[68143]: 2024-04-26 01:36:32.82468407 +0000 UTC m=+0.452951967 container start 984a7114bd058e87596344d50f76734e525bdca4ff87647edee5a445c454c1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T01:36:33.031 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:36:32 smithi028 ceph-mds[68180]: -- 172.21.15.28:0/2367268378 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e69a57fa40 con 0x55e69a553000 2024-04-26T01:36:33.031 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:36:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a[68175]: starting mds.a at 2024-04-26T01:36:33.031 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:36:32 smithi028 bash[68143]: 984a7114bd058e87596344d50f76734e525bdca4ff87647edee5a445c454c1da 2024-04-26T01:36:33.032 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:36:32 smithi028 systemd[1]: Started Ceph mds.a for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:32 smithi053 ceph-mon[32593]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:33.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:32 smithi169 ceph-mon[32851]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 -- 172.21.15.28:0/324686950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d7406a750 msgr2=0x7f5d7406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 --2- 172.21.15.28:0/324686950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d7406a750 0x7f5d7406abb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0098f0 tx=0x7f5d5c009c00 comp rx=0 tx=0).stop 2024-04-26T01:36:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 -- 172.21.15.28:0/324686950 shutdown_connections 2024-04-26T01:36:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 --2- 172.21.15.28:0/324686950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7406b0f0 0x7f5d7410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 --2- 172.21.15.28:0/324686950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d74108a30 0x7f5d74108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 --2- 172.21.15.28:0/324686950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d7406a750 0x7f5d7406abb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 -- 172.21.15.28:0/324686950 >> 172.21.15.28:0/324686950 conn(0x7f5d740788f0 msgr2=0x7f5d74078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 -- 172.21.15.28:0/324686950 shutdown_connections 2024-04-26T01:36:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.588+0000 7f5d7a400700 1 -- 172.21.15.28:0/324686950 wait complete. 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.589+0000 7f5d7a400700 1 Processor -- start 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d7a400700 1 -- start start 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d7a400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d7406a750 0x7f5d74199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d7a400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d7406b0f0 0x7f5d74199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d7a400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d74108a30 0x7f5d7419da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d7a400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74112bd0 con 0x7f5d7406a750 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d7a400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74112a50 con 0x7f5d74108a30 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d7a400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74112d50 con 0x7f5d7406b0f0 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d73e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d74108a30 0x7f5d7419da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d73400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d7406a750 0x7f5d74199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d6aa00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d7406b0f0 0x7f5d74199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d73e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d74108a30 0x7f5d7419da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48454/0 (socket says 172.21.15.28:48454) 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.590+0000 7f5d73e00700 1 -- 172.21.15.28:0/1528293944 learned_addr learned my addr 172.21.15.28:0/1528293944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d73e00700 1 -- 172.21.15.28:0/1528293944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d7406b0f0 msgr2=0x7f5d74199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d73e00700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d7406b0f0 0x7f5d74199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d73e00700 1 -- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d7406a750 msgr2=0x7f5d74199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d73e00700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d7406a750 0x7f5d74199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d73e00700 1 -- 172.21.15.28:0/1528293944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7419e150 con 0x7f5d74108a30 2024-04-26T01:36:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d73e00700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d74108a30 0x7f5d7419da10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00ef30 tx=0x7f5d6c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d70c00700 1 -- 172.21.15.28:0/1528293944 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0092e0 con 0x7f5d74108a30 2024-04-26T01:36:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d70c00700 1 -- 172.21.15.28:0/1528293944 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6c004bb0 con 0x7f5d74108a30 2024-04-26T01:36:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.591+0000 7f5d70c00700 1 -- 172.21.15.28:0/1528293944 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c004020 con 0x7f5d74108a30 2024-04-26T01:36:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.592+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7419e3e0 con 0x7f5d74108a30 2024-04-26T01:36:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.592+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d74101940 con 0x7f5d74108a30 2024-04-26T01:36:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.593+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d74104170 con 0x7f5d74108a30 2024-04-26T01:36:33.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.595+0000 7f5d70c00700 1 -- 172.21.15.28:0/1528293944 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5d6c01b030 con 0x7f5d74108a30 2024-04-26T01:36:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.596+0000 7f5d70c00700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5d580a3cc0 0x7f5d580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.596+0000 7f5d70c00700 1 -- 172.21.15.28:0/1528293944 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5d6c0cfa20 con 0x7f5d74108a30 2024-04-26T01:36:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.596+0000 7f5d73400700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5d580a3cc0 0x7f5d580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:33.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.597+0000 7f5d73400700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5d580a3cc0 0x7f5d580a6180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009750 tx=0x7f5d64012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.600+0000 7f5d70c00700 1 -- 172.21.15.28:0/1528293944 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d6c022020 con 0x7f5d74108a30 2024-04-26T01:36:33.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.881+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d7402df80 con 0x7f5d74108a30 2024-04-26T01:36:33.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.882+0000 7f5d70c00700 1 -- 172.21.15.28:0/1528293944 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f5d6c010040 con 0x7f5d74108a30 2024-04-26T01:36:33.878 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:36:33.879 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24491,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24497,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T01:36:33.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.885+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5d580a3cc0 msgr2=0x7f5d580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:33.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.885+0000 7f5d7a400700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5d580a3cc0 0x7f5d580a6180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009750 tx=0x7f5d64012040 comp rx=0 tx=0).stop 2024-04-26T01:36:33.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.885+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d74108a30 msgr2=0x7f5d7419da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:33.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.885+0000 7f5d7a400700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d74108a30 0x7f5d7419da10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00ef30 tx=0x7f5d6c00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.886+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 shutdown_connections 2024-04-26T01:36:33.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.886+0000 7f5d7a400700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f5d580a3cc0 0x7f5d580a6180 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.886+0000 7f5d7a400700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d74108a30 0x7f5d7419da10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.886+0000 7f5d7a400700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d7406a750 0x7f5d74199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.886+0000 7f5d7a400700 1 --2- 172.21.15.28:0/1528293944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d7406b0f0 0x7f5d74199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:33.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.886+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 >> 172.21.15.28:0/1528293944 conn(0x7f5d740788f0 msgr2=0x7f5d740ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:33.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.886+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 shutdown_connections 2024-04-26T01:36:33.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:33.886+0000 7f5d7a400700 1 -- 172.21.15.28:0/1528293944 wait complete. 2024-04-26T01:36:33.883 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-04-26T01:36:34.066 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.28:6840/2299937376,v1:172.21.15.28:6841/2299937376] up:boot 2024-04-26T01:36:34.066 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: fsmap 11 up:standby 2024-04-26T01:36:34.066 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:36:34.066 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.066 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.066 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.067 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:34.067 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:34.067 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.067 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: Deploying daemon mds.b on smithi053 2024-04-26T01:36:34.067 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:33 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1528293944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:34.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.28:6840/2299937376,v1:172.21.15.28:6841/2299937376] up:boot 2024-04-26T01:36:34.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: fsmap 11 up:standby 2024-04-26T01:36:34.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:36:34.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:34.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:34.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: Deploying daemon mds.b on smithi053 2024-04-26T01:36:34.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:33 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1528293944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.28:6840/2299937376,v1:172.21.15.28:6841/2299937376] up:boot 2024-04-26T01:36:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: fsmap 11 up:standby 2024-04-26T01:36:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:36:34.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:36:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:36:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: Deploying daemon mds.b on smithi053 2024-04-26T01:36:34.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:33 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1528293944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:35.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:35 smithi169 ceph-mon[32851]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:35.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:35 smithi028 ceph-mon[28247]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:35 smithi053 ceph-mon[32593]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:35.468 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 systemd[1]: Starting Ceph mds.b for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:36:35.826 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 podman[62654]: 2024-04-26 01:36:35.459127374 +0000 UTC m=+0.018520552 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:36:35.827 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 podman[62654]: 2024-04-26 01:36:35.578396689 +0000 UTC m=+0.137789839 container create b9e03549be77c1d36178e58f7ea1f62a7b31450ca5c6949049635f1662a91c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:36:35.827 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 podman[62654]: 2024-04-26 01:36:35.831450208 +0000 UTC m=+0.390843368 container init b9e03549be77c1d36178e58f7ea1f62a7b31450ca5c6949049635f1662a91c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True) 2024-04-26T01:36:36.217 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 podman[62654]: 2024-04-26 01:36:35.834467458 +0000 UTC m=+0.393860621 container start b9e03549be77c1d36178e58f7ea1f62a7b31450ca5c6949049635f1662a91c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-26T01:36:36.217 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 ceph-mds[62673]: -- 172.21.15.53:0/1671708897 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564f706a9a40 con 0x564f7067cc00 2024-04-26T01:36:36.217 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b[62669]: starting mds.b at 2024-04-26T01:36:36.217 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 bash[62654]: b9e03549be77c1d36178e58f7ea1f62a7b31450ca5c6949049635f1662a91c95 2024-04-26T01:36:36.218 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:36:35 smithi053 systemd[1]: Started Ceph mds.b for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:36:36.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:36:37.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:37.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] up:boot 2024-04-26T01:36:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: fsmap 12 up:standby 2024-04-26T01:36:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:36:37.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:36 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] up:boot 2024-04-26T01:36:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: fsmap 12 up:standby 2024-04-26T01:36:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:36:37.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:36 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:37.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:37.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.237 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.237 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:37.237 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] up:boot 2024-04-26T01:36:37.237 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: fsmap 12 up:standby 2024-04-26T01:36:37.237 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:36:37.237 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:36 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:39.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:38 smithi028 ceph-mon[28247]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:39.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:38 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:38 smithi169 ceph-mon[32851]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:38 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:38 smithi053 ceph-mon[32593]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:38 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.324+0000 7f60f0800700 1 -- 172.21.15.28:0/4275676711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 msgr2=0x7f60e810ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.324+0000 7f60f0800700 1 --2- 172.21.15.28:0/4275676711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 0x7f60e810ef00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009990 tx=0x7f60d8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.325+0000 7f60f0800700 1 -- 172.21.15.28:0/4275676711 shutdown_connections 2024-04-26T01:36:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.325+0000 7f60f0800700 1 --2- 172.21.15.28:0/4275676711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60e810f440 0x7f60e8111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.325+0000 7f60f0800700 1 --2- 172.21.15.28:0/4275676711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60e8074c70 0x7f60e8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.325+0000 7f60f0800700 1 --2- 172.21.15.28:0/4275676711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 0x7f60e810ef00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.325+0000 7f60f0800700 1 -- 172.21.15.28:0/4275676711 >> 172.21.15.28:0/4275676711 conn(0x7f60e8079ba0 msgr2=0x7f60e8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.325+0000 7f60f0800700 1 -- 172.21.15.28:0/4275676711 shutdown_connections 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.325+0000 7f60f0800700 1 -- 172.21.15.28:0/4275676711 wait complete. 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.326+0000 7f60f0800700 1 Processor -- start 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.326+0000 7f60f0800700 1 -- start start 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60f0800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60e8074c70 0x7f60e819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60f0800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 0x7f60e819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60f0800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60e810f440 0x7f60e819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60f0800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60e8114e50 con 0x7f60e8075590 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60f0800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f60e8114cd0 con 0x7f60e8074c70 2024-04-26T01:36:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60f0800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f60e8114fd0 con 0x7f60e810f440 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60ede00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60e8074c70 0x7f60e819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60ed400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 0x7f60e819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60ede00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60e8074c70 0x7f60e819b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35530/0 (socket says 172.21.15.28:35530) 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60ede00700 1 -- 172.21.15.28:0/3805582398 learned_addr learned my addr 172.21.15.28:0/3805582398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.327+0000 7f60ee800700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60e810f440 0x7f60e819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60ede00700 1 -- 172.21.15.28:0/3805582398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60e810f440 msgr2=0x7f60e819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60ede00700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60e810f440 0x7f60e819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60ede00700 1 -- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 msgr2=0x7f60e819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60ede00700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 0x7f60e819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60ede00700 1 -- 172.21.15.28:0/3805582398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e81a0320 con 0x7f60e8074c70 2024-04-26T01:36:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60ed400700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 0x7f60e819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:36:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60ede00700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60e8074c70 0x7f60e819b360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00e9a0 tx=0x7f60dc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60e2a00700 1 -- 172.21.15.28:0/3805582398 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc00ca10 con 0x7f60e8074c70 2024-04-26T01:36:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60e2a00700 1 -- 172.21.15.28:0/3805582398 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60dc00cbb0 con 0x7f60e8074c70 2024-04-26T01:36:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e81a0610 con 0x7f60e8074c70 2024-04-26T01:36:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.328+0000 7f60e2a00700 1 -- 172.21.15.28:0/3805582398 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc018680 con 0x7f60e8074c70 2024-04-26T01:36:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.329+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e81a7e90 con 0x7f60e8074c70 2024-04-26T01:36:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.330+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e8069730 con 0x7f60e8074c70 2024-04-26T01:36:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.332+0000 7f60e2a00700 1 -- 172.21.15.28:0/3805582398 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f60dc010430 con 0x7f60e8074c70 2024-04-26T01:36:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.333+0000 7f60e2a00700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f60d00a3c40 0x7f60d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.333+0000 7f60e2a00700 1 -- 172.21.15.28:0/3805582398 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f60dc0d0180 con 0x7f60e8074c70 2024-04-26T01:36:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.333+0000 7f60ed400700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f60d00a3c40 0x7f60d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.334+0000 7f60ed400700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f60d00a3c40 0x7f60d00a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f60e8111090 tx=0x7f60d801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.338+0000 7f60e2a00700 1 -- 172.21.15.28:0/3805582398 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60dc09cf60 con 0x7f60e8074c70 2024-04-26T01:36:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.642+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60e802df80 con 0x7f60e8074c70 2024-04-26T01:36:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.643+0000 7f60e2a00700 1 -- 172.21.15.28:0/3805582398 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10473 (secure 0 0 0) 0x7f60dc09c860 con 0x7f60e8074c70 2024-04-26T01:36:40.640 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:36:40.640 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24491,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24497,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T01:36:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f60d00a3c40 msgr2=0x7f60d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f60d00a3c40 0x7f60d00a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f60e8111090 tx=0x7f60d801a040 comp rx=0 tx=0).stop 2024-04-26T01:36:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60e8074c70 msgr2=0x7f60e819b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60e8074c70 0x7f60e819b360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00e9a0 tx=0x7f60dc00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 shutdown_connections 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f60d00a3c40 0x7f60d00a6100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60e810f440 0x7f60e819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60e8074c70 0x7f60e819b360 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 --2- 172.21.15.28:0/3805582398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e8075590 0x7f60e819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 >> 172.21.15.28:0/3805582398 conn(0x7f60e8079ba0 msgr2=0x7f60e80ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 shutdown_connections 2024-04-26T01:36:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:40.647+0000 7f60f0800700 1 -- 172.21.15.28:0/3805582398 wait complete. 2024-04-26T01:36:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-04-26T01:36:41.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:40 smithi028 ceph-mon[28247]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:41.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:40 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3805582398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:41.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:40 smithi169 ceph-mon[32851]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:41.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:40 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3805582398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:41.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:40 smithi053 ceph-mon[32593]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:41.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:40 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3805582398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:36:41.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-04-26T01:36:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:42.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:42.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:42.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:42 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:42.386 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.386 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.386 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:42.386 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:42.386 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.386 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:42 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:42.388 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.388 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.388 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:42.388 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:42.388 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:42.388 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:42 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:43.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:43 smithi169 ceph-mon[32851]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:43 smithi053 ceph-mon[32593]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:43.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:43 smithi028 ceph-mon[28247]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:44.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:44 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.404+0000 7f80ee800700 1 -- 172.21.15.28:0/3044737062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8102a90 msgr2=0x7f80e8102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.404+0000 7f80ee800700 1 --2- 172.21.15.28:0/3044737062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8102a90 0x7f80e8102f10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f80d0009990 tx=0x7f80d0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.405+0000 7f80ee800700 1 -- 172.21.15.28:0/3044737062 shutdown_connections 2024-04-26T01:36:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.405+0000 7f80ee800700 1 --2- 172.21.15.28:0/3044737062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e8103450 0x7f80e810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.405+0000 7f80ee800700 1 --2- 172.21.15.28:0/3044737062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80e8108a50 0x7f80e8108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.405+0000 7f80ee800700 1 --2- 172.21.15.28:0/3044737062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8102a90 0x7f80e8102f10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.405+0000 7f80ee800700 1 -- 172.21.15.28:0/3044737062 >> 172.21.15.28:0/3044737062 conn(0x7f80e8079ba0 msgr2=0x7f80e8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.405+0000 7f80ee800700 1 -- 172.21.15.28:0/3044737062 shutdown_connections 2024-04-26T01:36:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.405+0000 7f80ee800700 1 -- 172.21.15.28:0/3044737062 wait complete. 2024-04-26T01:36:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.406+0000 7f80ee800700 1 Processor -- start 2024-04-26T01:36:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.406+0000 7f80ee800700 1 -- start start 2024-04-26T01:36:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.406+0000 7f80ee800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e8102a90 0x7f80e8199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.406+0000 7f80ee800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80e8103450 0x7f80e81996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.406+0000 7f80ee800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8108a50 0x7f80e819da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.406+0000 7f80ee800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8112c50 con 0x7f80e8108a50 2024-04-26T01:36:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.406+0000 7f80ee800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8112ad0 con 0x7f80e8102a90 2024-04-26T01:36:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.406+0000 7f80ee800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8112dd0 con 0x7f80e8103450 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8108a50 0x7f80e819da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e8102a90 0x7f80e8199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8108a50 0x7f80e819da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38490/0 (socket says 172.21.15.28:38490) 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7e00700 1 -- 172.21.15.28:0/2524615359 learned_addr learned my addr 172.21.15.28:0/2524615359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80dfe00700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80e8103450 0x7f80e81996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7400700 1 -- 172.21.15.28:0/2524615359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80e8103450 msgr2=0x7f80e81996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7400700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80e8103450 0x7f80e81996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7400700 1 -- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8108a50 msgr2=0x7f80e819da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7400700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8108a50 0x7f80e819da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.407+0000 7f80e7400700 1 -- 172.21.15.28:0/2524615359 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e819e190 con 0x7f80e8102a90 2024-04-26T01:36:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.408+0000 7f80e7400700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e8102a90 0x7f80e8199180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f80d800e9f0 tx=0x7f80d800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.408+0000 7f80e4c00700 1 -- 172.21.15.28:0/2524615359 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d800cb70 con 0x7f80e8102a90 2024-04-26T01:36:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.408+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e819e480 con 0x7f80e8102a90 2024-04-26T01:36:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.408+0000 7f80e4c00700 1 -- 172.21.15.28:0/2524615359 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80d800ee90 con 0x7f80e8102a90 2024-04-26T01:36:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.408+0000 7f80e4c00700 1 -- 172.21.15.28:0/2524615359 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8003d40 con 0x7f80e8102a90 2024-04-26T01:36:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.408+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e81a5d00 con 0x7f80e8102a90 2024-04-26T01:36:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.409+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e8069730 con 0x7f80e8102a90 2024-04-26T01:36:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.411+0000 7f80e4c00700 1 -- 172.21.15.28:0/2524615359 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f80d8003ee0 con 0x7f80e8102a90 2024-04-26T01:36:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.412+0000 7f80e4c00700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f80cc0a3c40 0x7f80cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.413+0000 7f80e4c00700 1 -- 172.21.15.28:0/2524615359 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f80d80cfff0 con 0x7f80e8102a90 2024-04-26T01:36:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.413+0000 7f80dfe00700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f80cc0a3c40 0x7f80cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.413+0000 7f80dfe00700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f80cc0a3c40 0x7f80cc0a6100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f80e807a210 tx=0x7f80d000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.417+0000 7f80e4c00700 1 -- 172.21.15.28:0/2524615359 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80d8020390 con 0x7f80e8102a90 2024-04-26T01:36:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:44 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.649+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f80e819a4e0 con 0x7f80e8102a90 2024-04-26T01:36:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.650+0000 7f80e4c00700 1 -- 172.21.15.28:0/2524615359 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f80d809ce10 con 0x7f80e8102a90 2024-04-26T01:36:44.646 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:36:44.646 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-04-26T01:36:44.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:36:44.646 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-04-26T01:36:44.646 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-04-26T01:36:44.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:36:44.646 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-04-26T01:36:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f80cc0a3c40 msgr2=0x7f80cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f80cc0a3c40 0x7f80cc0a6100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f80e807a210 tx=0x7f80d000b260 comp rx=0 tx=0).stop 2024-04-26T01:36:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e8102a90 msgr2=0x7f80e8199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e8102a90 0x7f80e8199180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f80d800e9f0 tx=0x7f80d800ed00 comp rx=0 tx=0).stop 2024-04-26T01:36:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 shutdown_connections 2024-04-26T01:36:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f80cc0a3c40 0x7f80cc0a6100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e8108a50 0x7f80e819da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e8102a90 0x7f80e8199180 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 --2- 172.21.15.28:0/2524615359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80e8103450 0x7f80e81996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.653+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 >> 172.21.15.28:0/2524615359 conn(0x7f80e8079ba0 msgr2=0x7f80e81017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.654+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 shutdown_connections 2024-04-26T01:36:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:44.654+0000 7f80ee800700 1 -- 172.21.15.28:0/2524615359 wait complete. 2024-04-26T01:36:44.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:44.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:44 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:45.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:45 smithi028 ceph-mon[28247]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:45.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:45 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2524615359' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:36:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:45 smithi053 ceph-mon[32593]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:45 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2524615359' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:36:45.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:45 smithi169 ceph-mon[32851]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:45.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:45 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2524615359' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:36:46.199 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T01:36:46.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T01:36:46.668 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:47 smithi053 ceph-mon[32593]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:47.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:47 smithi028 ceph-mon[28247]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:47.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:47 smithi169 ceph-mon[32851]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 -- 172.21.15.28:0/3450053557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaac10f440 msgr2=0x7ffaac111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 --2- 172.21.15.28:0/3450053557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaac10f440 0x7ffaac111830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40098f0 tx=0x7ffaa4009c00 comp rx=0 tx=0).stop 2024-04-26T01:36:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 -- 172.21.15.28:0/3450053557 shutdown_connections 2024-04-26T01:36:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 --2- 172.21.15.28:0/3450053557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaac10f440 0x7ffaac111830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 --2- 172.21.15.28:0/3450053557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffaac074c70 0x7ffaac075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 --2- 172.21.15.28:0/3450053557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaac075590 0x7ffaac10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:48.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 -- 172.21.15.28:0/3450053557 >> 172.21.15.28:0/3450053557 conn(0x7ffaac079ba0 msgr2=0x7ffaac079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:48.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 -- 172.21.15.28:0/3450053557 shutdown_connections 2024-04-26T01:36:48.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.865+0000 7ffab2600700 1 -- 172.21.15.28:0/3450053557 wait complete. 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.866+0000 7ffab2600700 1 Processor -- start 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.866+0000 7ffab2600700 1 -- start start 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffab2600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffaac074c70 0x7ffaac19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffab2600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaac075590 0x7ffaac19b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffab2600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaac10f440 0x7ffaac19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffab2600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac114e20 con 0x7ffaac10f440 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffab2600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac114ca0 con 0x7ffaac074c70 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffab2600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac114fa0 con 0x7ffaac075590 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffaab400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffaac074c70 0x7ffaac19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffaab400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffaac074c70 0x7ffaac19b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:60478/0 (socket says 172.21.15.28:60478) 2024-04-26T01:36:48.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffaab400700 1 -- 172.21.15.28:0/485803831 learned_addr learned my addr 172.21.15.28:0/485803831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffaa2a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaac075590 0x7ffaac19b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.867+0000 7ffaabe00700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaac10f440 0x7ffaac19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffaa2a00700 1 -- 172.21.15.28:0/485803831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffaac074c70 msgr2=0x7ffaac19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffaa2a00700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffaac074c70 0x7ffaac19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffaa2a00700 1 -- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaac10f440 msgr2=0x7ffaac19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffaa2a00700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaac10f440 0x7ffaac19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffaa2a00700 1 -- 172.21.15.28:0/485803831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac1a0300 con 0x7ffaac075590 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffaa2a00700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaac075590 0x7ffaac19b830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400e9c0 tx=0x7ffa9400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffaa8c00700 1 -- 172.21.15.28:0/485803831 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9400cb70 con 0x7ffaac075590 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac1a05f0 con 0x7ffaac075590 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.868+0000 7ffaa8c00700 1 -- 172.21.15.28:0/485803831 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9400ee60 con 0x7ffaac075590 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.869+0000 7ffaa8c00700 1 -- 172.21.15.28:0/485803831 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94018680 con 0x7ffaac075590 2024-04-26T01:36:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.869+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac1a7e70 con 0x7ffaac075590 2024-04-26T01:36:48.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.870+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaac069730 con 0x7ffaac075590 2024-04-26T01:36:48.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.872+0000 7ffaa8c00700 1 -- 172.21.15.28:0/485803831 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffa9401f020 con 0x7ffaac075590 2024-04-26T01:36:48.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.873+0000 7ffaa8c00700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ffa900a3c40 0x7ffa900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.873+0000 7ffaa8c00700 1 -- 172.21.15.28:0/485803831 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffa940cfef0 con 0x7ffaac075590 2024-04-26T01:36:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.874+0000 7ffaab400700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ffa900a3c40 0x7ffa900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.874+0000 7ffaab400700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ffa900a3c40 0x7ffa900a6100 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009fd0 tx=0x7ffa9c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:48.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:48.876+0000 7ffaa8c00700 1 -- 172.21.15.28:0/485803831 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa9409ccd0 con 0x7ffaac075590 2024-04-26T01:36:49.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.109+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ffaac02df80 con 0x7ffaac075590 2024-04-26T01:36:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.217+0000 7ffaa8c00700 1 -- 172.21.15.28:0/485803831 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7ffa9409c5d0 con 0x7ffaac075590 2024-04-26T01:36:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ffa900a3c40 msgr2=0x7ffa900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ffa900a3c40 0x7ffa900a6100 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009fd0 tx=0x7ffa9c00b040 comp rx=0 tx=0).stop 2024-04-26T01:36:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaac075590 msgr2=0x7ffaac19b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaac075590 0x7ffaac19b830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400e9c0 tx=0x7ffa9400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:36:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 shutdown_connections 2024-04-26T01:36:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ffa900a3c40 0x7ffa900a6100 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaac10f440 0x7ffaac19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffaac074c70 0x7ffaac19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 --2- 172.21.15.28:0/485803831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaac075590 0x7ffaac19b830 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.224+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 >> 172.21.15.28:0/485803831 conn(0x7ffaac079ba0 msgr2=0x7ffaac0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.229+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 shutdown_connections 2024-04-26T01:36:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:49.229+0000 7ffab2600700 1 -- 172.21.15.28:0/485803831 wait complete. 2024-04-26T01:36:49.229 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-04-26T01:36:49.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:49 smithi169 ceph-mon[32851]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:49 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:49 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/485803831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:36:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:49 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:36:49.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:49 smithi053 ceph-mon[32593]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:49.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:49.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:49.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:49 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:49 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/485803831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:36:49.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:49 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:36:49.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:49 smithi028 ceph-mon[28247]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:49.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:49.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:49.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:49 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:36:49.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:49 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/485803831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:36:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:49 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:36:50.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T01:36:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:50 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:36:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:50 smithi053 ceph-mon[32593]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T01:36:50.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:50 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:36:50.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:50 smithi028 ceph-mon[28247]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T01:36:50.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:50 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:36:50.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:50 smithi169 ceph-mon[32851]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T01:36:50.677 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:51 smithi028 ceph-mon[28247]: pgmap v325: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:51 smithi028 ceph-mon[28247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:36:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:51 smithi028 ceph-mon[28247]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T01:36:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:51 smithi169 ceph-mon[32851]: pgmap v325: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:51 smithi169 ceph-mon[32851]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:36:51.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:51 smithi169 ceph-mon[32851]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T01:36:51.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:51 smithi053 ceph-mon[32593]: pgmap v325: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:51.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:51 smithi053 ceph-mon[32593]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:36:51.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:51 smithi053 ceph-mon[32593]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T01:36:52.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:52 smithi028 ceph-mon[28247]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:36:52.539 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:52 smithi169 ceph-mon[32851]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:36:52.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:52 smithi053 ceph-mon[32593]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:36:52.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.972+0000 7fae1fa00700 1 -- 172.21.15.28:0/3309315032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae181034b0 msgr2=0x7fae1810f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:52.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.972+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3309315032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae181034b0 0x7fae1810f970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fae10009f90 tx=0x7fae1000b330 comp rx=0 tx=0).stop 2024-04-26T01:36:52.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.973+0000 7fae1fa00700 1 -- 172.21.15.28:0/3309315032 shutdown_connections 2024-04-26T01:36:52.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.973+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3309315032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae181034b0 0x7fae1810f970 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:52.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.973+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3309315032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae18108aa0 0x7fae18108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:52.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.973+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3309315032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae18102af0 0x7fae18102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:52.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.973+0000 7fae1fa00700 1 -- 172.21.15.28:0/3309315032 >> 172.21.15.28:0/3309315032 conn(0x7fae18079b90 msgr2=0x7fae18079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:52.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.973+0000 7fae1fa00700 1 -- 172.21.15.28:0/3309315032 shutdown_connections 2024-04-26T01:36:52.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.973+0000 7fae1fa00700 1 -- 172.21.15.28:0/3309315032 wait complete. 2024-04-26T01:36:52.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.974+0000 7fae1fa00700 1 Processor -- start 2024-04-26T01:36:52.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.975+0000 7fae1fa00700 1 -- start start 2024-04-26T01:36:52.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.975+0000 7fae1fa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae18102af0 0x7fae18074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:52.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.975+0000 7fae1fa00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae181034b0 0x7fae18075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.975+0000 7fae1fa00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae18108aa0 0x7fae180757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.975+0000 7fae1fa00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae18112cf0 con 0x7fae18102af0 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.975+0000 7fae1fa00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fae18112b70 con 0x7fae18108aa0 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.975+0000 7fae1fa00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fae18112e70 con 0x7fae181034b0 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.975+0000 7fae1d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae18102af0 0x7fae18074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae0fe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae181034b0 0x7fae18075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae0fe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae181034b0 0x7fae18075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:50944/0 (socket says 172.21.15.28:50944) 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae1d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae18102af0 0x7fae18074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35594/0 (socket says 172.21.15.28:35594) 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae0fe00700 1 -- 172.21.15.28:0/3575875659 learned_addr learned my addr 172.21.15.28:0/3575875659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae1da00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae18108aa0 0x7fae180757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:52.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae1d000700 1 -- 172.21.15.28:0/3575875659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae181034b0 msgr2=0x7fae18075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:52.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae1d000700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae181034b0 0x7fae18075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae1d000700 1 -- 172.21.15.28:0/3575875659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae18108aa0 msgr2=0x7fae180757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae1d000700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae18108aa0 0x7fae180757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae1d000700 1 -- 172.21.15.28:0/3575875659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae181a7680 con 0x7fae18102af0 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.976+0000 7fae1da00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae18108aa0 0x7fae180757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.977+0000 7fae1d000700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae18102af0 0x7fae18074d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fae0800e9c0 tx=0x7fae0800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.977+0000 7fae16000700 1 -- 172.21.15.28:0/3575875659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0800c9c0 con 0x7fae18102af0 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.977+0000 7fae16000700 1 -- 172.21.15.28:0/3575875659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae0800cb60 con 0x7fae18102af0 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.977+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae181a7970 con 0x7fae18102af0 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.977+0000 7fae16000700 1 -- 172.21.15.28:0/3575875659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08018420 con 0x7fae18102af0 2024-04-26T01:36:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.977+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae181a7eb0 con 0x7fae18102af0 2024-04-26T01:36:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.978+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae181041e0 con 0x7fae18102af0 2024-04-26T01:36:52.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.981+0000 7fae16000700 1 -- 172.21.15.28:0/3575875659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fae08010430 con 0x7fae18102af0 2024-04-26T01:36:52.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.982+0000 7fae16000700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fadfc0a3c40 0x7fadfc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:52.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.982+0000 7fae16000700 1 -- 172.21.15.28:0/3575875659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7fae080d0660 con 0x7fae18102af0 2024-04-26T01:36:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.982+0000 7fae0fe00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fadfc0a3c40 0x7fadfc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.983+0000 7fae0fe00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fadfc0a3c40 0x7fadfc0a6100 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae1807a100 tx=0x7fae00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:52.985+0000 7fae16000700 1 -- 172.21.15.28:0/3575875659 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae0809d2d0 con 0x7fae18102af0 2024-04-26T01:36:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.222+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fae1802df80 con 0x7fae18102af0 2024-04-26T01:36:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.243+0000 7fae16000700 1 -- 172.21.15.28:0/3575875659 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) ==== 170+0+0 (secure 0 0 0) 0x7fae0809cbd0 con 0x7fae18102af0 2024-04-26T01:36:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.249+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fadfc0a3c40 msgr2=0x7fadfc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.249+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fadfc0a3c40 0x7fadfc0a6100 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae1807a100 tx=0x7fae00009260 comp rx=0 tx=0).stop 2024-04-26T01:36:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.249+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae18102af0 msgr2=0x7fae18074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.249+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae18102af0 0x7fae18074d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fae0800e9c0 tx=0x7fae0800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:36:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.250+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 shutdown_connections 2024-04-26T01:36:53.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.250+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fadfc0a3c40 0x7fadfc0a6100 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:53.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.250+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae18108aa0 0x7fae180757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:53.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.250+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae18102af0 0x7fae18074d40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:53.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.250+0000 7fae1fa00700 1 --2- 172.21.15.28:0/3575875659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae181034b0 0x7fae18075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:53.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.250+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 >> 172.21.15.28:0/3575875659 conn(0x7fae18079b90 msgr2=0x7fae18100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.252+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 shutdown_connections 2024-04-26T01:36:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:53.252+0000 7fae1fa00700 1 -- 172.21.15.28:0/3575875659 wait complete. 2024-04-26T01:36:53.251 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-04-26T01:36:53.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:53 smithi028 ceph-mon[28247]: pgmap v328: 33 pgs: 15 active+clean, 18 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:53.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:53 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3575875659' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:36:53.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:53 smithi169 ceph-mon[32851]: pgmap v328: 33 pgs: 15 active+clean, 18 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:53.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:53 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3575875659' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:36:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:53 smithi053 ceph-mon[32593]: pgmap v328: 33 pgs: 15 active+clean, 18 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:53.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:53 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3575875659' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:36:54.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T01:36:54.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:54 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3575875659' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:36:54.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:54 smithi028 ceph-mon[28247]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:36:54.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:54 smithi028 ceph-mon[28247]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:36:54.572 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:54.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:54 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3575875659' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:36:54.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:54 smithi169 ceph-mon[32851]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:36:54.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:54 smithi169 ceph-mon[32851]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:36:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:54 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3575875659' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:36:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:54 smithi053 ceph-mon[32593]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:36:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:54 smithi053 ceph-mon[32593]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:36:55.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:55 smithi028 ceph-mon[28247]: pgmap v330: 97 pgs: 2 creating+peering, 66 unknown, 29 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:55.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:55 smithi028 ceph-mon[28247]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:36:55.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:55 smithi169 ceph-mon[32851]: pgmap v330: 97 pgs: 2 creating+peering, 66 unknown, 29 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:55.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:55 smithi169 ceph-mon[32851]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:36:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:55 smithi053 ceph-mon[32593]: pgmap v330: 97 pgs: 2 creating+peering, 66 unknown, 29 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:55 smithi053 ceph-mon[32593]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 -- 172.21.15.28:0/1110053978 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5dc076f70 msgr2=0x7fb5dc077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 --2- 172.21.15.28:0/1110053978 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5dc076f70 0x7fb5dc077350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc0098f0 tx=0x7fb5cc009c00 comp rx=0 tx=0).stop 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 -- 172.21.15.28:0/1110053978 shutdown_connections 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 --2- 172.21.15.28:0/1110053978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc1138d0 0x7fb5dc115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 --2- 172.21.15.28:0/1110053978 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5dc076f70 0x7fb5dc077350 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 --2- 172.21.15.28:0/1110053978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5dc077890 0x7fb5dc113390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 -- 172.21.15.28:0/1110053978 >> 172.21.15.28:0/1110053978 conn(0x7fb5dc100720 msgr2=0x7fb5dc102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 -- 172.21.15.28:0/1110053978 shutdown_connections 2024-04-26T01:36:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.854+0000 7fb5e1400700 1 -- 172.21.15.28:0/1110053978 wait complete. 2024-04-26T01:36:56.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.855+0000 7fb5e1400700 1 Processor -- start 2024-04-26T01:36:56.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.856+0000 7fb5e1400700 1 -- start start 2024-04-26T01:36:56.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.856+0000 7fb5e1400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5dc076f70 0x7fb5dc19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:56.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.856+0000 7fb5e1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc077890 0x7fb5dc19fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.856+0000 7fb5e1400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5dc1138d0 0x7fb5dc1a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.856+0000 7fb5e1400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc119240 con 0x7fb5dc077890 2024-04-26T01:36:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.856+0000 7fb5e1400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1190c0 con 0x7fb5dc076f70 2024-04-26T01:36:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.856+0000 7fb5e1400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1193c0 con 0x7fb5dc1138d0 2024-04-26T01:36:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.856+0000 7fb5da800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5dc076f70 0x7fb5dc19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5d9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc077890 0x7fb5dc19fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5da800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5dc076f70 0x7fb5dc19f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:60520/0 (socket says 172.21.15.28:60520) 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5da800700 1 -- 172.21.15.28:0/3680247963 learned_addr learned my addr 172.21.15.28:0/3680247963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5db200700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5dc1138d0 0x7fb5dc1a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5da800700 1 -- 172.21.15.28:0/3680247963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5dc1138d0 msgr2=0x7fb5dc1a4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5da800700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5dc1138d0 0x7fb5dc1a4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5da800700 1 -- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc077890 msgr2=0x7fb5dc19fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5da800700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc077890 0x7fb5dc19fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5da800700 1 -- 172.21.15.28:0/3680247963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5dc1a4750 con 0x7fb5dc076f70 2024-04-26T01:36:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.857+0000 7fb5da800700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5dc076f70 0x7fb5dc19f790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc00f990 tx=0x7fb5cc00f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.858+0000 7fb5d3400700 1 -- 172.21.15.28:0/3680247963 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc027070 con 0x7fb5dc076f70 2024-04-26T01:36:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.858+0000 7fb5d3400700 1 -- 172.21.15.28:0/3680247963 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5cc005570 con 0x7fb5dc076f70 2024-04-26T01:36:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.858+0000 7fb5d3400700 1 -- 172.21.15.28:0/3680247963 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc004930 con 0x7fb5dc076f70 2024-04-26T01:36:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.858+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5dc1a49e0 con 0x7fb5dc076f70 2024-04-26T01:36:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.858+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5dc1ac2c0 con 0x7fb5dc076f70 2024-04-26T01:36:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.859+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5dc069730 con 0x7fb5dc076f70 2024-04-26T01:36:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.861+0000 7fb5d3400700 1 -- 172.21.15.28:0/3680247963 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb5cc02a070 con 0x7fb5dc076f70 2024-04-26T01:36:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.865+0000 7fb5d3400700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb5c00a3c40 0x7fb5c00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.865+0000 7fb5d9e00700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb5c00a3c40 0x7fb5c00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.865+0000 7fb5d3400700 1 -- 172.21.15.28:0/3680247963 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7879+0+0 (secure 0 0 0) 0x7fb5cc0d0eb0 con 0x7fb5dc076f70 2024-04-26T01:36:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.866+0000 7fb5d9e00700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb5c00a3c40 0x7fb5c00a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc079b00 tx=0x7fb5c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:56.866+0000 7fb5d3400700 1 -- 172.21.15.28:0/3680247963 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5cc013030 con 0x7fb5dc076f70 2024-04-26T01:36:57.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.150+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb5dc02df80 con 0x7fb5dc076f70 2024-04-26T01:36:57.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.295+0000 7fb5d3400700 1 -- 172.21.15.28:0/3680247963 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fb5cc09db50 con 0x7fb5dc076f70 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.303+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb5c00a3c40 msgr2=0x7fb5c00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.303+0000 7fb5e1400700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb5c00a3c40 0x7fb5c00a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc079b00 tx=0x7fb5c8009260 comp rx=0 tx=0).stop 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.303+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5dc076f70 msgr2=0x7fb5dc19f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.303+0000 7fb5e1400700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5dc076f70 0x7fb5dc19f790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc00f990 tx=0x7fb5cc00f9c0 comp rx=0 tx=0).stop 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.304+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 shutdown_connections 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.304+0000 7fb5e1400700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb5c00a3c40 0x7fb5c00a6100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.304+0000 7fb5e1400700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5dc1138d0 0x7fb5dc1a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.304+0000 7fb5e1400700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5dc076f70 0x7fb5dc19f790 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.304+0000 7fb5e1400700 1 --2- 172.21.15.28:0/3680247963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc077890 0x7fb5dc19fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.304+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 >> 172.21.15.28:0/3680247963 conn(0x7fb5dc100720 msgr2=0x7fb5dc102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.305+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 shutdown_connections 2024-04-26T01:36:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:36:57.306+0000 7fb5e1400700 1 -- 172.21.15.28:0/3680247963 wait complete. 2024-04-26T01:36:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T01:36:57.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:57 smithi028 ceph-mon[28247]: pgmap v333: 97 pgs: 22 creating+peering, 22 unknown, 53 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:57.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:57 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3680247963' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:36:57.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:57 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:36:57.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:57 smithi028 ceph-mon[28247]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T01:36:57.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:57 smithi169 ceph-mon[32851]: pgmap v333: 97 pgs: 22 creating+peering, 22 unknown, 53 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:57.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:57 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3680247963' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:36:57.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:57 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:36:57.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:57 smithi169 ceph-mon[32851]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T01:36:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:57 smithi053 ceph-mon[32593]: pgmap v333: 97 pgs: 22 creating+peering, 22 unknown, 53 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:57 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3680247963' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:36:57.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:57 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:36:57.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:57 smithi053 ceph-mon[32593]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T01:36:58.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:36:58.571 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:58.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:58 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:36:58.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:58 smithi169 ceph-mon[32851]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:36:58.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:58 smithi169 ceph-mon[32851]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T01:36:58.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:58 smithi169 ceph-mon[32851]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T01:36:58.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:58 smithi169 ceph-mon[32851]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:36:58.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:58 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] up:active 2024-04-26T01:36:58.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:58 smithi169 ceph-mon[32851]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:36:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:58 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:36:58.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:58 smithi053 ceph-mon[32593]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:36:58.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:58 smithi053 ceph-mon[32593]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T01:36:58.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:58 smithi053 ceph-mon[32593]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T01:36:58.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:58 smithi053 ceph-mon[32593]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:36:58.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:58 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] up:active 2024-04-26T01:36:58.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:58 smithi053 ceph-mon[32593]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:36:58.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:58 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:36:58.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:58 smithi028 ceph-mon[28247]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:36:58.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:58 smithi028 ceph-mon[28247]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T01:36:58.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:58 smithi028 ceph-mon[28247]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T01:36:58.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:58 smithi028 ceph-mon[28247]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:36:58.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:58 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] up:active 2024-04-26T01:36:58.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:58 smithi028 ceph-mon[28247]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:36:59.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:59 smithi028 ceph-mon[28247]: pgmap v335: 97 pgs: 22 creating+peering, 2 unknown, 73 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:59.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:59 smithi028 ceph-mon[28247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:36:59.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:36:59 smithi028 ceph-mon[28247]: Cluster is now healthy 2024-04-26T01:36:59.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:59 smithi169 ceph-mon[32851]: pgmap v335: 97 pgs: 22 creating+peering, 2 unknown, 73 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:59.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:59 smithi169 ceph-mon[32851]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:36:59.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:36:59 smithi169 ceph-mon[32851]: Cluster is now healthy 2024-04-26T01:36:59.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:59 smithi053 ceph-mon[32593]: pgmap v335: 97 pgs: 22 creating+peering, 2 unknown, 73 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:36:59.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:59 smithi053 ceph-mon[32593]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:36:59.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:36:59 smithi053 ceph-mon[32593]: Cluster is now healthy 2024-04-26T01:37:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.888+0000 7f7fc6e00700 1 -- 172.21.15.28:0/897365248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fc0105880 msgr2=0x7f7fc01098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.888+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/897365248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fc0105880 0x7f7fc01098d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0099c0 tx=0x7f7fac009cd0 comp rx=0 tx=0).stop 2024-04-26T01:37:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.888+0000 7f7fc6e00700 1 -- 172.21.15.28:0/897365248 shutdown_connections 2024-04-26T01:37:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.888+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/897365248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0109fb0 0x7f7fc0111b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.888+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/897365248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc0104ed0 0x7f7fc01052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.888+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/897365248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fc0105880 0x7f7fc01098d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.889+0000 7f7fc6e00700 1 -- 172.21.15.28:0/897365248 >> 172.21.15.28:0/897365248 conn(0x7f7fc0100760 msgr2=0x7f7fc0102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.889+0000 7f7fc6e00700 1 -- 172.21.15.28:0/897365248 shutdown_connections 2024-04-26T01:37:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.889+0000 7f7fc6e00700 1 -- 172.21.15.28:0/897365248 wait complete. 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.890+0000 7f7fc6e00700 1 Processor -- start 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.890+0000 7f7fc6e00700 1 -- start start 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.890+0000 7f7fc6e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104ed0 0x7f7fc019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.890+0000 7f7fc6e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fc0105880 0x7f7fc019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.890+0000 7f7fc6e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc0109fb0 0x7f7fc019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.890+0000 7f7fc6e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0114e20 con 0x7f7fc0104ed0 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.890+0000 7f7fc6e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0114ca0 con 0x7f7fc0109fb0 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.890+0000 7f7fc6e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0114fa0 con 0x7f7fc0105880 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbfe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104ed0 0x7f7fc019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbfe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104ed0 0x7f7fc019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51676/0 (socket says 172.21.15.28:51676) 2024-04-26T01:37:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbfe00700 1 -- 172.21.15.28:0/853142539 learned_addr learned my addr 172.21.15.28:0/853142539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbf400700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fc0105880 0x7f7fc019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fc4e00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc0109fb0 0x7f7fc019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbfe00700 1 -- 172.21.15.28:0/853142539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fc0105880 msgr2=0x7f7fc019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbfe00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fc0105880 0x7f7fc019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbfe00700 1 -- 172.21.15.28:0/853142539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc0109fb0 msgr2=0x7f7fc019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbfe00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc0109fb0 0x7f7fc019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.891+0000 7f7fbfe00700 1 -- 172.21.15.28:0/853142539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc01a0350 con 0x7f7fc0104ed0 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.892+0000 7f7fbfe00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104ed0 0x7f7fc019b390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000eb30 tx=0x7f7fb000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.892+0000 7f7fbcc00700 1 -- 172.21.15.28:0/853142539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb000cbd0 con 0x7f7fc0104ed0 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.892+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc01a0640 con 0x7f7fc0104ed0 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.892+0000 7f7fbcc00700 1 -- 172.21.15.28:0/853142539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb00044d0 con 0x7f7fc0104ed0 2024-04-26T01:37:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.892+0000 7f7fbcc00700 1 -- 172.21.15.28:0/853142539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0003e30 con 0x7f7fc0104ed0 2024-04-26T01:37:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.892+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc01a7ec0 con 0x7f7fc0104ed0 2024-04-26T01:37:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.893+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc0069730 con 0x7f7fc0104ed0 2024-04-26T01:37:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.895+0000 7f7fbcc00700 1 -- 172.21.15.28:0/853142539 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7fb0004000 con 0x7f7fc0104ed0 2024-04-26T01:37:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.897+0000 7f7fbcc00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7fa40a3cc0 0x7f7fa40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.897+0000 7f7fbcc00700 1 -- 172.21.15.28:0/853142539 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f7fb00d0800 con 0x7f7fc0104ed0 2024-04-26T01:37:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.897+0000 7f7fbf400700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7fa40a3cc0 0x7f7fa40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.898+0000 7f7fbf400700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7fa40a3cc0 0x7f7fa40a6180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7fac000c00 tx=0x7f7fac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:00.901+0000 7f7fbcc00700 1 -- 172.21.15.28:0/853142539 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7fb0053ff0 con 0x7f7fc0104ed0 2024-04-26T01:37:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.133+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7fc002df80 con 0x7f7fc0104ed0 2024-04-26T01:37:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.137+0000 7f7fbcc00700 1 -- 172.21.15.28:0/853142539 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22195 (secure 0 0 0) 0x7f7fb00042f0 con 0x7f7fc0104ed0 2024-04-26T01:37:01.134 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:37:01.134 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:36:57.152710+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:37:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7fa40a3cc0 msgr2=0x7f7fa40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7fa40a3cc0 0x7f7fa40a6180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7fac000c00 tx=0x7f7fac005fb0 comp rx=0 tx=0).stop 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104ed0 msgr2=0x7f7fc019b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104ed0 0x7f7fc019b390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000eb30 tx=0x7f7fb000ee40 comp rx=0 tx=0).stop 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 shutdown_connections 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7fa40a3cc0 0x7f7fa40a6180 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fc0109fb0 0x7f7fc019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104ed0 0x7f7fc019b390 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 --2- 172.21.15.28:0/853142539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fc0105880 0x7f7fc019b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 >> 172.21.15.28:0/853142539 conn(0x7f7fc0100760 msgr2=0x7f7fc0102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 shutdown_connections 2024-04-26T01:37:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:01.140+0000 7f7fc6e00700 1 -- 172.21.15.28:0/853142539 wait complete. 2024-04-26T01:37:01.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:01 smithi028 ceph-mon[28247]: pgmap v336: 97 pgs: 97 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-26T01:37:01.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:01 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/853142539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:01.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:01 smithi169 ceph-mon[32851]: pgmap v336: 97 pgs: 97 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-26T01:37:01.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:01 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/853142539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:01 smithi053 ceph-mon[32593]: pgmap v336: 97 pgs: 97 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-26T01:37:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:01 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/853142539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:01.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:37:02.415 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:02.909 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:37:02 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: 2024-04-26T01:37:02.461+0000 7f37e472c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T01:37:03.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:03 smithi028 ceph-mon[28247]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 4 op/s 2024-04-26T01:37:03.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:03 smithi169 ceph-mon[32851]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 4 op/s 2024-04-26T01:37:03.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:03 smithi053 ceph-mon[32593]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 4 op/s 2024-04-26T01:37:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 -- 172.21.15.28:0/3465102216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19b006b0f0 msgr2=0x7f19b010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 --2- 172.21.15.28:0/3465102216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19b006b0f0 0x7f19b010f900 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f19a800b0a0 tx=0x7f19a800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 -- 172.21.15.28:0/3465102216 shutdown_connections 2024-04-26T01:37:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 --2- 172.21.15.28:0/3465102216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19b006b0f0 0x7f19b010f900 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 --2- 172.21.15.28:0/3465102216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b0108a30 0x7f19b0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 --2- 172.21.15.28:0/3465102216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19b006a750 0x7f19b006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 -- 172.21.15.28:0/3465102216 >> 172.21.15.28:0/3465102216 conn(0x7f19b00788f0 msgr2=0x7f19b0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 -- 172.21.15.28:0/3465102216 shutdown_connections 2024-04-26T01:37:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.535+0000 7f19b7a00700 1 -- 172.21.15.28:0/3465102216 wait complete. 2024-04-26T01:37:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.536+0000 7f19b7a00700 1 Processor -- start 2024-04-26T01:37:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b7a00700 1 -- start start 2024-04-26T01:37:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b7a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b006a750 0x7f19b0199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b7a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19b006b0f0 0x7f19b0199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b7a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19b0108a30 0x7f19b019d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b7a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19b0112bd0 con 0x7f19b0108a30 2024-04-26T01:37:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b7a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f19b0112a50 con 0x7f19b006b0f0 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b7a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f19b0112d50 con 0x7f19b006a750 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b5000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b006a750 0x7f19b0199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b5000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b006a750 0x7f19b0199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51942/0 (socket says 172.21.15.28:51942) 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b5000700 1 -- 172.21.15.28:0/2298164770 learned_addr learned my addr 172.21.15.28:0/2298164770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19afe00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19b006b0f0 0x7f19b0199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.537+0000 7f19b5a00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19b0108a30 0x7f19b019d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19b5000700 1 -- 172.21.15.28:0/2298164770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19b006b0f0 msgr2=0x7f19b0199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19b5000700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19b006b0f0 0x7f19b0199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19b5000700 1 -- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19b0108a30 msgr2=0x7f19b019d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19b5000700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19b0108a30 0x7f19b019d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19b5000700 1 -- 172.21.15.28:0/2298164770 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b019e100 con 0x7f19b006a750 2024-04-26T01:37:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19b5000700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b006a750 0x7f19b0199140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f19a400e9c0 tx=0x7f19a400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:04.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19ad600700 1 -- 172.21.15.28:0/2298164770 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a400cbd0 con 0x7f19b006a750 2024-04-26T01:37:04.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19ad600700 1 -- 172.21.15.28:0/2298164770 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19a400ee60 con 0x7f19b006a750 2024-04-26T01:37:04.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b019e3f0 con 0x7f19b006a750 2024-04-26T01:37:04.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.538+0000 7f19ad600700 1 -- 172.21.15.28:0/2298164770 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a40055c0 con 0x7f19b006a750 2024-04-26T01:37:04.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.539+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b01a5c70 con 0x7f19b006a750 2024-04-26T01:37:04.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.540+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19b0069730 con 0x7f19b006a750 2024-04-26T01:37:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.542+0000 7f19ad600700 1 -- 172.21.15.28:0/2298164770 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f19a4004750 con 0x7f19b006a750 2024-04-26T01:37:04.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.543+0000 7f19ad600700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f19980a3c40 0x7f19980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:04.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.543+0000 7f19ad600700 1 -- 172.21.15.28:0/2298164770 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f19a40d0530 con 0x7f19b006a750 2024-04-26T01:37:04.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.543+0000 7f19afe00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f19980a3c40 0x7f19980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:04.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.544+0000 7f19afe00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f19980a3c40 0x7f19980a6100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f19b0078ed0 tx=0x7f199c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:04.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.546+0000 7f19ad600700 1 -- 172.21.15.28:0/2298164770 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19a409cfc0 con 0x7f19b006a750 2024-04-26T01:37:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.778+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19b002df80 con 0x7f19b006a750 2024-04-26T01:37:04.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.780+0000 7f19ad600700 1 -- 172.21.15.28:0/2298164770 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22195 (secure 0 0 0) 0x7f19a409c8c0 con 0x7f19b006a750 2024-04-26T01:37:04.777 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:37:04.777 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:36:57.152710+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:37:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f19980a3c40 msgr2=0x7f19980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f19980a3c40 0x7f19980a6100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f19b0078ed0 tx=0x7f199c0073d0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b006a750 msgr2=0x7f19b0199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b006a750 0x7f19b0199140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f19a400e9c0 tx=0x7f19a400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 shutdown_connections 2024-04-26T01:37:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f19980a3c40 0x7f19980a6100 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19b0108a30 0x7f19b019d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b006a750 0x7f19b0199140 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 --2- 172.21.15.28:0/2298164770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19b006b0f0 0x7f19b0199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.784+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 >> 172.21.15.28:0/2298164770 conn(0x7f19b00788f0 msgr2=0x7f19b00ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.785+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 shutdown_connections 2024-04-26T01:37:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:04.785+0000 7f19b7a00700 1 -- 172.21.15.28:0/2298164770 wait complete. 2024-04-26T01:37:05.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:05 smithi028 ceph-mon[28247]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:37:05.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:05 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2298164770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:05.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-04-26T01:37:05.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:05 smithi169 ceph-mon[32851]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:37:05.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:05 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2298164770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:05 smithi053 ceph-mon[32593]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:37:05.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:05 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2298164770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:06.092 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:07 smithi028 ceph-mon[28247]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:37:07.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:07 smithi169 ceph-mon[32851]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:37:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:07 smithi053 ceph-mon[32593]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:37:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 -- 172.21.15.28:0/4111014112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1048109c80 msgr2=0x7f104810dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 --2- 172.21.15.28:0/4111014112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1048109c80 0x7f104810dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f10300099c0 tx=0x7f1030009cd0 comp rx=0 tx=0).stop 2024-04-26T01:37:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 -- 172.21.15.28:0/4111014112 shutdown_connections 2024-04-26T01:37:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 --2- 172.21.15.28:0/4111014112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f104810e3b0 0x7f1048115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 --2- 172.21.15.28:0/4111014112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10481092d0 0x7f10481096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 --2- 172.21.15.28:0/4111014112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1048109c80 0x7f104810dcd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 -- 172.21.15.28:0/4111014112 >> 172.21.15.28:0/4111014112 conn(0x7f1048076210 msgr2=0x7f1048107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 -- 172.21.15.28:0/4111014112 shutdown_connections 2024-04-26T01:37:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.307+0000 7f104f600700 1 -- 172.21.15.28:0/4111014112 wait complete. 2024-04-26T01:37:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.308+0000 7f104f600700 1 Processor -- start 2024-04-26T01:37:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.308+0000 7f104f600700 1 -- start start 2024-04-26T01:37:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104f600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10481092d0 0x7f104819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104f600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1048109c80 0x7f104819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104f600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f104810e3b0 0x7f10481a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104f600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1048119220 con 0x7f10481092d0 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104f600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f10481190a0 con 0x7f1048109c80 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104f600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f10481193a0 con 0x7f104810e3b0 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104cc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10481092d0 0x7f104819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104cc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10481092d0 0x7f104819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51722/0 (socket says 172.21.15.28:51722) 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104cc00700 1 -- 172.21.15.28:0/718182676 learned_addr learned my addr 172.21.15.28:0/718182676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f103fe00700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1048109c80 0x7f104819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.309+0000 7f104d600700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f104810e3b0 0x7f10481a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f103fe00700 1 -- 172.21.15.28:0/718182676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f104810e3b0 msgr2=0x7f10481a4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f103fe00700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f104810e3b0 0x7f10481a4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f103fe00700 1 -- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10481092d0 msgr2=0x7f104819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f103fe00700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10481092d0 0x7f104819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f103fe00700 1 -- 172.21.15.28:0/718182676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10481a4750 con 0x7f1048109c80 2024-04-26T01:37:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f103fe00700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1048109c80 0x7f104819fcd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f103000b580 tx=0x7f1030005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f1046000700 1 -- 172.21.15.28:0/718182676 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030026070 con 0x7f1048109c80 2024-04-26T01:37:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f1046000700 1 -- 172.21.15.28:0/718182676 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1030003c70 con 0x7f1048109c80 2024-04-26T01:37:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.310+0000 7f1046000700 1 -- 172.21.15.28:0/718182676 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103000f930 con 0x7f1048109c80 2024-04-26T01:37:08.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.311+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10481a49e0 con 0x7f1048109c80 2024-04-26T01:37:08.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.311+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10481ac2c0 con 0x7f1048109c80 2024-04-26T01:37:08.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.312+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104810a7b0 con 0x7f1048109c80 2024-04-26T01:37:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.314+0000 7f1046000700 1 -- 172.21.15.28:0/718182676 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1030029070 con 0x7f1048109c80 2024-04-26T01:37:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.315+0000 7f1046000700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f102c0a3cc0 0x7f102c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.316+0000 7f1046000700 1 -- 172.21.15.28:0/718182676 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f10300d0a30 con 0x7f1048109c80 2024-04-26T01:37:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.316+0000 7f104cc00700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f102c0a3cc0 0x7f102c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.316+0000 7f104cc00700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f102c0a3cc0 0x7f102c0a6180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1038009d20 tx=0x7f1038009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:08.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.318+0000 7f1046000700 1 -- 172.21.15.28:0/718182676 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1030013070 con 0x7f1048109c80 2024-04-26T01:37:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:08.598+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f10481a0aa0 con 0x7f1048109c80 2024-04-26T01:37:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.350+0000 7f1046000700 1 -- 172.21.15.28:0/718182676 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f103009d550 con 0x7f1048109c80 2024-04-26T01:37:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f102c0a3cc0 msgr2=0x7f102c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f102c0a3cc0 0x7f102c0a6180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1038009d20 tx=0x7f1038009260 comp rx=0 tx=0).stop 2024-04-26T01:37:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1048109c80 msgr2=0x7f104819fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1048109c80 0x7f104819fcd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f103000b580 tx=0x7f1030005280 comp rx=0 tx=0).stop 2024-04-26T01:37:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 shutdown_connections 2024-04-26T01:37:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f102c0a3cc0 0x7f102c0a6180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f104810e3b0 0x7f10481a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10481092d0 0x7f104819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 --2- 172.21.15.28:0/718182676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1048109c80 0x7f104819fcd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 >> 172.21.15.28:0/718182676 conn(0x7f1048076210 msgr2=0x7f104810f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:09.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.355+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 shutdown_connections 2024-04-26T01:37:09.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:09.356+0000 7f104f600700 1 -- 172.21.15.28:0/718182676 wait complete. 2024-04-26T01:37:09.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:09 smithi169 ceph-mon[32851]: pgmap v340: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:37:09.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:09 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/718182676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:37:09.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:09 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:37:09.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:09 smithi053 ceph-mon[32593]: pgmap v340: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:37:09.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:09 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/718182676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:37:09.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:09 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:37:09.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:09 smithi028 ceph-mon[28247]: pgmap v340: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:37:09.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:09 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/718182676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:37:09.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:09 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:37:10.225 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-26T01:37:10.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-04-26T01:37:10.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:10 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:37:10.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:10 smithi028 ceph-mon[28247]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:37:10.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:10 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:37:10.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:10 smithi169 ceph-mon[32851]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:37:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:10.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:10 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:37:10.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:10 smithi053 ceph-mon[32593]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:37:11.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:11 smithi169 ceph-mon[32851]: pgmap v341: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-26T01:37:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:11 smithi053 ceph-mon[32593]: pgmap v341: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-26T01:37:11.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:11 smithi028 ceph-mon[28247]: pgmap v341: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-26T01:37:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 -- 172.21.15.28:0/1967727502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3230109c80 msgr2=0x7f323010dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 --2- 172.21.15.28:0/1967727502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3230109c80 0x7f323010dcd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f321c0099c0 tx=0x7f321c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:37:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 -- 172.21.15.28:0/1967727502 shutdown_connections 2024-04-26T01:37:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 --2- 172.21.15.28:0/1967727502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323010e3b0 0x7f3230115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 --2- 172.21.15.28:0/1967727502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32301092d0 0x7f32301096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 --2- 172.21.15.28:0/1967727502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3230109c80 0x7f323010dcd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 -- 172.21.15.28:0/1967727502 >> 172.21.15.28:0/1967727502 conn(0x7f3230076210 msgr2=0x7f3230107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 -- 172.21.15.28:0/1967727502 shutdown_connections 2024-04-26T01:37:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.994+0000 7f3237800700 1 -- 172.21.15.28:0/1967727502 wait complete. 2024-04-26T01:37:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.995+0000 7f3237800700 1 Processor -- start 2024-04-26T01:37:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.995+0000 7f3237800700 1 -- start start 2024-04-26T01:37:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3237800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32301092d0 0x7f323019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3237800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3230109c80 0x7f323019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3237800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323010e3b0 0x7f32301a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3237800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3230119270 con 0x7f323010e3b0 2024-04-26T01:37:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3237800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f32301190f0 con 0x7f32301092d0 2024-04-26T01:37:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3237800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f32301193f0 con 0x7f3230109c80 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3234e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32301092d0 0x7f323019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3234e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32301092d0 0x7f323019f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57412/0 (socket says 172.21.15.28:57412) 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3234e00700 1 -- 172.21.15.28:0/1917308429 learned_addr learned my addr 172.21.15.28:0/1917308429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f3235800700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323010e3b0 0x7f32301a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.996+0000 7f322fe00700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3230109c80 0x7f323019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.997+0000 7f3234e00700 1 -- 172.21.15.28:0/1917308429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3230109c80 msgr2=0x7f323019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.997+0000 7f3234e00700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3230109c80 0x7f323019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.997+0000 7f3234e00700 1 -- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323010e3b0 msgr2=0x7f32301a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.997+0000 7f3234e00700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323010e3b0 0x7f32301a3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.997+0000 7f3234e00700 1 -- 172.21.15.28:0/1917308429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32301a46e0 con 0x7f32301092d0 2024-04-26T01:37:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.997+0000 7f3234e00700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32301092d0 0x7f323019f720 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f322400ef10 tx=0x7f322400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.998+0000 7f322d600700 1 -- 172.21.15.28:0/1917308429 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322400c9c0 con 0x7f32301092d0 2024-04-26T01:37:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.998+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32301a4970 con 0x7f32301092d0 2024-04-26T01:37:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.998+0000 7f322d600700 1 -- 172.21.15.28:0/1917308429 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3224019070 con 0x7f32301092d0 2024-04-26T01:37:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.998+0000 7f322d600700 1 -- 172.21.15.28:0/1917308429 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224014480 con 0x7f32301092d0 2024-04-26T01:37:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.998+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32301ac250 con 0x7f32301092d0 2024-04-26T01:37:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:12.999+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3230069730 con 0x7f32301092d0 2024-04-26T01:37:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.001+0000 7f322d600700 1 -- 172.21.15.28:0/1917308429 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3224012050 con 0x7f32301092d0 2024-04-26T01:37:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.002+0000 7f322d600700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f32180a3c40 0x7f32180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.003+0000 7f322d600700 1 -- 172.21.15.28:0/1917308429 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f32240d0440 con 0x7f32301092d0 2024-04-26T01:37:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.003+0000 7f322fe00700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f32180a3c40 0x7f32180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.003+0000 7f322fe00700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f32180a3c40 0x7f32180a6100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f321c000c00 tx=0x7f321c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.005+0000 7f322d600700 1 -- 172.21.15.28:0/1917308429 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3224010040 con 0x7f32301092d0 2024-04-26T01:37:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.288+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f323002df80 con 0x7f32301092d0 2024-04-26T01:37:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.368+0000 7f322d600700 1 -- 172.21.15.28:0/1917308429 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7f322409ced0 con 0x7f32301092d0 2024-04-26T01:37:13.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.373+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f32180a3c40 msgr2=0x7f32180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:13.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.373+0000 7f3237800700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f32180a3c40 0x7f32180a6100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f321c000c00 tx=0x7f321c005fb0 comp rx=0 tx=0).stop 2024-04-26T01:37:13.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.373+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32301092d0 msgr2=0x7f323019f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:13.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.373+0000 7f3237800700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32301092d0 0x7f323019f720 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f322400ef10 tx=0x7f322400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:37:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.374+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 shutdown_connections 2024-04-26T01:37:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.374+0000 7f3237800700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f32180a3c40 0x7f32180a6100 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.374+0000 7f3237800700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323010e3b0 0x7f32301a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.374+0000 7f3237800700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32301092d0 0x7f323019f720 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.374+0000 7f3237800700 1 --2- 172.21.15.28:0/1917308429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3230109c80 0x7f323019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.374+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 >> 172.21.15.28:0/1917308429 conn(0x7f3230076210 msgr2=0x7f3230077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.374+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 shutdown_connections 2024-04-26T01:37:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:13.374+0000 7f3237800700 1 -- 172.21.15.28:0/1917308429 wait complete. 2024-04-26T01:37:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:13 smithi169 ceph-mon[32851]: pgmap v342: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-04-26T01:37:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:13 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1917308429' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:37:13.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:13 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:37:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:13 smithi053 ceph-mon[32593]: pgmap v342: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-04-26T01:37:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:13 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1917308429' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:37:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:13 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:37:13.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:13 smithi028 ceph-mon[28247]: pgmap v342: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-04-26T01:37:13.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:13 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1917308429' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:37:13.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:13 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:37:14.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-04-26T01:37:14.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:14 smithi028 ceph-mon[28247]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:37:14.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:14 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:37:14.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:14 smithi028 ceph-mon[28247]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:37:14.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:14 smithi028 ceph-mon[28247]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:37:14.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:14 smithi028 ceph-mon[28247]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-26T01:37:14.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:14 smithi028 ceph-mon[28247]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-26T01:37:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:14 smithi169 ceph-mon[32851]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:37:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:14 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:37:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:14 smithi169 ceph-mon[32851]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:37:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:14 smithi169 ceph-mon[32851]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:37:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:14 smithi169 ceph-mon[32851]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-26T01:37:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:14 smithi169 ceph-mon[32851]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-26T01:37:14.677 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:14 smithi053 ceph-mon[32593]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:14 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:14 smithi053 ceph-mon[32593]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:14 smithi053 ceph-mon[32593]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:14 smithi053 ceph-mon[32593]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-26T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:14 smithi053 ceph-mon[32593]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-26T01:37:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:15 smithi028 ceph-mon[28247]: pgmap v343: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-26T01:37:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:15 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] up:active 2024-04-26T01:37:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:15 smithi028 ceph-mon[28247]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:37:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:15 smithi028 ceph-mon[28247]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:37:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:15 smithi028 ceph-mon[28247]: Cluster is now healthy 2024-04-26T01:37:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:15 smithi028 ceph-mon[28247]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-26T01:37:15.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:15 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-26T01:37:15.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:15 smithi028 ceph-mon[28247]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-26T01:37:15.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:15 smithi169 ceph-mon[32851]: pgmap v343: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-26T01:37:15.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:15 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] up:active 2024-04-26T01:37:15.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:15 smithi169 ceph-mon[32851]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:37:15.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:15 smithi169 ceph-mon[32851]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:37:15.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:15 smithi169 ceph-mon[32851]: Cluster is now healthy 2024-04-26T01:37:15.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:15 smithi169 ceph-mon[32851]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-26T01:37:15.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:15 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-26T01:37:15.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:15 smithi169 ceph-mon[32851]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-26T01:37:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:15 smithi053 ceph-mon[32593]: pgmap v343: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-26T01:37:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:15 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] up:active 2024-04-26T01:37:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:15 smithi053 ceph-mon[32593]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:37:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:15 smithi053 ceph-mon[32593]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:37:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:15 smithi053 ceph-mon[32593]: Cluster is now healthy 2024-04-26T01:37:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:15 smithi053 ceph-mon[32593]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-26T01:37:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:15 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-26T01:37:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:15 smithi053 ceph-mon[32593]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-26T01:37:16.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:16 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] up:active 2024-04-26T01:37:16.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:16 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:37:16.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:16 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] up:active 2024-04-26T01:37:16.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:16 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:37:16.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:16 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] up:active 2024-04-26T01:37:16.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:16 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:37:16.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 -- 172.21.15.28:0/3477305670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f485410f440 msgr2=0x7f4854111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:16.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 --2- 172.21.15.28:0/3477305670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f485410f440 0x7f4854111830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f484c00b0a0 tx=0x7f484c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:37:16.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 -- 172.21.15.28:0/3477305670 shutdown_connections 2024-04-26T01:37:16.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 --2- 172.21.15.28:0/3477305670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f485410f440 0x7f4854111830 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:16.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 --2- 172.21.15.28:0/3477305670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4854074c70 0x7f4854075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:16.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 --2- 172.21.15.28:0/3477305670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4854075590 0x7f485410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:16.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 -- 172.21.15.28:0/3477305670 >> 172.21.15.28:0/3477305670 conn(0x7f4854079ba0 msgr2=0x7f4854079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:16.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 -- 172.21.15.28:0/3477305670 shutdown_connections 2024-04-26T01:37:16.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.962+0000 7f485a400700 1 -- 172.21.15.28:0/3477305670 wait complete. 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.963+0000 7f485a400700 1 Processor -- start 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.963+0000 7f485a400700 1 -- start start 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.963+0000 7f485a400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4854074c70 0x7f485419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.963+0000 7f485a400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4854075590 0x7f485419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f485a400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485410f440 0x7f485419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f485a400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4854114e20 con 0x7f485410f440 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f485a400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4854114ca0 con 0x7f4854074c70 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f485a400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4854114fa0 con 0x7f4854075590 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485410f440 0x7f485419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485410f440 0x7f485419fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43226/0 (socket says 172.21.15.28:43226) 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853e00700 1 -- 172.21.15.28:0/2255642335 learned_addr learned my addr 172.21.15.28:0/2255642335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f484aa00700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4854075590 0x7f485419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4854074c70 0x7f485419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853e00700 1 -- 172.21.15.28:0/2255642335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4854075590 msgr2=0x7f485419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853e00700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4854075590 0x7f485419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853e00700 1 -- 172.21.15.28:0/2255642335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4854074c70 msgr2=0x7f485419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853e00700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4854074c70 0x7f485419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.964+0000 7f4853e00700 1 -- 172.21.15.28:0/2255642335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48541a0270 con 0x7f485410f440 2024-04-26T01:37:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.965+0000 7f4853e00700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485410f440 0x7f485419fb30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f484c006070 tx=0x7f484c004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.965+0000 7f4850c00700 1 -- 172.21.15.28:0/2255642335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c021070 con 0x7f485410f440 2024-04-26T01:37:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.965+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48541a0500 con 0x7f485410f440 2024-04-26T01:37:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.965+0000 7f4850c00700 1 -- 172.21.15.28:0/2255642335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f484c007b80 con 0x7f485410f440 2024-04-26T01:37:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.965+0000 7f4850c00700 1 -- 172.21.15.28:0/2255642335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c0052a0 con 0x7f485410f440 2024-04-26T01:37:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.965+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48541a7de0 con 0x7f485410f440 2024-04-26T01:37:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.966+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4854069730 con 0x7f485410f440 2024-04-26T01:37:16.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.970+0000 7f4850c00700 1 -- 172.21.15.28:0/2255642335 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f484c024070 con 0x7f485410f440 2024-04-26T01:37:16.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.971+0000 7f4850c00700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48380a3c40 0x7f48380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:16.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.972+0000 7f4850c00700 1 -- 172.21.15.28:0/2255642335 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f484c00f020 con 0x7f485410f440 2024-04-26T01:37:16.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.972+0000 7f4853400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48380a3c40 0x7f48380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:16.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.972+0000 7f4853400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48380a3c40 0x7f48380a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4844005950 tx=0x7f484400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:16.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:16.975+0000 7f4850c00700 1 -- 172.21.15.28:0/2255642335 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f484c02b020 con 0x7f485410f440 2024-04-26T01:37:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.253+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f48541a7fb0 con 0x7f485410f440 2024-04-26T01:37:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.414+0000 7f4850c00700 1 -- 172.21.15.28:0/2255642335 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7f484c09e8d0 con 0x7f485410f440 2024-04-26T01:37:17.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.418+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48380a3c40 msgr2=0x7f48380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.418+0000 7f485a400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48380a3c40 0x7f48380a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4844005950 tx=0x7f484400b410 comp rx=0 tx=0).stop 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.418+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485410f440 msgr2=0x7f485419fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.418+0000 7f485a400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485410f440 0x7f485419fb30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f484c006070 tx=0x7f484c004cf0 comp rx=0 tx=0).stop 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.419+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 shutdown_connections 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.419+0000 7f485a400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f48380a3c40 0x7f48380a6100 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.419+0000 7f485a400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485410f440 0x7f485419fb30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.419+0000 7f485a400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4854074c70 0x7f485419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.419+0000 7f485a400700 1 --2- 172.21.15.28:0/2255642335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4854075590 0x7f485419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.419+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 >> 172.21.15.28:0/2255642335 conn(0x7f4854079ba0 msgr2=0x7f48540ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:17.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.419+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 shutdown_connections 2024-04-26T01:37:17.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:17.420+0000 7f485a400700 1 -- 172.21.15.28:0/2255642335 wait complete. 2024-04-26T01:37:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:17 smithi028 ceph-mon[28247]: pgmap v344: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 2 op/s 2024-04-26T01:37:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:17 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2255642335' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:37:17.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:17 smithi053 ceph-mon[32593]: pgmap v344: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 2 op/s 2024-04-26T01:37:17.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:17 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2255642335' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:37:17.909 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:37:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: 2024-04-26T01:37:17.462+0000 7f37e472c700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T01:37:17.909 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:37:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: 2024-04-26T01:37:17.462+0000 7f37e472c700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T01:37:17.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:17 smithi169 ceph-mon[32851]: pgmap v344: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 2 op/s 2024-04-26T01:37:17.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:17 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2255642335' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:37:18.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-04-26T01:37:18.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:18 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2255642335' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:37:18.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:18 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:37:18.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:18 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:37:18.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:18 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2255642335' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:37:18.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:18 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:37:18.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:18 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:37:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:18.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:18 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2255642335' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:37:18.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:18 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:37:18.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:18 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:37:19.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:19 smithi028 ceph-mon[28247]: pgmap v345: 97 pgs: 97 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:37:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:19 smithi053 ceph-mon[32593]: pgmap v345: 97 pgs: 97 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:37:19.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:19 smithi169 ceph-mon[32851]: pgmap v345: 97 pgs: 97 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:37:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 -- 172.21.15.28:0/4118887904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10075590 msgr2=0x7f2d1010ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 --2- 172.21.15.28:0/4118887904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10075590 0x7f2d1010ef00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80098f0 tx=0x7f2cf8009c00 comp rx=0 tx=0).stop 2024-04-26T01:37:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 -- 172.21.15.28:0/4118887904 shutdown_connections 2024-04-26T01:37:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 --2- 172.21.15.28:0/4118887904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1010f440 0x7f2d10111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 --2- 172.21.15.28:0/4118887904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d10074c70 0x7f2d10075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 --2- 172.21.15.28:0/4118887904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10075590 0x7f2d1010ef00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 -- 172.21.15.28:0/4118887904 >> 172.21.15.28:0/4118887904 conn(0x7f2d10079ba0 msgr2=0x7f2d10079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 -- 172.21.15.28:0/4118887904 shutdown_connections 2024-04-26T01:37:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.938+0000 7f2d15400700 1 -- 172.21.15.28:0/4118887904 wait complete. 2024-04-26T01:37:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.939+0000 7f2d15400700 1 Processor -- start 2024-04-26T01:37:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.939+0000 7f2d15400700 1 -- start start 2024-04-26T01:37:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d15400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10074c70 0x7f2d1019b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d15400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10075590 0x7f2d1019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d15400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d1010f440 0x7f2d1019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d15400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d10114e10 con 0x7f2d10075590 2024-04-26T01:37:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d15400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2d10114c90 con 0x7f2d10074c70 2024-04-26T01:37:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d15400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2d10114f90 con 0x7f2d1010f440 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d0e800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10074c70 0x7f2d1019b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d0e800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10074c70 0x7f2d1019b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:55576/0 (socket says 172.21.15.28:55576) 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d0e800700 1 -- 172.21.15.28:0/926152186 learned_addr learned my addr 172.21.15.28:0/926152186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d05e00700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10075590 0x7f2d1019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.940+0000 7f2d0f200700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d1010f440 0x7f2d1019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d0e800700 1 -- 172.21.15.28:0/926152186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d1010f440 msgr2=0x7f2d1019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d0e800700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d1010f440 0x7f2d1019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d0e800700 1 -- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10075590 msgr2=0x7f2d1019b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d0e800700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10075590 0x7f2d1019b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d0e800700 1 -- 172.21.15.28:0/926152186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d101a03a0 con 0x7f2d10074c70 2024-04-26T01:37:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d0e800700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10074c70 0x7f2d1019b3e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000e9f0 tx=0x7f2d0000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d07e00700 1 -- 172.21.15.28:0/926152186 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00004a30 con 0x7f2d10074c70 2024-04-26T01:37:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d07e00700 1 -- 172.21.15.28:0/926152186 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d0000cc70 con 0x7f2d10074c70 2024-04-26T01:37:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d101a0690 con 0x7f2d10074c70 2024-04-26T01:37:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.941+0000 7f2d07e00700 1 -- 172.21.15.28:0/926152186 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00005540 con 0x7f2d10074c70 2024-04-26T01:37:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.942+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d101a7f10 con 0x7f2d10074c70 2024-04-26T01:37:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.943+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d10069730 con 0x7f2d10074c70 2024-04-26T01:37:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.945+0000 7f2d07e00700 1 -- 172.21.15.28:0/926152186 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2d00004bd0 con 0x7f2d10074c70 2024-04-26T01:37:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.946+0000 7f2d07e00700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2cf40a3cc0 0x7f2cf40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.946+0000 7f2d07e00700 1 -- 172.21.15.28:0/926152186 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f2d000d03b0 con 0x7f2d10074c70 2024-04-26T01:37:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.947+0000 7f2d05e00700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2cf40a3cc0 0x7f2cf40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.947+0000 7f2d05e00700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2cf40a3cc0 0x7f2cf40a6180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8009810 tx=0x7f2cf8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:20.949+0000 7f2d07e00700 1 -- 172.21.15.28:0/926152186 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d0000ce20 con 0x7f2d10074c70 2024-04-26T01:37:21.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.235+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f2d1002e1a0 con 0x7f2d10074c70 2024-04-26T01:37:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.444+0000 7f2d07e00700 1 -- 172.21.15.28:0/926152186 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7f2d0009ce40 con 0x7f2d10074c70 2024-04-26T01:37:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2cf40a3cc0 msgr2=0x7f2cf40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2cf40a3cc0 0x7f2cf40a6180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8009810 tx=0x7f2cf8019040 comp rx=0 tx=0).stop 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10074c70 msgr2=0x7f2d1019b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10074c70 0x7f2d1019b3e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000e9f0 tx=0x7f2d0000ed00 comp rx=0 tx=0).stop 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 shutdown_connections 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2cf40a3cc0 0x7f2cf40a6180 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d1010f440 0x7f2d1019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d10074c70 0x7f2d1019b3e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 --2- 172.21.15.28:0/926152186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10075590 0x7f2d1019b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.450+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 >> 172.21.15.28:0/926152186 conn(0x7f2d10079ba0 msgr2=0x7f2d100ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.451+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 shutdown_connections 2024-04-26T01:37:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:21.451+0000 7f2d15400700 1 -- 172.21.15.28:0/926152186 wait complete. 2024-04-26T01:37:21.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:21 smithi053 ceph-mon[32593]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-26T01:37:21.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:21 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/926152186' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:37:21.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:21 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:37:21.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:21 smithi028 ceph-mon[28247]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-26T01:37:21.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:21 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/926152186' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:37:21.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:21 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:37:21.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:21 smithi169 ceph-mon[32851]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-26T01:37:21.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:21 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/926152186' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:37:21.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:21 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:37:22.507 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-26T01:37:22.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-04-26T01:37:22.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:22 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:37:22.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:22 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:37:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:22 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:37:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:22 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:37:22.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:22 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:37:22.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:22 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:37:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:23.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:23 smithi169 ceph-mon[32851]: pgmap v347: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T01:37:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:23 smithi053 ceph-mon[32593]: pgmap v347: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T01:37:23.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:23 smithi028 ceph-mon[28247]: pgmap v347: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T01:37:24.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T01:37:24.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T01:37:24.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [7, 11]}]: dispatch 2024-04-26T01:37:24.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2024-04-26T01:37:24.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-26T01:37:24.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T01:37:24.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-26T01:37:24.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:24 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T01:37:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T01:37:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T01:37:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [7, 11]}]: dispatch 2024-04-26T01:37:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2024-04-26T01:37:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-26T01:37:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T01:37:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-26T01:37:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:24 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T01:37:24.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T01:37:24.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T01:37:24.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [7, 11]}]: dispatch 2024-04-26T01:37:24.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2024-04-26T01:37:24.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-26T01:37:24.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T01:37:24.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-26T01:37:24.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:24 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 -- 172.21.15.28:0/862149837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 msgr2=0x7f229406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 --2- 172.21.15.28:0/862149837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 0x7f229406abb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2280009990 tx=0x7f2280009ca0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 -- 172.21.15.28:0/862149837 shutdown_connections 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 --2- 172.21.15.28:0/862149837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f229406b0f0 0x7f229410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 --2- 172.21.15.28:0/862149837 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2294108a30 0x7f2294108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 --2- 172.21.15.28:0/862149837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 0x7f229406abb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 -- 172.21.15.28:0/862149837 >> 172.21.15.28:0/862149837 conn(0x7f22940788f0 msgr2=0x7f2294078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 -- 172.21.15.28:0/862149837 shutdown_connections 2024-04-26T01:37:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.218+0000 7f229b600700 1 -- 172.21.15.28:0/862149837 wait complete. 2024-04-26T01:37:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.219+0000 7f229b600700 1 Processor -- start 2024-04-26T01:37:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.220+0000 7f229b600700 1 -- start start 2024-04-26T01:37:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.220+0000 7f229b600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 0x7f229419b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.220+0000 7f229b600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f229406b0f0 0x7f229419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.220+0000 7f229b600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2294108a30 0x7f229419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.220+0000 7f229b600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2294112c20 con 0x7f229406b0f0 2024-04-26T01:37:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.220+0000 7f229b600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2294112aa0 con 0x7f229406a750 2024-04-26T01:37:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.220+0000 7f229b600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2294112da0 con 0x7f2294108a30 2024-04-26T01:37:25.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f228be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f229406b0f0 0x7f229419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:25.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2298c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 0x7f229419b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:25.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2298c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 0x7f229419b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:55592/0 (socket says 172.21.15.28:55592) 2024-04-26T01:37:25.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2298c00700 1 -- 172.21.15.28:0/3363673401 learned_addr learned my addr 172.21.15.28:0/3363673401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:25.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2299600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2294108a30 0x7f229419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2299600700 1 -- 172.21.15.28:0/3363673401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 msgr2=0x7f229419b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2299600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 0x7f229419b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2299600700 1 -- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f229406b0f0 msgr2=0x7f229419b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2299600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f229406b0f0 0x7f229419b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2299600700 1 -- 172.21.15.28:0/3363673401 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22941a0420 con 0x7f2294108a30 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.221+0000 7f2299600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2294108a30 0x7f229419fce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f228c00efc0 tx=0x7f228c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.222+0000 7f2292000700 1 -- 172.21.15.28:0/3363673401 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c0092e0 con 0x7f2294108a30 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.222+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22941a06b0 con 0x7f2294108a30 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.222+0000 7f2292000700 1 -- 172.21.15.28:0/3363673401 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f228c004bb0 con 0x7f2294108a30 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.222+0000 7f2292000700 1 -- 172.21.15.28:0/3363673401 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c004020 con 0x7f2294108a30 2024-04-26T01:37:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.222+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22941a7f90 con 0x7f2294108a30 2024-04-26T01:37:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.223+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2294069730 con 0x7f2294108a30 2024-04-26T01:37:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.225+0000 7f2292000700 1 -- 172.21.15.28:0/3363673401 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f228c01b030 con 0x7f2294108a30 2024-04-26T01:37:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.227+0000 7f2292000700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f22700a3c40 0x7f22700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.227+0000 7f2292000700 1 -- 172.21.15.28:0/3363673401 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f228c0d0160 con 0x7f2294108a30 2024-04-26T01:37:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.227+0000 7f2298c00700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f22700a3c40 0x7f22700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.228+0000 7f2298c00700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f22700a3c40 0x7f22700a6100 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2278009ce0 tx=0x7f2278009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.229+0000 7f2292000700 1 -- 172.21.15.28:0/3363673401 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f228c022020 con 0x7f2294108a30 2024-04-26T01:37:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.506+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f229419c770 con 0x7f22700a3c40 2024-04-26T01:37:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.541+0000 7f2292000700 1 -- 172.21.15.28:0/3363673401 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f229419c770 con 0x7f22700a3c40 2024-04-26T01:37:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f22700a3c40 msgr2=0x7f22700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f22700a3c40 0x7f22700a6100 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2278009ce0 tx=0x7f2278009260 comp rx=0 tx=0).stop 2024-04-26T01:37:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2294108a30 msgr2=0x7f229419fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2294108a30 0x7f229419fce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f228c00efc0 tx=0x7f228c00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 shutdown_connections 2024-04-26T01:37:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f22700a3c40 0x7f22700a6100 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2294108a30 0x7f229419fce0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f229406a750 0x7f229419b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 --2- 172.21.15.28:0/3363673401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f229406b0f0 0x7f229419b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 >> 172.21.15.28:0/3363673401 conn(0x7f22940788f0 msgr2=0x7f229410dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 shutdown_connections 2024-04-26T01:37:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:25.544+0000 7f229b600700 1 -- 172.21.15.28:0/3363673401 wait complete. 2024-04-26T01:37:25.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: pgmap v348: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:37:25.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [7, 11]}]': finished 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-mon[28247]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:37:25.515+0000 7f1fec200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:37:25.515+0000 7f1fec200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:37:25.782 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:37:25.515+0000 7f1fec200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:37:25.783 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:37:25.515+0000 7f1fec200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:37:25.783 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:37:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:37:25.515+0000 7f1fec200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:37:25.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: pgmap v348: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:37:25.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T01:37:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T01:37:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [7, 11]}]': finished 2024-04-26T01:37:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2024-04-26T01:37:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-26T01:37:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T01:37:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-26T01:37:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T01:37:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:25 smithi169 ceph-mon[32851]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: pgmap v348: 97 pgs: 97 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [7, 11]}]': finished 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T01:37:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:25 smithi053 ceph-mon[32593]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:37:26.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T01:37:26.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:26 smithi028 ceph-mon[28247]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:37:26.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:26 smithi028 ceph-mon[28247]: from='client.24625 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:26.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:26 smithi169 ceph-mon[32851]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:37:26.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:26 smithi169 ceph-mon[32851]: from='client.24625 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:26.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:26 smithi053 ceph-mon[32593]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:37:26.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:26 smithi053 ceph-mon[32593]: from='client.24625 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:27.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:27 smithi028 ceph-mon[28247]: pgmap v351: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-26T01:37:27.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:27 smithi028 ceph-mon[28247]: mgrmap e17: x(active, since 9m), standbys: y, z 2024-04-26T01:37:27.869 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:27 smithi053 ceph-mon[32593]: pgmap v351: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-26T01:37:27.869 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:27 smithi053 ceph-mon[32593]: mgrmap e17: x(active, since 9m), standbys: y, z 2024-04-26T01:37:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:27 smithi169 ceph-mon[32851]: pgmap v351: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-26T01:37:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:27 smithi169 ceph-mon[32851]: mgrmap e17: x(active, since 9m), standbys: y, z 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 -- 172.21.15.28:0/3127826334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8108960 msgr2=0x7fc5d8108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 --2- 172.21.15.28:0/3127826334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8108960 0x7fc5d8108d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009990 tx=0x7fc5c8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 -- 172.21.15.28:0/3127826334 shutdown_connections 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 --2- 172.21.15.28:0/3127826334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8103320 0x7fc5d810f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 --2- 172.21.15.28:0/3127826334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8108960 0x7fc5d8108d40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 --2- 172.21.15.28:0/3127826334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d8102960 0x7fc5d8102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 -- 172.21.15.28:0/3127826334 >> 172.21.15.28:0/3127826334 conn(0x7fc5d80fe450 msgr2=0x7fc5d8100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 -- 172.21.15.28:0/3127826334 shutdown_connections 2024-04-26T01:37:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.040+0000 7fc5dda00700 1 -- 172.21.15.28:0/3127826334 wait complete. 2024-04-26T01:37:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.041+0000 7fc5dda00700 1 Processor -- start 2024-04-26T01:37:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.042+0000 7fc5dda00700 1 -- start start 2024-04-26T01:37:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.042+0000 7fc5dda00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d8102960 0x7fc5d819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.042+0000 7fc5dda00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8103320 0x7fc5d819b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.042+0000 7fc5dda00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8108960 0x7fc5d819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.042+0000 7fc5dda00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d8112b40 con 0x7fc5d8103320 2024-04-26T01:37:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.042+0000 7fc5dda00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d81129c0 con 0x7fc5d8108960 2024-04-26T01:37:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.042+0000 7fc5dda00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d8112cc0 con 0x7fc5d8102960 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d8102960 0x7fc5d819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8103320 0x7fc5d819b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d8102960 0x7fc5d819b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:49882/0 (socket says 172.21.15.28:49882) 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 -- 172.21.15.28:0/1203709834 learned_addr learned my addr 172.21.15.28:0/1203709834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5dd000700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8108960 0x7fc5d819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 -- 172.21.15.28:0/1203709834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8108960 msgr2=0x7fc5d819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8108960 0x7fc5d819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 -- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8103320 msgr2=0x7fc5d819b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8103320 0x7fc5d819b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 -- 172.21.15.28:0/1203709834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d81a01e0 con 0x7fc5d8102960 2024-04-26T01:37:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7400700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8103320 0x7fc5d819b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:37:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.043+0000 7fc5d7e00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d8102960 0x7fc5d819b1d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80057f0 tx=0x7fc5c800bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.044+0000 7fc5d4c00700 1 -- 172.21.15.28:0/1203709834 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8026070 con 0x7fc5d8102960 2024-04-26T01:37:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.044+0000 7fc5d4c00700 1 -- 172.21.15.28:0/1203709834 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5c8021410 con 0x7fc5d8102960 2024-04-26T01:37:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.044+0000 7fc5d4c00700 1 -- 172.21.15.28:0/1203709834 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8017650 con 0x7fc5d8102960 2024-04-26T01:37:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.044+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d81a0470 con 0x7fc5d8102960 2024-04-26T01:37:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.044+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d81a7d50 con 0x7fc5d8102960 2024-04-26T01:37:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.045+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d8069730 con 0x7fc5d8102960 2024-04-26T01:37:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.047+0000 7fc5d4c00700 1 -- 172.21.15.28:0/1203709834 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc5c8029070 con 0x7fc5d8102960 2024-04-26T01:37:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.048+0000 7fc5d4c00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc5bc0a3cd0 0x7fc5bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.049+0000 7fc5d4c00700 1 -- 172.21.15.28:0/1203709834 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fc5c80d11e0 con 0x7fc5d8102960 2024-04-26T01:37:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.049+0000 7fc5d7400700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc5bc0a3cd0 0x7fc5bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.049+0000 7fc5d7400700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc5bc0a3cd0 0x7fc5bc0a6190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc5d810fb20 tx=0x7fc5c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.053+0000 7fc5d4c00700 1 -- 172.21.15.28:0/1203709834 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5c805dff0 con 0x7fc5d8102960 2024-04-26T01:37:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.321+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc5d80ff9f0 con 0x7fc5bc0a3cd0 2024-04-26T01:37:29.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.366+0000 7fc5d4c00700 1 -- 172.21.15.28:0/1203709834 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc5d80ff9f0 con 0x7fc5bc0a3cd0 2024-04-26T01:37:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc5bc0a3cd0 msgr2=0x7fc5bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc5bc0a3cd0 0x7fc5bc0a6190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc5d810fb20 tx=0x7fc5c4009260 comp rx=0 tx=0).stop 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d8102960 msgr2=0x7fc5d819b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d8102960 0x7fc5d819b1d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80057f0 tx=0x7fc5c800bfd0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 shutdown_connections 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc5bc0a3cd0 0x7fc5bc0a6190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8108960 0x7fc5d819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d8102960 0x7fc5d819b1d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 --2- 172.21.15.28:0/1203709834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d8103320 0x7fc5d819b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 >> 172.21.15.28:0/1203709834 conn(0x7fc5d80fe450 msgr2=0x7fc5d810e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 shutdown_connections 2024-04-26T01:37:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:29.369+0000 7fc5dda00700 1 -- 172.21.15.28:0/1203709834 wait complete. 2024-04-26T01:37:29.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:29 smithi028 ceph-mon[28247]: pgmap v352: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 127 B/s wr, 6 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:29.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:29 smithi169 ceph-mon[32851]: pgmap v352: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 127 B/s wr, 6 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:29 smithi053 ceph-mon[32593]: pgmap v352: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 127 B/s wr, 6 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:30.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T01:37:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:30.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:30 smithi028 ceph-mon[28247]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:30.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:30 smithi169 ceph-mon[32851]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:30.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:30 smithi053 ceph-mon[32593]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:31.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:31 smithi028 ceph-mon[28247]: pgmap v353: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:31.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:31 smithi169 ceph-mon[32851]: pgmap v353: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:31 smithi053 ceph-mon[32593]: pgmap v353: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 -- 172.21.15.28:0/117341790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 msgr2=0x7fbd6410ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 --2- 172.21.15.28:0/117341790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 0x7fbd6410ef00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009990 tx=0x7fbd50009ca0 comp rx=0 tx=0).stop 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 -- 172.21.15.28:0/117341790 shutdown_connections 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 --2- 172.21.15.28:0/117341790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd6410f440 0x7fbd64111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 --2- 172.21.15.28:0/117341790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd64074c70 0x7fbd64075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 --2- 172.21.15.28:0/117341790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 0x7fbd6410ef00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 -- 172.21.15.28:0/117341790 >> 172.21.15.28:0/117341790 conn(0x7fbd64079ba0 msgr2=0x7fbd64079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 -- 172.21.15.28:0/117341790 shutdown_connections 2024-04-26T01:37:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.794+0000 7fbd69c00700 1 -- 172.21.15.28:0/117341790 wait complete. 2024-04-26T01:37:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.795+0000 7fbd69c00700 1 Processor -- start 2024-04-26T01:37:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.795+0000 7fbd69c00700 1 -- start start 2024-04-26T01:37:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.795+0000 7fbd69c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd64074c70 0x7fbd6419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.795+0000 7fbd69c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 0x7fbd6419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.795+0000 7fbd69c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd6410f440 0x7fbd6419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd69c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64114e50 con 0x7fbd64074c70 2024-04-26T01:37:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd69c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64114cd0 con 0x7fbd64075590 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd69c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64114fd0 con 0x7fbd6410f440 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd64074c70 0x7fbd6419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 0x7fbd6419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 0x7fbd6419b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48950/0 (socket says 172.21.15.28:48950) 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd64074c70 0x7fbd6419b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48474/0 (socket says 172.21.15.28:48474) 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62000700 1 -- 172.21.15.28:0/3957472448 learned_addr learned my addr 172.21.15.28:0/3957472448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd63400700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd6410f440 0x7fbd6419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62a00700 1 -- 172.21.15.28:0/3957472448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd6410f440 msgr2=0x7fbd6419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62a00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd6410f440 0x7fbd6419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62a00700 1 -- 172.21.15.28:0/3957472448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 msgr2=0x7fbd6419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.796+0000 7fbd62a00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 0x7fbd6419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.797+0000 7fbd62a00700 1 -- 172.21.15.28:0/3957472448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd641a0370 con 0x7fbd64074c70 2024-04-26T01:37:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.797+0000 7fbd62000700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 0x7fbd6419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:37:32.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.797+0000 7fbd62a00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd64074c70 0x7fbd6419b360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400e9f0 tx=0x7fbd5400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:32.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.797+0000 7fbd5b400700 1 -- 172.21.15.28:0/3957472448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5400c9c0 con 0x7fbd64074c70 2024-04-26T01:37:32.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.797+0000 7fbd5b400700 1 -- 172.21.15.28:0/3957472448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd5400cb60 con 0x7fbd64074c70 2024-04-26T01:37:32.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.797+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd641a0660 con 0x7fbd64074c70 2024-04-26T01:37:32.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.797+0000 7fbd5b400700 1 -- 172.21.15.28:0/3957472448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd54018680 con 0x7fbd64074c70 2024-04-26T01:37:32.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.797+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd641a7ee0 con 0x7fbd64074c70 2024-04-26T01:37:32.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.798+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd64069730 con 0x7fbd64074c70 2024-04-26T01:37:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.800+0000 7fbd5b400700 1 -- 172.21.15.28:0/3957472448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbd54010430 con 0x7fbd64074c70 2024-04-26T01:37:32.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.802+0000 7fbd5b400700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbd480a3cd0 0x7fbd480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:32.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.802+0000 7fbd5b400700 1 -- 172.21.15.28:0/3957472448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fbd540d0850 con 0x7fbd64074c70 2024-04-26T01:37:32.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.802+0000 7fbd62000700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbd480a3cd0 0x7fbd480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:32.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.803+0000 7fbd62000700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbd480a3cd0 0x7fbd480a6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd641111e0 tx=0x7fbd5000b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:32.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:32.807+0000 7fbd5b400700 1 -- 172.21.15.28:0/3957472448 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd5409d0e0 con 0x7fbd64074c70 2024-04-26T01:37:33.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.077+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fbd64062e80 con 0x7fbd480a3cd0 2024-04-26T01:37:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.090+0000 7fbd5b400700 1 -- 172.21.15.28:0/3957472448 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fbd64062e80 con 0x7fbd480a3cd0 2024-04-26T01:37:33.086 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/5f8a97e9-ac31-40c2-900e-ffc9a31a9af5 2024-04-26T01:37:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.092+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbd480a3cd0 msgr2=0x7fbd480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.092+0000 7fbd69c00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbd480a3cd0 0x7fbd480a6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd641111e0 tx=0x7fbd5000b280 comp rx=0 tx=0).stop 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.092+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd64074c70 msgr2=0x7fbd6419b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.092+0000 7fbd69c00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd64074c70 0x7fbd6419b360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400e9f0 tx=0x7fbd5400ed00 comp rx=0 tx=0).stop 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.093+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 shutdown_connections 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.093+0000 7fbd69c00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fbd480a3cd0 0x7fbd480a6190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.093+0000 7fbd69c00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd6410f440 0x7fbd6419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.093+0000 7fbd69c00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd64074c70 0x7fbd6419b360 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.093+0000 7fbd69c00700 1 --2- 172.21.15.28:0/3957472448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd64075590 0x7fbd6419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.093+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 >> 172.21.15.28:0/3957472448 conn(0x7fbd64079ba0 msgr2=0x7fbd640ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.093+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 shutdown_connections 2024-04-26T01:37:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:33.093+0000 7fbd69c00700 1 -- 172.21.15.28:0/3957472448 wait complete. 2024-04-26T01:37:33.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:33 smithi028 ceph-mon[28247]: pgmap v354: 97 pgs: 97 active+clean; 602 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:33.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:33 smithi169 ceph-mon[32851]: pgmap v354: 97 pgs: 97 active+clean; 602 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:33.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T01:37:33.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:33 smithi053 ceph-mon[32593]: pgmap v354: 97 pgs: 97 active+clean; 602 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 94 B/s, 1 objects/s recovering 2024-04-26T01:37:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:34.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:34 smithi028 ceph-mon[28247]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:34.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:34 smithi169 ceph-mon[32851]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:34.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:34 smithi053 ceph-mon[32593]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:35.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:35 smithi028 ceph-mon[28247]: pgmap v355: 97 pgs: 97 active+clean; 602 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.0 KiB/s wr, 7 op/s; 81 B/s, 1 objects/s recovering 2024-04-26T01:37:35.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:35 smithi169 ceph-mon[32851]: pgmap v355: 97 pgs: 97 active+clean; 602 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.0 KiB/s wr, 7 op/s; 81 B/s, 1 objects/s recovering 2024-04-26T01:37:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:35 smithi053 ceph-mon[32593]: pgmap v355: 97 pgs: 97 active+clean; 602 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.0 KiB/s wr, 7 op/s; 81 B/s, 1 objects/s recovering 2024-04-26T01:37:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.849+0000 7ff00ce00700 1 -- 172.21.15.28:0/1783678169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008101930 msgr2=0x7ff008101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.849+0000 7ff00ce00700 1 --2- 172.21.15.28:0/1783678169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008101930 0x7ff008101db0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feff4009990 tx=0x7feff4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.850+0000 7ff00ce00700 1 -- 172.21.15.28:0/1783678169 shutdown_connections 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.850+0000 7ff00ce00700 1 --2- 172.21.15.28:0/1783678169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0081022f0 0x7ff00810e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.850+0000 7ff00ce00700 1 --2- 172.21.15.28:0/1783678169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff008107980 0x7ff008107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.850+0000 7ff00ce00700 1 --2- 172.21.15.28:0/1783678169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008101930 0x7ff008101db0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.850+0000 7ff00ce00700 1 -- 172.21.15.28:0/1783678169 >> 172.21.15.28:0/1783678169 conn(0x7ff0080788b0 msgr2=0x7ff008078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.850+0000 7ff00ce00700 1 -- 172.21.15.28:0/1783678169 shutdown_connections 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.850+0000 7ff00ce00700 1 -- 172.21.15.28:0/1783678169 wait complete. 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.851+0000 7ff00ce00700 1 Processor -- start 2024-04-26T01:37:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.851+0000 7ff00ce00700 1 -- start start 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.851+0000 7ff00ce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff008101930 0x7ff0081034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff00ce00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0081022f0 0x7ff008103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff00ce00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008107980 0x7ff008103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff00ce00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff008111ad0 con 0x7ff008101930 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff00ce00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff008111950 con 0x7ff0081022f0 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff00ce00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff008111c50 con 0x7ff008107980 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff008101930 0x7ff0081034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff008101930 0x7ff0081034e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48500/0 (socket says 172.21.15.28:48500) 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005e00700 1 -- 172.21.15.28:0/2149668535 learned_addr learned my addr 172.21.15.28:0/2149668535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005400700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0081022f0 0x7ff008103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff006800700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008107980 0x7ff008103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:36.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005e00700 1 -- 172.21.15.28:0/2149668535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008107980 msgr2=0x7ff008103f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005e00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008107980 0x7ff008103f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005e00700 1 -- 172.21.15.28:0/2149668535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0081022f0 msgr2=0x7ff008103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005e00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0081022f0 0x7ff008103a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff005e00700 1 -- 172.21.15.28:0/2149668535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0081a7630 con 0x7ff008101930 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.852+0000 7ff006800700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008107980 0x7ff008103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.853+0000 7ff005e00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff008101930 0x7ff0081034e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feff800e9a0 tx=0x7feff800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.853+0000 7feffea00700 1 -- 172.21.15.28:0/2149668535 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff800cbd0 con 0x7ff008101930 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.853+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0081a7920 con 0x7ff008101930 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.853+0000 7feffea00700 1 -- 172.21.15.28:0/2149668535 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feff800cd70 con 0x7ff008101930 2024-04-26T01:37:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.853+0000 7feffea00700 1 -- 172.21.15.28:0/2149668535 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff8018680 con 0x7ff008101930 2024-04-26T01:37:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.853+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0081a7e60 con 0x7ff008101930 2024-04-26T01:37:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.855+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe8005330 con 0x7ff008101930 2024-04-26T01:37:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.858+0000 7feffea00700 1 -- 172.21.15.28:0/2149668535 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feff8018820 con 0x7ff008101930 2024-04-26T01:37:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.860+0000 7feffea00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fefec0a3cd0 0x7fefec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.860+0000 7feffea00700 1 -- 172.21.15.28:0/2149668535 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7feff80d06a0 con 0x7ff008101930 2024-04-26T01:37:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.860+0000 7ff005400700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fefec0a3cd0 0x7fefec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:36.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.861+0000 7ff005400700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fefec0a3cd0 0x7fefec0a6190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feff4006010 tx=0x7feff401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:36.865+0000 7feffea00700 1 -- 172.21.15.28:0/2149668535 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feff809ced0 con 0x7ff008101930 2024-04-26T01:37:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.156+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fefe8000bc0 con 0x7fefec0a3cd0 2024-04-26T01:37:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.193+0000 7feffea00700 1 -- 172.21.15.28:0/2149668535 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fefe8000bc0 con 0x7fefec0a3cd0 2024-04-26T01:37:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fefec0a3cd0 msgr2=0x7fefec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fefec0a3cd0 0x7fefec0a6190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feff4006010 tx=0x7feff401a040 comp rx=0 tx=0).stop 2024-04-26T01:37:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff008101930 msgr2=0x7ff0081034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff008101930 0x7ff0081034e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feff800e9a0 tx=0x7feff800ed60 comp rx=0 tx=0).stop 2024-04-26T01:37:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 shutdown_connections 2024-04-26T01:37:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fefec0a3cd0 0x7fefec0a6190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff008107980 0x7ff008103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff008101930 0x7ff0081034e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 --2- 172.21.15.28:0/2149668535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0081022f0 0x7ff008103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.196+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 >> 172.21.15.28:0/2149668535 conn(0x7ff0080788b0 msgr2=0x7ff008105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.197+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 shutdown_connections 2024-04-26T01:37:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:37.197+0000 7ff00ce00700 1 -- 172.21.15.28:0/2149668535 wait complete. 2024-04-26T01:37:37.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:37 smithi169 ceph-mon[32851]: pgmap v356: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.4 KiB/s wr, 7 op/s; 73 B/s, 1 objects/s recovering 2024-04-26T01:37:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:37 smithi053 ceph-mon[32593]: pgmap v356: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.4 KiB/s wr, 7 op/s; 73 B/s, 1 objects/s recovering 2024-04-26T01:37:38.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:37 smithi028 ceph-mon[28247]: pgmap v356: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.4 KiB/s wr, 7 op/s; 73 B/s, 1 objects/s recovering 2024-04-26T01:37:38.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T01:37:38.517 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:38.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:38 smithi028 ceph-mon[28247]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:38.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:38 smithi169 ceph-mon[32851]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:38 smithi053 ceph-mon[32593]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:39.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:39 smithi169 ceph-mon[32851]: pgmap v357: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 6 op/s; 62 B/s, 1 objects/s recovering 2024-04-26T01:37:39.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:39 smithi053 ceph-mon[32593]: pgmap v357: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 6 op/s; 62 B/s, 1 objects/s recovering 2024-04-26T01:37:40.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:39 smithi028 ceph-mon[28247]: pgmap v357: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 6 op/s; 62 B/s, 1 objects/s recovering 2024-04-26T01:37:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 -- 172.21.15.28:0/583582717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae4075680 msgr2=0x7f7ae410ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/583582717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae4075680 0x7f7ae410ecf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009990 tx=0x7f7acc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:37:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 -- 172.21.15.28:0/583582717 shutdown_connections 2024-04-26T01:37:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/583582717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae410f230 0x7f7ae4111620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/583582717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ae4074d60 0x7f7ae4075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/583582717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae4075680 0x7f7ae410ecf0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 -- 172.21.15.28:0/583582717 >> 172.21.15.28:0/583582717 conn(0x7f7ae40fe4c0 msgr2=0x7f7ae41008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 -- 172.21.15.28:0/583582717 shutdown_connections 2024-04-26T01:37:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.969+0000 7f7ae8e00700 1 -- 172.21.15.28:0/583582717 wait complete. 2024-04-26T01:37:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.971+0000 7f7ae8e00700 1 Processor -- start 2024-04-26T01:37:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.971+0000 7f7ae8e00700 1 -- start start 2024-04-26T01:37:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.971+0000 7f7ae8e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4074d60 0x7f7ae419b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.971+0000 7f7ae8e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ae4075680 0x7f7ae419b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.971+0000 7f7ae8e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae410f230 0x7f7ae419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.971+0000 7f7ae8e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae4114d20 con 0x7f7ae4074d60 2024-04-26T01:37:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.971+0000 7f7ae8e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae4114ba0 con 0x7f7ae410f230 2024-04-26T01:37:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.971+0000 7f7ae8e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae4114ea0 con 0x7f7ae4075680 2024-04-26T01:37:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4074d60 0x7f7ae419b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4074d60 0x7f7ae419b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51240/0 (socket says 172.21.15.28:51240) 2024-04-26T01:37:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3400700 1 -- 172.21.15.28:0/703866521 learned_addr learned my addr 172.21.15.28:0/703866521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7adaa00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ae4075680 0x7f7ae419b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3e00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae410f230 0x7f7ae419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3400700 1 -- 172.21.15.28:0/703866521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ae4075680 msgr2=0x7f7ae419b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3400700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ae4075680 0x7f7ae419b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3400700 1 -- 172.21.15.28:0/703866521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae410f230 msgr2=0x7f7ae419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3400700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae410f230 0x7f7ae419fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3400700 1 -- 172.21.15.28:0/703866521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae41a0240 con 0x7f7ae4074d60 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7adaa00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ae4075680 0x7f7ae419b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.972+0000 7f7ae3e00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae410f230 0x7f7ae419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.973+0000 7f7ae3400700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4074d60 0x7f7ae419b230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400e9a0 tx=0x7f7ad400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.973+0000 7f7ae0c00700 1 -- 172.21.15.28:0/703866521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad400c9c0 con 0x7f7ae4074d60 2024-04-26T01:37:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.973+0000 7f7ae0c00700 1 -- 172.21.15.28:0/703866521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad400cb60 con 0x7f7ae4074d60 2024-04-26T01:37:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.973+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae41a0530 con 0x7f7ae4074d60 2024-04-26T01:37:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.973+0000 7f7ae0c00700 1 -- 172.21.15.28:0/703866521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4018680 con 0x7f7ae4074d60 2024-04-26T01:37:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.973+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae41a7db0 con 0x7f7ae4074d60 2024-04-26T01:37:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.975+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae4069730 con 0x7f7ae4074d60 2024-04-26T01:37:40.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.977+0000 7f7ae0c00700 1 -- 172.21.15.28:0/703866521 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7ad4010430 con 0x7f7ae4074d60 2024-04-26T01:37:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.979+0000 7f7ae0c00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ac80a3d50 0x7f7ac80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.979+0000 7f7ae0c00700 1 -- 172.21.15.28:0/703866521 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f7ad40d0810 con 0x7f7ae4074d60 2024-04-26T01:37:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.979+0000 7f7adaa00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ac80a3d50 0x7f7ac80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.980+0000 7f7adaa00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ac80a3d50 0x7f7ac80a6210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0098e0 tx=0x7f7acc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:40.984+0000 7f7ae0c00700 1 -- 172.21.15.28:0/703866521 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ad409d040 con 0x7f7ae4074d60 2024-04-26T01:37:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.256+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f7ae4062e80 con 0x7f7ac80a3d50 2024-04-26T01:37:41.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.265+0000 7f7ae0c00700 1 -- 172.21.15.28:0/703866521 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7ae4062e80 con 0x7f7ac80a3d50 2024-04-26T01:37:41.262 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/57e9c16f-4990-4d2e-88b0-536ce8dec643 2024-04-26T01:37:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.268+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ac80a3d50 msgr2=0x7f7ac80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.268+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ac80a3d50 0x7f7ac80a6210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0098e0 tx=0x7f7acc005960 comp rx=0 tx=0).stop 2024-04-26T01:37:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.268+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4074d60 msgr2=0x7f7ae419b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.268+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4074d60 0x7f7ae419b230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400e9a0 tx=0x7f7ad400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:37:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.268+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 shutdown_connections 2024-04-26T01:37:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.268+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7ac80a3d50 0x7f7ac80a6210 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.268+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ae410f230 0x7f7ae419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.268+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae4074d60 0x7f7ae419b230 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.269+0000 7f7ae8e00700 1 --2- 172.21.15.28:0/703866521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ae4075680 0x7f7ae419b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:41.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.269+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 >> 172.21.15.28:0/703866521 conn(0x7f7ae40fe4c0 msgr2=0x7f7ae40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:41.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.269+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 shutdown_connections 2024-04-26T01:37:41.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:41.269+0000 7f7ae8e00700 1 -- 172.21.15.28:0/703866521 wait complete. 2024-04-26T01:37:41.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:41 smithi169 ceph-mon[32851]: pgmap v358: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:37:41.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:41 smithi053 ceph-mon[32593]: pgmap v358: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:37:42.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:41 smithi028 ceph-mon[28247]: pgmap v358: 97 pgs: 97 active+clean; 608 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:37:42.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:37:42.691 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:42.848 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:42 smithi028 ceph-mon[28247]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:42.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:42 smithi169 ceph-mon[32851]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:42 smithi053 ceph-mon[32593]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:43.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:43 smithi028 ceph-mon[28247]: pgmap v359: 97 pgs: 97 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.4 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:37:43.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:43 smithi169 ceph-mon[32851]: pgmap v359: 97 pgs: 97 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.4 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:37:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:43 smithi053 ceph-mon[32593]: pgmap v359: 97 pgs: 97 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.4 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 -- 172.21.15.28:0/1146815623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe57810f470 msgr2=0x7fe578111860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 --2- 172.21.15.28:0/1146815623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe57810f470 0x7fe578111860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe57000b0a0 tx=0x7fe57000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 -- 172.21.15.28:0/1146815623 shutdown_connections 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 --2- 172.21.15.28:0/1146815623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe57810f470 0x7fe578111860 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 --2- 172.21.15.28:0/1146815623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe578074c70 0x7fe578075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 --2- 172.21.15.28:0/1146815623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe578075590 0x7fe57810ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 -- 172.21.15.28:0/1146815623 >> 172.21.15.28:0/1146815623 conn(0x7fe578079bf0 msgr2=0x7fe57807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 -- 172.21.15.28:0/1146815623 shutdown_connections 2024-04-26T01:37:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.834+0000 7fe57f200700 1 -- 172.21.15.28:0/1146815623 wait complete. 2024-04-26T01:37:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.835+0000 7fe57f200700 1 Processor -- start 2024-04-26T01:37:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57f200700 1 -- start start 2024-04-26T01:37:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57f200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe578074c70 0x7fe57819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57f200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe578075590 0x7fe57819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57f200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe57810f470 0x7fe57819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57f200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe578114e80 con 0x7fe578074c70 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57f200700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe578114d00 con 0x7fe578075590 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57f200700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe578115000 con 0x7fe57810f470 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe577e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe578074c70 0x7fe57819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57d200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe57810f470 0x7fe57819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57d200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe57810f470 0x7fe57819fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:50250/0 (socket says 172.21.15.28:50250) 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe57d200700 1 -- 172.21.15.28:0/3874065041 learned_addr learned my addr 172.21.15.28:0/3874065041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe577e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe578074c70 0x7fe57819b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51254/0 (socket says 172.21.15.28:51254) 2024-04-26T01:37:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.836+0000 7fe577400700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe578075590 0x7fe57819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe57d200700 1 -- 172.21.15.28:0/3874065041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe578075590 msgr2=0x7fe57819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe57d200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe578075590 0x7fe57819b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe57d200700 1 -- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe578074c70 msgr2=0x7fe57819b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe57d200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe578074c70 0x7fe57819b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe57d200700 1 -- 172.21.15.28:0/3874065041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5781a0310 con 0x7fe57810f470 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe57d200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe57810f470 0x7fe57819fbd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe57000b780 tx=0x7fe570004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe574c00700 1 -- 172.21.15.28:0/3874065041 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570021070 con 0x7fe57810f470 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe574c00700 1 -- 172.21.15.28:0/3874065041 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe570007e30 con 0x7fe57810f470 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.837+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5781a05a0 con 0x7fe57810f470 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.838+0000 7fe574c00700 1 -- 172.21.15.28:0/3874065041 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570004700 con 0x7fe57810f470 2024-04-26T01:37:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.838+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5781a7e80 con 0x7fe57810f470 2024-04-26T01:37:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.839+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe578069730 con 0x7fe57810f470 2024-04-26T01:37:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.841+0000 7fe574c00700 1 -- 172.21.15.28:0/3874065041 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe570007980 con 0x7fe57810f470 2024-04-26T01:37:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.842+0000 7fe574c00700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe55c0a3d50 0x7fe55c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.842+0000 7fe574c00700 1 -- 172.21.15.28:0/3874065041 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fe57000f020 con 0x7fe57810f470 2024-04-26T01:37:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.842+0000 7fe577e00700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe55c0a3d50 0x7fe55c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.843+0000 7fe577e00700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe55c0a3d50 0x7fe55c0a6210 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe568009c60 tx=0x7fe568009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:44.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:44.847+0000 7fe574c00700 1 -- 172.21.15.28:0/3874065041 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe57002a020 con 0x7fe57810f470 2024-04-26T01:37:45.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.131+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe57819c690 con 0x7fe57810f470 2024-04-26T01:37:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.133+0000 7fe574c00700 1 -- 172.21.15.28:0/3874065041 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11861 (secure 0 0 0) 0x7fe57819c690 con 0x7fe57810f470 2024-04-26T01:37:45.130 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:37:45.130 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:37:21.237658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe55c0a3d50 msgr2=0x7fe55c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe55c0a3d50 0x7fe55c0a6210 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe568009c60 tx=0x7fe568009260 comp rx=0 tx=0).stop 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe57810f470 msgr2=0x7fe57819fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe57810f470 0x7fe57819fbd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe57000b780 tx=0x7fe570004d00 comp rx=0 tx=0).stop 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 shutdown_connections 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe55c0a3d50 0x7fe55c0a6210 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe57810f470 0x7fe57819fbd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe578074c70 0x7fe57819b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 --2- 172.21.15.28:0/3874065041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe578075590 0x7fe57819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 >> 172.21.15.28:0/3874065041 conn(0x7fe578079bf0 msgr2=0x7fe5780ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 shutdown_connections 2024-04-26T01:37:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:45.136+0000 7fe57f200700 1 -- 172.21.15.28:0/3874065041 wait complete. 2024-04-26T01:37:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-26T01:37:45.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:45 smithi169 ceph-mon[32851]: pgmap v360: 97 pgs: 97 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-26T01:37:45.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:45 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3874065041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:37:45.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:45 smithi053 ceph-mon[32593]: pgmap v360: 97 pgs: 97 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-26T01:37:45.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:45 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3874065041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:37:46.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:45 smithi028 ceph-mon[28247]: pgmap v360: 97 pgs: 97 active+clean; 625 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-26T01:37:46.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:45 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3874065041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:37:46.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:37:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:47.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:47 smithi169 ceph-mon[32851]: pgmap v361: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-26T01:37:47.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:47 smithi053 ceph-mon[32593]: pgmap v361: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-26T01:37:47.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:47 smithi028 ceph-mon[28247]: pgmap v361: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-26T01:37:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.819+0000 7fc643000700 1 -- 172.21.15.28:0/2115187464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc63c074c70 msgr2=0x7fc63c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.819+0000 7fc643000700 1 --2- 172.21.15.28:0/2115187464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc63c074c70 0x7fc63c075050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009990 tx=0x7fc62c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:37:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.819+0000 7fc643000700 1 -- 172.21.15.28:0/2115187464 shutdown_connections 2024-04-26T01:37:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.819+0000 7fc643000700 1 --2- 172.21.15.28:0/2115187464 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c10f710 0x7fc63c111b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.819+0000 7fc643000700 1 --2- 172.21.15.28:0/2115187464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc63c074c70 0x7fc63c075050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.819+0000 7fc643000700 1 --2- 172.21.15.28:0/2115187464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc63c075590 0x7fc63c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.819+0000 7fc643000700 1 -- 172.21.15.28:0/2115187464 >> 172.21.15.28:0/2115187464 conn(0x7fc63c07a9d0 msgr2=0x7fc63c07ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.820+0000 7fc643000700 1 -- 172.21.15.28:0/2115187464 shutdown_connections 2024-04-26T01:37:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.820+0000 7fc643000700 1 -- 172.21.15.28:0/2115187464 wait complete. 2024-04-26T01:37:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.821+0000 7fc643000700 1 Processor -- start 2024-04-26T01:37:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.821+0000 7fc643000700 1 -- start start 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.821+0000 7fc643000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc63c074c70 0x7fc63c19b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.821+0000 7fc643000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc63c075590 0x7fc63c19ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc643000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c10f710 0x7fc63c19fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc643000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c114f30 con 0x7fc63c075590 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc643000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c114db0 con 0x7fc63c074c70 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc643000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c1150b0 con 0x7fc63c10f710 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63be00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc63c074c70 0x7fc63c19b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63be00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc63c074c70 0x7fc63c19b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39414/0 (socket says 172.21.15.28:39414) 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63be00700 1 -- 172.21.15.28:0/3880390350 learned_addr learned my addr 172.21.15.28:0/3880390350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc63c075590 0x7fc63c19ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63be00700 1 -- 172.21.15.28:0/3880390350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c10f710 msgr2=0x7fc63c19fd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63be00700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c10f710 0x7fc63c19fd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63be00700 1 -- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc63c075590 msgr2=0x7fc63c19ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63be00700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc63c075590 0x7fc63c19ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.822+0000 7fc63be00700 1 -- 172.21.15.28:0/3880390350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c1a04d0 con 0x7fc63c074c70 2024-04-26T01:37:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.823+0000 7fc63be00700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc63c074c70 0x7fc63c19b4c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc62c005910 tx=0x7fc62c00b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.823+0000 7fc638c00700 1 -- 172.21.15.28:0/3880390350 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c027070 con 0x7fc63c074c70 2024-04-26T01:37:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.823+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c1a0760 con 0x7fc63c074c70 2024-04-26T01:37:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.823+0000 7fc638c00700 1 -- 172.21.15.28:0/3880390350 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc62c022470 con 0x7fc63c074c70 2024-04-26T01:37:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.823+0000 7fc638c00700 1 -- 172.21.15.28:0/3880390350 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c017650 con 0x7fc63c074c70 2024-04-26T01:37:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.823+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c1a8040 con 0x7fc63c074c70 2024-04-26T01:37:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.824+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc63c069730 con 0x7fc63c074c70 2024-04-26T01:37:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.826+0000 7fc638c00700 1 -- 172.21.15.28:0/3880390350 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc62c02a070 con 0x7fc63c074c70 2024-04-26T01:37:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.827+0000 7fc638c00700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6200a3cd0 0x7fc6200a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.827+0000 7fc638c00700 1 -- 172.21.15.28:0/3880390350 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fc62c0d1250 con 0x7fc63c074c70 2024-04-26T01:37:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.828+0000 7fc63b400700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6200a3cd0 0x7fc6200a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.828+0000 7fc63b400700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6200a3cd0 0x7fc6200a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc63c111410 tx=0x7fc628009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:48.831+0000 7fc638c00700 1 -- 172.21.15.28:0/3880390350 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc62c013070 con 0x7fc63c074c70 2024-04-26T01:37:48.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:48 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:48.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:48 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:48.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:48 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:49.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.063+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc63c02df80 con 0x7fc63c074c70 2024-04-26T01:37:49.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.065+0000 7fc638c00700 1 -- 172.21.15.28:0/3880390350 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22630 (secure 0 0 0) 0x7fc62c09db10 con 0x7fc63c074c70 2024-04-26T01:37:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:37:49.063 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:37:25.472266+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":7,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:37:49.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6200a3cd0 msgr2=0x7fc6200a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:49.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6200a3cd0 0x7fc6200a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc63c111410 tx=0x7fc628009260 comp rx=0 tx=0).stop 2024-04-26T01:37:49.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc63c074c70 msgr2=0x7fc63c19b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:49.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc63c074c70 0x7fc63c19b4c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc62c005910 tx=0x7fc62c00b670 comp rx=0 tx=0).stop 2024-04-26T01:37:49.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 shutdown_connections 2024-04-26T01:37:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6200a3cd0 0x7fc6200a6190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c10f710 0x7fc63c19fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc63c074c70 0x7fc63c19b4c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 --2- 172.21.15.28:0/3880390350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc63c075590 0x7fc63c19ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 >> 172.21.15.28:0/3880390350 conn(0x7fc63c07a9d0 msgr2=0x7fc63c1012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.068+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 shutdown_connections 2024-04-26T01:37:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:49.069+0000 7fc643000700 1 -- 172.21.15.28:0/3880390350 wait complete. 2024-04-26T01:37:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:49 smithi028 ceph-mon[28247]: pgmap v362: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-26T01:37:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:49 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3880390350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:49.889 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:37:49.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:37:49.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:49 smithi169 ceph-mon[32851]: pgmap v362: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-26T01:37:49.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:49 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3880390350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:49.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:49 smithi053 ceph-mon[32593]: pgmap v362: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-26T01:37:49.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:49 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3880390350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:37:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:51 smithi028 ceph-mon[28247]: pgmap v363: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-04-26T01:37:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:51 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:51 smithi169 ceph-mon[32851]: pgmap v363: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-04-26T01:37:51.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:51 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:51 smithi053 ceph-mon[32593]: pgmap v363: 97 pgs: 97 active+clean; 634 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-04-26T01:37:51.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:51.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:51 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.219+0000 7f06afc00700 1 -- 172.21.15.28:0/1726794917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a8108a50 msgr2=0x7f06a8108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.219+0000 7f06afc00700 1 --2- 172.21.15.28:0/1726794917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a8108a50 0x7f06a8108e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f069c0098f0 tx=0x7f069c009c00 comp rx=0 tx=0).stop 2024-04-26T01:37:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.219+0000 7f06afc00700 1 -- 172.21.15.28:0/1726794917 shutdown_connections 2024-04-26T01:37:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.219+0000 7f06afc00700 1 --2- 172.21.15.28:0/1726794917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06a8103450 0x7f06a810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.219+0000 7f06afc00700 1 --2- 172.21.15.28:0/1726794917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a8108a50 0x7f06a8108e30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.219+0000 7f06afc00700 1 --2- 172.21.15.28:0/1726794917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a8102a90 0x7f06a8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.219+0000 7f06afc00700 1 -- 172.21.15.28:0/1726794917 >> 172.21.15.28:0/1726794917 conn(0x7f06a8079ba0 msgr2=0x7f06a8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.219+0000 7f06afc00700 1 -- 172.21.15.28:0/1726794917 shutdown_connections 2024-04-26T01:37:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.220+0000 7f06afc00700 1 -- 172.21.15.28:0/1726794917 wait complete. 2024-04-26T01:37:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.221+0000 7f06afc00700 1 Processor -- start 2024-04-26T01:37:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.221+0000 7f06afc00700 1 -- start start 2024-04-26T01:37:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06afc00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a8102a90 0x7f06a819b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06afc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a8103450 0x7f06a819b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06afc00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06a8108a50 0x7f06a819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06afc00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8112c50 con 0x7f06a8103450 2024-04-26T01:37:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06afc00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8112ad0 con 0x7f06a8102a90 2024-04-26T01:37:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06afc00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8112dd0 con 0x7f06a8108a50 2024-04-26T01:37:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06adc00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06a8108a50 0x7f06a819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06ad200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a8102a90 0x7f06a819b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06adc00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06a8108a50 0x7f06a819fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:38194/0 (socket says 172.21.15.28:38194) 2024-04-26T01:37:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06adc00700 1 -- 172.21.15.28:0/147054190 learned_addr learned my addr 172.21.15.28:0/147054190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.222+0000 7f06a7e00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a8103450 0x7f06a819b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.223+0000 7f06ad200700 1 -- 172.21.15.28:0/147054190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06a8108a50 msgr2=0x7f06a819fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.223+0000 7f06ad200700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06a8108a50 0x7f06a819fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.223+0000 7f06ad200700 1 -- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a8103450 msgr2=0x7f06a819b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.223+0000 7f06ad200700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a8103450 0x7f06a819b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.223+0000 7f06ad200700 1 -- 172.21.15.28:0/147054190 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a81a0450 con 0x7f06a8102a90 2024-04-26T01:37:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.223+0000 7f06ad200700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a8102a90 0x7f06a819b440 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f069c00c010 tx=0x7f069c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.223+0000 7f06a5600700 1 -- 172.21.15.28:0/147054190 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0051c0 con 0x7f06a8102a90 2024-04-26T01:37:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.224+0000 7f06a5600700 1 -- 172.21.15.28:0/147054190 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f069c00b560 con 0x7f06a8102a90 2024-04-26T01:37:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.224+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a81a06e0 con 0x7f06a8102a90 2024-04-26T01:37:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.224+0000 7f06a5600700 1 -- 172.21.15.28:0/147054190 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0039e0 con 0x7f06a8102a90 2024-04-26T01:37:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.224+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a81a7fc0 con 0x7f06a8102a90 2024-04-26T01:37:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.225+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a8069730 con 0x7f06a8102a90 2024-04-26T01:37:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.227+0000 7f06a5600700 1 -- 172.21.15.28:0/147054190 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f069c00b930 con 0x7f06a8102a90 2024-04-26T01:37:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.228+0000 7f06a5600700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f06900a3d50 0x7f06900a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.229+0000 7f06a5600700 1 -- 172.21.15.28:0/147054190 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f069c0d13c0 con 0x7f06a8102a90 2024-04-26T01:37:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.229+0000 7f06a7e00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f06900a3d50 0x7f06900a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.229+0000 7f06a7e00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f06900a3d50 0x7f06900a6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0694009750 tx=0x7f0694009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.234+0000 7f06a5600700 1 -- 172.21.15.28:0/147054190 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f069c013030 con 0x7f06a8102a90 2024-04-26T01:37:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.468+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f06a819c7a0 con 0x7f06900a3d50 2024-04-26T01:37:53.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:53 smithi028 ceph-mon[28247]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-04-26T01:37:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:53 smithi053 ceph-mon[32593]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-04-26T01:37:53.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.483+0000 7f06a5600700 1 -- 172.21.15.28:0/147054190 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258950 (secure 0 0 0) 0x7f06a819c7a0 con 0x7f06900a3d50 2024-04-26T01:37:53.484 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:37:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f06900a3d50 msgr2=0x7f06900a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f06900a3d50 0x7f06900a6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0694009750 tx=0x7f0694009310 comp rx=0 tx=0).stop 2024-04-26T01:37:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a8102a90 msgr2=0x7f06a819b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a8102a90 0x7f06a819b440 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f069c00c010 tx=0x7f069c00fa00 comp rx=0 tx=0).stop 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 shutdown_connections 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f06900a3d50 0x7f06900a6210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06a8108a50 0x7f06a819fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a8102a90 0x7f06a819b440 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 --2- 172.21.15.28:0/147054190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a8103450 0x7f06a819b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 >> 172.21.15.28:0/147054190 conn(0x7f06a8079ba0 msgr2=0x7f06a810df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.488+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 shutdown_connections 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:53.489+0000 7f06afc00700 1 -- 172.21.15.28:0/147054190 wait complete. 2024-04-26T01:37:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-26T01:37:53.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:53 smithi169 ceph-mon[32851]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-04-26T01:37:54.490 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-04-26T01:37:51.781127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657756,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":352,"num_read_kb":300,"num_write":349,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181492,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373644,"kb_used_data":11992,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568180,"statfs":{"total":1151940427776,"available":1151557816320,"internally_reserved":0,"allocated":12279808,"data_stored":7716146,"data_compressed":113448,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":370193724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35187,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":76,"num_write":21,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005865"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492333+0000","last_change":"2024-04-26T01:36:54.288460+0000","last_active":"2024-04-26T01:37:25.492333+0000","last_peered":"2024-04-26T01:37:25.492333+0000","last_clean":"2024-04-26T01:37:25.492333+0000","last_became_active":"2024-04-26T01:36:54.288033+0000","last_became_peered":"2024-04-26T01:36:54.288033+0000","last_unstale":"2024-04-26T01:37:25.492333+0000","last_undegraded":"2024-04-26T01:37:25.492333+0000","last_fullsized":"2024-04-26T01:37:25.492333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:33.737622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501153+0000","last_change":"2024-04-26T01:36:54.287367+0000","last_active":"2024-04-26T01:37:25.501153+0000","last_peered":"2024-04-26T01:37:25.501153+0000","last_clean":"2024-04-26T01:37:25.501153+0000","last_became_active":"2024-04-26T01:36:54.287024+0000","last_became_peered":"2024-04-26T01:36:54.287024+0000","last_unstale":"2024-04-26T01:37:25.501153+0000","last_undegraded":"2024-04-26T01:37:25.501153+0000","last_fullsized":"2024-04-26T01:37:25.501153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:31.014627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.502368+0000","last_change":"2024-04-26T01:36:54.288433+0000","last_active":"2024-04-26T01:37:25.502368+0000","last_peered":"2024-04-26T01:37:25.502368+0000","last_clean":"2024-04-26T01:37:25.502368+0000","last_became_active":"2024-04-26T01:36:54.288002+0000","last_became_peered":"2024-04-26T01:36:54.288002+0000","last_unstale":"2024-04-26T01:37:25.502368+0000","last_undegraded":"2024-04-26T01:37:25.502368+0000","last_fullsized":"2024-04-26T01:37:25.502368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:37.885179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.498038+0000","last_change":"2024-04-26T01:36:54.273857+0000","last_active":"2024-04-26T01:37:25.498038+0000","last_peered":"2024-04-26T01:37:25.498038+0000","last_clean":"2024-04-26T01:37:25.498038+0000","last_became_active":"2024-04-26T01:36:54.272376+0000","last_became_peered":"2024-04-26T01:36:54.272376+0000","last_unstale":"2024-04-26T01:37:25.498038+0000","last_undegraded":"2024-04-26T01:37:25.498038+0000","last_fullsized":"2024-04-26T01:37:25.498038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:30.110044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494496+0000","last_change":"2024-04-26T01:36:54.288606+0000","last_active":"2024-04-26T01:37:25.494496+0000","last_peered":"2024-04-26T01:37:25.494496+0000","last_clean":"2024-04-26T01:37:25.494496+0000","last_became_active":"2024-04-26T01:36:54.288247+0000","last_became_peered":"2024-04-26T01:36:54.288247+0000","last_unstale":"2024-04-26T01:37:25.494496+0000","last_undegraded":"2024-04-26T01:37:25.494496+0000","last_fullsized":"2024-04-26T01:37:25.494496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:47:56.944164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494229+0000","last_change":"2024-04-26T01:36:54.277701+0000","last_active":"2024-04-26T01:37:25.494229+0000","last_peered":"2024-04-26T01:37:25.494229+0000","last_clean":"2024-04-26T01:37:25.494229+0000","last_became_active":"2024-04-26T01:36:54.277306+0000","last_became_peered":"2024-04-26T01:36:54.277306+0000","last_unstale":"2024-04-26T01:37:25.494229+0000","last_undegraded":"2024-04-26T01:37:25.494229+0000","last_fullsized":"2024-04-26T01:37:25.494229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:42:05.058525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491329+0000","last_change":"2024-04-26T01:36:54.274090+0000","last_active":"2024-04-26T01:37:25.491329+0000","last_peered":"2024-04-26T01:37:25.491329+0000","last_clean":"2024-04-26T01:37:25.491329+0000","last_became_active":"2024-04-26T01:36:54.273700+0000","last_became_peered":"2024-04-26T01:36:54.273700+0000","last_unstale":"2024-04-26T01:37:25.491329+0000","last_undegraded":"2024-04-26T01:37:25.491329+0000","last_fullsized":"2024-04-26T01:37:25.491329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:32.207534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501321+0000","last_change":"2024-04-26T01:37:25.499759+0000","last_active":"2024-04-26T01:37:25.501321+0000","last_peered":"2024-04-26T01:37:25.501321+0000","last_clean":"2024-04-26T01:37:25.501321+0000","last_became_active":"2024-04-26T01:37:25.499317+0000","last_became_peered":"2024-04-26T01:37:25.499317+0000","last_unstale":"2024-04-26T01:37:25.501321+0000","last_undegraded":"2024-04-26T01:37:25.501321+0000","last_fullsized":"2024-04-26T01:37:25.501321+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:39.972315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489354+0000","last_change":"2024-04-26T01:36:54.277457+0000","last_active":"2024-04-26T01:37:25.489354+0000","last_peered":"2024-04-26T01:37:25.489354+0000","last_clean":"2024-04-26T01:37:25.489354+0000","last_became_active":"2024-04-26T01:36:54.277118+0000","last_became_peered":"2024-04-26T01:36:54.277118+0000","last_unstale":"2024-04-26T01:37:25.489354+0000","last_undegraded":"2024-04-26T01:37:25.489354+0000","last_fullsized":"2024-04-26T01:37:25.489354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:05.628027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495077+0000","last_change":"2024-04-26T01:36:54.278494+0000","last_active":"2024-04-26T01:37:25.495077+0000","last_peered":"2024-04-26T01:37:25.495077+0000","last_clean":"2024-04-26T01:37:25.495077+0000","last_became_active":"2024-04-26T01:36:54.278188+0000","last_became_peered":"2024-04-26T01:36:54.278188+0000","last_unstale":"2024-04-26T01:37:25.495077+0000","last_undegraded":"2024-04-26T01:37:25.495077+0000","last_fullsized":"2024-04-26T01:37:25.495077+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:59.150771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491742+0000","last_change":"2024-04-26T01:36:54.277424+0000","last_active":"2024-04-26T01:37:25.491742+0000","last_peered":"2024-04-26T01:37:25.491742+0000","last_clean":"2024-04-26T01:37:25.491742+0000","last_became_active":"2024-04-26T01:36:54.275752+0000","last_became_peered":"2024-04-26T01:36:54.275752+0000","last_unstale":"2024-04-26T01:37:25.491742+0000","last_undegraded":"2024-04-26T01:37:25.491742+0000","last_fullsized":"2024-04-26T01:37:25.491742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:22.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.503692+0000","last_change":"2024-04-26T01:37:25.502448+0000","last_active":"2024-04-26T01:37:25.503692+0000","last_peered":"2024-04-26T01:37:25.503692+0000","last_clean":"2024-04-26T01:37:25.503692+0000","last_became_active":"2024-04-26T01:37:25.501161+0000","last_became_peered":"2024-04-26T01:37:25.501161+0000","last_unstale":"2024-04-26T01:37:25.503692+0000","last_undegraded":"2024-04-26T01:37:25.503692+0000","last_fullsized":"2024-04-26T01:37:25.503692+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:41:40.443780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492930+0000","last_change":"2024-04-26T01:36:54.268721+0000","last_active":"2024-04-26T01:37:25.492930+0000","last_peered":"2024-04-26T01:37:25.492930+0000","last_clean":"2024-04-26T01:37:25.492930+0000","last_became_active":"2024-04-26T01:36:54.268007+0000","last_became_peered":"2024-04-26T01:36:54.268007+0000","last_unstale":"2024-04-26T01:37:25.492930+0000","last_undegraded":"2024-04-26T01:37:25.492930+0000","last_fullsized":"2024-04-26T01:37:25.492930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:55.120130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.500807+0000","last_change":"2024-04-26T01:36:54.286595+0000","last_active":"2024-04-26T01:37:25.500807+0000","last_peered":"2024-04-26T01:37:25.500807+0000","last_clean":"2024-04-26T01:37:25.500807+0000","last_became_active":"2024-04-26T01:36:54.286236+0000","last_became_peered":"2024-04-26T01:36:54.286236+0000","last_unstale":"2024-04-26T01:37:25.500807+0000","last_undegraded":"2024-04-26T01:37:25.500807+0000","last_fullsized":"2024-04-26T01:37:25.500807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:59.443467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.500156+0000","last_change":"2024-04-26T01:36:54.293254+0000","last_active":"2024-04-26T01:37:25.500156+0000","last_peered":"2024-04-26T01:37:25.500156+0000","last_clean":"2024-04-26T01:37:25.500156+0000","last_became_active":"2024-04-26T01:36:54.292853+0000","last_became_peered":"2024-04-26T01:36:54.292853+0000","last_unstale":"2024-04-26T01:37:25.500156+0000","last_undegraded":"2024-04-26T01:37:25.500156+0000","last_fullsized":"2024-04-26T01:37:25.500156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:19.844006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488715+0000","last_change":"2024-04-26T01:36:54.285346+0000","last_active":"2024-04-26T01:37:25.488715+0000","last_peered":"2024-04-26T01:37:25.488715+0000","last_clean":"2024-04-26T01:37:25.488715+0000","last_became_active":"2024-04-26T01:36:54.284949+0000","last_became_peered":"2024-04-26T01:36:54.284949+0000","last_unstale":"2024-04-26T01:37:25.488715+0000","last_undegraded":"2024-04-26T01:37:25.488715+0000","last_fullsized":"2024-04-26T01:37:25.488715+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:33:29.276240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492076+0000","last_change":"2024-04-26T01:36:54.283161+0000","last_active":"2024-04-26T01:37:25.492076+0000","last_peered":"2024-04-26T01:37:25.492076+0000","last_clean":"2024-04-26T01:37:25.492076+0000","last_became_active":"2024-04-26T01:36:54.282908+0000","last_became_peered":"2024-04-26T01:36:54.282908+0000","last_unstale":"2024-04-26T01:37:25.492076+0000","last_undegraded":"2024-04-26T01:37:25.492076+0000","last_fullsized":"2024-04-26T01:37:25.492076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:26.503964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497700+0000","last_change":"2024-04-26T01:36:54.282516+0000","last_active":"2024-04-26T01:37:25.497700+0000","last_peered":"2024-04-26T01:37:25.497700+0000","last_clean":"2024-04-26T01:37:25.497700+0000","last_became_active":"2024-04-26T01:36:54.278634+0000","last_became_peered":"2024-04-26T01:36:54.278634+0000","last_unstale":"2024-04-26T01:37:25.497700+0000","last_undegraded":"2024-04-26T01:37:25.497700+0000","last_fullsized":"2024-04-26T01:37:25.497700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:10:38.275720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491997+0000","last_change":"2024-04-26T01:36:54.289502+0000","last_active":"2024-04-26T01:37:25.491997+0000","last_peered":"2024-04-26T01:37:25.491997+0000","last_clean":"2024-04-26T01:37:25.491997+0000","last_became_active":"2024-04-26T01:36:54.289079+0000","last_became_peered":"2024-04-26T01:36:54.289079+0000","last_unstale":"2024-04-26T01:37:25.491997+0000","last_undegraded":"2024-04-26T01:37:25.491997+0000","last_fullsized":"2024-04-26T01:37:25.491997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:51.412032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.498019+0000","last_change":"2024-04-26T01:36:50.243164+0000","last_active":"2024-04-26T01:37:25.498019+0000","last_peered":"2024-04-26T01:37:25.498019+0000","last_clean":"2024-04-26T01:37:25.498019+0000","last_became_active":"2024-04-26T01:36:50.242508+0000","last_became_peered":"2024-04-26T01:36:50.242508+0000","last_unstale":"2024-04-26T01:37:25.498019+0000","last_undegraded":"2024-04-26T01:37:25.498019+0000","last_fullsized":"2024-04-26T01:37:25.498019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:06:01.662698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514971+0000","last_change":"2024-04-26T01:36:54.279809+0000","last_active":"2024-04-26T01:37:25.514971+0000","last_peered":"2024-04-26T01:37:25.514971+0000","last_clean":"2024-04-26T01:37:25.514971+0000","last_became_active":"2024-04-26T01:36:54.279509+0000","last_became_peered":"2024-04-26T01:36:54.279509+0000","last_unstale":"2024-04-26T01:37:25.514971+0000","last_undegraded":"2024-04-26T01:37:25.514971+0000","last_fullsized":"2024-04-26T01:37:25.514971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:54:26.734136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.515137+0000","last_change":"2024-04-26T01:36:50.261942+0000","last_active":"2024-04-26T01:37:25.515137+0000","last_peered":"2024-04-26T01:37:25.515137+0000","last_clean":"2024-04-26T01:37:25.515137+0000","last_became_active":"2024-04-26T01:36:50.261564+0000","last_became_peered":"2024-04-26T01:36:50.261564+0000","last_unstale":"2024-04-26T01:37:25.515137+0000","last_undegraded":"2024-04-26T01:37:25.515137+0000","last_fullsized":"2024-04-26T01:37:25.515137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:25.102557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501782+0000","last_change":"2024-04-26T01:36:54.288011+0000","last_active":"2024-04-26T01:37:25.501782+0000","last_peered":"2024-04-26T01:37:25.501782+0000","last_clean":"2024-04-26T01:37:25.501782+0000","last_became_active":"2024-04-26T01:36:54.286853+0000","last_became_peered":"2024-04-26T01:36:54.286853+0000","last_unstale":"2024-04-26T01:37:25.501782+0000","last_undegraded":"2024-04-26T01:37:25.501782+0000","last_fullsized":"2024-04-26T01:37:25.501782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:53.422470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495834+0000","last_change":"2024-04-26T01:36:50.235291+0000","last_active":"2024-04-26T01:37:25.495834+0000","last_peered":"2024-04-26T01:37:25.495834+0000","last_clean":"2024-04-26T01:37:25.495834+0000","last_became_active":"2024-04-26T01:36:50.234939+0000","last_became_peered":"2024-04-26T01:36:50.234939+0000","last_unstale":"2024-04-26T01:37:25.495834+0000","last_undegraded":"2024-04-26T01:37:25.495834+0000","last_fullsized":"2024-04-26T01:37:25.495834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:13.048185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488964+0000","last_change":"2024-04-26T01:36:54.282731+0000","last_active":"2024-04-26T01:37:25.488964+0000","last_peered":"2024-04-26T01:37:25.488964+0000","last_clean":"2024-04-26T01:37:25.488964+0000","last_became_active":"2024-04-26T01:36:54.282384+0000","last_became_peered":"2024-04-26T01:36:54.282384+0000","last_unstale":"2024-04-26T01:37:25.488964+0000","last_undegraded":"2024-04-26T01:37:25.488964+0000","last_fullsized":"2024-04-26T01:37:25.488964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:09.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495596+0000","last_change":"2024-04-26T01:36:50.233581+0000","last_active":"2024-04-26T01:37:25.495596+0000","last_peered":"2024-04-26T01:37:25.495596+0000","last_clean":"2024-04-26T01:37:25.495596+0000","last_became_active":"2024-04-26T01:36:50.233210+0000","last_became_peered":"2024-04-26T01:36:50.233210+0000","last_unstale":"2024-04-26T01:37:25.495596+0000","last_undegraded":"2024-04-26T01:37:25.495596+0000","last_fullsized":"2024-04-26T01:37:25.495596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:01.223954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489527+0000","last_change":"2024-04-26T01:36:54.274657+0000","last_active":"2024-04-26T01:37:25.489527+0000","last_peered":"2024-04-26T01:37:25.489527+0000","last_clean":"2024-04-26T01:37:25.489527+0000","last_became_active":"2024-04-26T01:36:54.274333+0000","last_became_peered":"2024-04-26T01:36:54.274333+0000","last_unstale":"2024-04-26T01:37:25.489527+0000","last_undegraded":"2024-04-26T01:37:25.489527+0000","last_fullsized":"2024-04-26T01:37:25.489527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:37.243307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:45.551125+0000","last_change":"2024-04-26T01:37:25.502311+0000","last_active":"2024-04-26T01:37:45.551125+0000","last_peered":"2024-04-26T01:37:45.551125+0000","last_clean":"2024-04-26T01:37:45.551125+0000","last_became_active":"2024-04-26T01:37:25.493259+0000","last_became_peered":"2024-04-26T01:37:25.493259+0000","last_unstale":"2024-04-26T01:37:45.551125+0000","last_undegraded":"2024-04-26T01:37:45.551125+0000","last_fullsized":"2024-04-26T01:37:45.551125+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:46:49.224905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494982+0000","last_change":"2024-04-26T01:36:54.275116+0000","last_active":"2024-04-26T01:37:25.494982+0000","last_peered":"2024-04-26T01:37:25.494982+0000","last_clean":"2024-04-26T01:37:25.494982+0000","last_became_active":"2024-04-26T01:36:54.274815+0000","last_became_peered":"2024-04-26T01:36:54.274815+0000","last_unstale":"2024-04-26T01:37:25.494982+0000","last_undegraded":"2024-04-26T01:37:25.494982+0000","last_fullsized":"2024-04-26T01:37:25.494982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:04.427670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.505327+0000","last_change":"2024-04-26T01:37:25.505327+0000","last_active":"2024-04-26T01:37:25.505327+0000","last_peered":"2024-04-26T01:37:25.505327+0000","last_clean":"2024-04-26T01:37:25.505327+0000","last_became_active":"2024-04-26T01:37:25.497881+0000","last_became_peered":"2024-04-26T01:37:25.497881+0000","last_unstale":"2024-04-26T01:37:25.505327+0000","last_undegraded":"2024-04-26T01:37:25.505327+0000","last_fullsized":"2024-04-26T01:37:25.505327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:09.181372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489215+0000","last_change":"2024-04-26T01:36:54.276415+0000","last_active":"2024-04-26T01:37:25.489215+0000","last_peered":"2024-04-26T01:37:25.489215+0000","last_clean":"2024-04-26T01:37:25.489215+0000","last_became_active":"2024-04-26T01:36:54.276080+0000","last_became_peered":"2024-04-26T01:36:54.276080+0000","last_unstale":"2024-04-26T01:37:25.489215+0000","last_undegraded":"2024-04-26T01:37:25.489215+0000","last_fullsized":"2024-04-26T01:37:25.489215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:08:17.171620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491798+0000","last_change":"2024-04-26T01:36:50.241552+0000","last_active":"2024-04-26T01:37:25.491798+0000","last_peered":"2024-04-26T01:37:25.491798+0000","last_clean":"2024-04-26T01:37:25.491798+0000","last_became_active":"2024-04-26T01:36:50.241022+0000","last_became_peered":"2024-04-26T01:36:50.241022+0000","last_unstale":"2024-04-26T01:37:25.491798+0000","last_undegraded":"2024-04-26T01:37:25.491798+0000","last_fullsized":"2024-04-26T01:37:25.491798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:55.503831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514256+0000","last_change":"2024-04-26T01:36:54.276641+0000","last_active":"2024-04-26T01:37:25.514256+0000","last_peered":"2024-04-26T01:37:25.514256+0000","last_clean":"2024-04-26T01:37:25.514256+0000","last_became_active":"2024-04-26T01:36:54.276309+0000","last_became_peered":"2024-04-26T01:36:54.276309+0000","last_unstale":"2024-04-26T01:37:25.514256+0000","last_undegraded":"2024-04-26T01:37:25.514256+0000","last_fullsized":"2024-04-26T01:37:25.514256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:53.725584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488938+0000","last_change":"2024-04-26T01:36:50.235504+0000","last_active":"2024-04-26T01:37:25.488938+0000","last_peered":"2024-04-26T01:37:25.488938+0000","last_clean":"2024-04-26T01:37:25.488938+0000","last_became_active":"2024-04-26T01:36:50.235161+0000","last_became_peered":"2024-04-26T01:36:50.235161+0000","last_unstale":"2024-04-26T01:37:25.488938+0000","last_undegraded":"2024-04-26T01:37:25.488938+0000","last_fullsized":"2024-04-26T01:37:25.488938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:05.951521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494601+0000","last_change":"2024-04-26T01:36:54.298236+0000","last_active":"2024-04-26T01:37:25.494601+0000","last_peered":"2024-04-26T01:37:25.494601+0000","last_clean":"2024-04-26T01:37:25.494601+0000","last_became_active":"2024-04-26T01:36:54.297850+0000","last_became_peered":"2024-04-26T01:36:54.297850+0000","last_unstale":"2024-04-26T01:37:25.494601+0000","last_undegraded":"2024-04-26T01:37:25.494601+0000","last_fullsized":"2024-04-26T01:37:25.494601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:35:35.244502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:48.556085+0000","last_change":"2024-04-26T01:36:50.241418+0000","last_active":"2024-04-26T01:37:48.556085+0000","last_peered":"2024-04-26T01:37:48.556085+0000","last_clean":"2024-04-26T01:37:48.556085+0000","last_became_active":"2024-04-26T01:36:50.241044+0000","last_became_peered":"2024-04-26T01:36:50.241044+0000","last_unstale":"2024-04-26T01:37:48.556085+0000","last_undegraded":"2024-04-26T01:37:48.556085+0000","last_fullsized":"2024-04-26T01:37:48.556085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:13.317953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489357+0000","last_change":"2024-04-26T01:36:54.284410+0000","last_active":"2024-04-26T01:37:25.489357+0000","last_peered":"2024-04-26T01:37:25.489357+0000","last_clean":"2024-04-26T01:37:25.489357+0000","last_became_active":"2024-04-26T01:36:54.283629+0000","last_became_peered":"2024-04-26T01:36:54.283629+0000","last_unstale":"2024-04-26T01:37:25.489357+0000","last_undegraded":"2024-04-26T01:37:25.489357+0000","last_fullsized":"2024-04-26T01:37:25.489357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:15.380006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.500348+0000","last_change":"2024-04-26T01:37:25.499874+0000","last_active":"2024-04-26T01:37:25.500348+0000","last_peered":"2024-04-26T01:37:25.500348+0000","last_clean":"2024-04-26T01:37:25.500348+0000","last_became_active":"2024-04-26T01:37:25.494878+0000","last_became_peered":"2024-04-26T01:37:25.494878+0000","last_unstale":"2024-04-26T01:37:25.500348+0000","last_undegraded":"2024-04-26T01:37:25.500348+0000","last_fullsized":"2024-04-26T01:37:25.500348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:48:12.678331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.502056+0000","last_change":"2024-04-26T01:36:54.285118+0000","last_active":"2024-04-26T01:37:25.502056+0000","last_peered":"2024-04-26T01:37:25.502056+0000","last_clean":"2024-04-26T01:37:25.502056+0000","last_became_active":"2024-04-26T01:36:54.284554+0000","last_became_peered":"2024-04-26T01:36:54.284554+0000","last_unstale":"2024-04-26T01:37:25.502056+0000","last_undegraded":"2024-04-26T01:37:25.502056+0000","last_fullsized":"2024-04-26T01:37:25.502056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:13.721684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489680+0000","last_change":"2024-04-26T01:36:50.262451+0000","last_active":"2024-04-26T01:37:25.489680+0000","last_peered":"2024-04-26T01:37:25.489680+0000","last_clean":"2024-04-26T01:37:25.489680+0000","last_became_active":"2024-04-26T01:36:50.262056+0000","last_became_peered":"2024-04-26T01:36:50.262056+0000","last_unstale":"2024-04-26T01:37:25.489680+0000","last_undegraded":"2024-04-26T01:37:25.489680+0000","last_fullsized":"2024-04-26T01:37:25.489680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:50.813675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488943+0000","last_change":"2024-04-26T01:36:54.283975+0000","last_active":"2024-04-26T01:37:25.488943+0000","last_peered":"2024-04-26T01:37:25.488943+0000","last_clean":"2024-04-26T01:37:25.488943+0000","last_became_active":"2024-04-26T01:36:54.283564+0000","last_became_peered":"2024-04-26T01:36:54.283564+0000","last_unstale":"2024-04-26T01:37:25.488943+0000","last_undegraded":"2024-04-26T01:37:25.488943+0000","last_fullsized":"2024-04-26T01:37:25.488943+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:53.881763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":130,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:49.490712+0000","last_change":"2024-04-26T01:36:50.238839+0000","last_active":"2024-04-26T01:37:49.490712+0000","last_peered":"2024-04-26T01:37:49.490712+0000","last_clean":"2024-04-26T01:37:49.490712+0000","last_became_active":"2024-04-26T01:36:50.238452+0000","last_became_peered":"2024-04-26T01:36:50.238452+0000","last_unstale":"2024-04-26T01:37:49.490712+0000","last_undegraded":"2024-04-26T01:37:49.490712+0000","last_fullsized":"2024-04-26T01:37:49.490712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:05:40.426825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63153,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":74,"num_read_kb":98,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495641+0000","last_change":"2024-04-26T01:36:54.271446+0000","last_active":"2024-04-26T01:37:25.495641+0000","last_peered":"2024-04-26T01:37:25.495641+0000","last_clean":"2024-04-26T01:37:25.495641+0000","last_became_active":"2024-04-26T01:36:54.271122+0000","last_became_peered":"2024-04-26T01:36:54.271122+0000","last_unstale":"2024-04-26T01:37:25.495641+0000","last_undegraded":"2024-04-26T01:37:25.495641+0000","last_fullsized":"2024-04-26T01:37:25.495641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:28:24.275481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491672+0000","last_change":"2024-04-26T01:36:50.237726+0000","last_active":"2024-04-26T01:37:25.491672+0000","last_peered":"2024-04-26T01:37:25.491672+0000","last_clean":"2024-04-26T01:37:25.491672+0000","last_became_active":"2024-04-26T01:36:50.237215+0000","last_became_peered":"2024-04-26T01:36:50.237215+0000","last_unstale":"2024-04-26T01:37:25.491672+0000","last_undegraded":"2024-04-26T01:37:25.491672+0000","last_fullsized":"2024-04-26T01:37:25.491672+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:41.781876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491099+0000","last_change":"2024-04-26T01:36:54.290616+0000","last_active":"2024-04-26T01:37:25.491099+0000","last_peered":"2024-04-26T01:37:25.491099+0000","last_clean":"2024-04-26T01:37:25.491099+0000","last_became_active":"2024-04-26T01:36:54.290293+0000","last_became_peered":"2024-04-26T01:36:54.290293+0000","last_unstale":"2024-04-26T01:37:25.491099+0000","last_undegraded":"2024-04-26T01:37:25.491099+0000","last_fullsized":"2024-04-26T01:37:25.491099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:07.439333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489132+0000","last_change":"2024-04-26T01:36:50.237578+0000","last_active":"2024-04-26T01:37:25.489132+0000","last_peered":"2024-04-26T01:37:25.489132+0000","last_clean":"2024-04-26T01:37:25.489132+0000","last_became_active":"2024-04-26T01:36:50.237224+0000","last_became_peered":"2024-04-26T01:36:50.237224+0000","last_unstale":"2024-04-26T01:37:25.489132+0000","last_undegraded":"2024-04-26T01:37:25.489132+0000","last_fullsized":"2024-04-26T01:37:25.489132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:46:38.078631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491672+0000","last_change":"2024-04-26T01:36:54.280818+0000","last_active":"2024-04-26T01:37:25.491672+0000","last_peered":"2024-04-26T01:37:25.491672+0000","last_clean":"2024-04-26T01:37:25.491672+0000","last_became_active":"2024-04-26T01:36:54.280440+0000","last_became_peered":"2024-04-26T01:36:54.280440+0000","last_unstale":"2024-04-26T01:37:25.491672+0000","last_undegraded":"2024-04-26T01:37:25.491672+0000","last_fullsized":"2024-04-26T01:37:25.491672+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:47.185219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:47.507524+0000","last_change":"2024-04-26T01:36:50.234602+0000","last_active":"2024-04-26T01:37:47.507524+0000","last_peered":"2024-04-26T01:37:47.507524+0000","last_clean":"2024-04-26T01:37:47.507524+0000","last_became_active":"2024-04-26T01:36:50.234175+0000","last_became_peered":"2024-04-26T01:36:50.234175+0000","last_unstale":"2024-04-26T01:37:47.507524+0000","last_undegraded":"2024-04-26T01:37:47.507524+0000","last_fullsized":"2024-04-26T01:37:47.507524+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:17:14.236069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491239+0000","last_change":"2024-04-26T01:36:54.271466+0000","last_active":"2024-04-26T01:37:25.491239+0000","last_peered":"2024-04-26T01:37:25.491239+0000","last_clean":"2024-04-26T01:37:25.491239+0000","last_became_active":"2024-04-26T01:36:54.271086+0000","last_became_peered":"2024-04-26T01:36:54.271086+0000","last_unstale":"2024-04-26T01:37:25.491239+0000","last_undegraded":"2024-04-26T01:37:25.491239+0000","last_fullsized":"2024-04-26T01:37:25.491239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:41.052895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.518730+0000","last_change":"2024-04-26T01:37:25.518730+0000","last_active":"2024-04-26T01:37:25.518730+0000","last_peered":"2024-04-26T01:37:25.518730+0000","last_clean":"2024-04-26T01:37:25.518730+0000","last_became_active":"2024-04-26T01:37:25.504464+0000","last_became_peered":"2024-04-26T01:37:25.504464+0000","last_unstale":"2024-04-26T01:37:25.518730+0000","last_undegraded":"2024-04-26T01:37:25.518730+0000","last_fullsized":"2024-04-26T01:37:25.518730+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:07.578917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488951+0000","last_change":"2024-04-26T01:36:54.273621+0000","last_active":"2024-04-26T01:37:25.488951+0000","last_peered":"2024-04-26T01:37:25.488951+0000","last_clean":"2024-04-26T01:37:25.488951+0000","last_became_active":"2024-04-26T01:36:54.273218+0000","last_became_peered":"2024-04-26T01:36:54.273218+0000","last_unstale":"2024-04-26T01:37:25.488951+0000","last_undegraded":"2024-04-26T01:37:25.488951+0000","last_fullsized":"2024-04-26T01:37:25.488951+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:55.543943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.509030+0000","last_change":"2024-04-26T01:37:25.508575+0000","last_active":"2024-04-26T01:37:25.509030+0000","last_peered":"2024-04-26T01:37:25.509030+0000","last_clean":"2024-04-26T01:37:25.509030+0000","last_became_active":"2024-04-26T01:37:25.496407+0000","last_became_peered":"2024-04-26T01:37:25.496407+0000","last_unstale":"2024-04-26T01:37:25.509030+0000","last_undegraded":"2024-04-26T01:37:25.509030+0000","last_fullsized":"2024-04-26T01:37:25.509030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:05.878252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492135+0000","last_change":"2024-04-26T01:36:54.272147+0000","last_active":"2024-04-26T01:37:25.492135+0000","last_peered":"2024-04-26T01:37:25.492135+0000","last_clean":"2024-04-26T01:37:25.492135+0000","last_became_active":"2024-04-26T01:36:54.271737+0000","last_became_peered":"2024-04-26T01:36:54.271737+0000","last_unstale":"2024-04-26T01:37:25.492135+0000","last_undegraded":"2024-04-26T01:37:25.492135+0000","last_fullsized":"2024-04-26T01:37:25.492135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:41.445073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.496852+0000","last_change":"2024-04-26T01:36:50.264444+0000","last_active":"2024-04-26T01:37:25.496852+0000","last_peered":"2024-04-26T01:37:25.496852+0000","last_clean":"2024-04-26T01:37:25.496852+0000","last_became_active":"2024-04-26T01:36:50.264058+0000","last_became_peered":"2024-04-26T01:36:50.264058+0000","last_unstale":"2024-04-26T01:37:25.496852+0000","last_undegraded":"2024-04-26T01:37:25.496852+0000","last_fullsized":"2024-04-26T01:37:25.496852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:56:15.428588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.493923+0000","last_change":"2024-04-26T01:36:54.289537+0000","last_active":"2024-04-26T01:37:25.493923+0000","last_peered":"2024-04-26T01:37:25.493923+0000","last_clean":"2024-04-26T01:37:25.493923+0000","last_became_active":"2024-04-26T01:36:54.289149+0000","last_became_peered":"2024-04-26T01:36:54.289149+0000","last_unstale":"2024-04-26T01:37:25.493923+0000","last_undegraded":"2024-04-26T01:37:25.493923+0000","last_fullsized":"2024-04-26T01:37:25.493923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:07.801551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491498+0000","last_change":"2024-04-26T01:36:50.242190+0000","last_active":"2024-04-26T01:37:25.491498+0000","last_peered":"2024-04-26T01:37:25.491498+0000","last_clean":"2024-04-26T01:37:25.491498+0000","last_became_active":"2024-04-26T01:36:50.241890+0000","last_became_peered":"2024-04-26T01:36:50.241890+0000","last_unstale":"2024-04-26T01:37:25.491498+0000","last_undegraded":"2024-04-26T01:37:25.491498+0000","last_fullsized":"2024-04-26T01:37:25.491498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:29.472502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494503+0000","last_change":"2024-04-26T01:36:54.294699+0000","last_active":"2024-04-26T01:37:25.494503+0000","last_peered":"2024-04-26T01:37:25.494503+0000","last_clean":"2024-04-26T01:37:25.494503+0000","last_became_active":"2024-04-26T01:36:54.294313+0000","last_became_peered":"2024-04-26T01:36:54.294313+0000","last_unstale":"2024-04-26T01:37:25.494503+0000","last_undegraded":"2024-04-26T01:37:25.494503+0000","last_fullsized":"2024-04-26T01:37:25.494503+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:15.026357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489176+0000","last_change":"2024-04-26T01:36:54.276798+0000","last_active":"2024-04-26T01:37:25.489176+0000","last_peered":"2024-04-26T01:37:25.489176+0000","last_clean":"2024-04-26T01:37:25.489176+0000","last_became_active":"2024-04-26T01:36:54.275955+0000","last_became_peered":"2024-04-26T01:36:54.275955+0000","last_unstale":"2024-04-26T01:37:25.489176+0000","last_undegraded":"2024-04-26T01:37:25.489176+0000","last_fullsized":"2024-04-26T01:37:25.489176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:17.020909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:47.506381+0000","last_change":"2024-04-26T01:36:50.263356+0000","last_active":"2024-04-26T01:37:47.506381+0000","last_peered":"2024-04-26T01:37:47.506381+0000","last_clean":"2024-04-26T01:37:47.506381+0000","last_became_active":"2024-04-26T01:36:50.261596+0000","last_became_peered":"2024-04-26T01:36:50.261596+0000","last_unstale":"2024-04-26T01:37:47.506381+0000","last_undegraded":"2024-04-26T01:37:47.506381+0000","last_fullsized":"2024-04-26T01:37:47.506381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:10.773347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514523+0000","last_change":"2024-04-26T01:36:54.288135+0000","last_active":"2024-04-26T01:37:25.514523+0000","last_peered":"2024-04-26T01:37:25.514523+0000","last_clean":"2024-04-26T01:37:25.514523+0000","last_became_active":"2024-04-26T01:36:54.287728+0000","last_became_peered":"2024-04-26T01:36:54.287728+0000","last_unstale":"2024-04-26T01:37:25.514523+0000","last_undegraded":"2024-04-26T01:37:25.514523+0000","last_fullsized":"2024-04-26T01:37:25.514523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:18.428151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497273+0000","last_change":"2024-04-26T01:36:50.243522+0000","last_active":"2024-04-26T01:37:25.497273+0000","last_peered":"2024-04-26T01:37:25.497273+0000","last_clean":"2024-04-26T01:37:25.497273+0000","last_became_active":"2024-04-26T01:36:50.243084+0000","last_became_peered":"2024-04-26T01:36:50.243084+0000","last_unstale":"2024-04-26T01:37:25.497273+0000","last_undegraded":"2024-04-26T01:37:25.497273+0000","last_fullsized":"2024-04-26T01:37:25.497273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:03.159750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491542+0000","last_change":"2024-04-26T01:35:13.920486+0000","last_active":"2024-04-26T01:37:25.491542+0000","last_peered":"2024-04-26T01:37:25.491542+0000","last_clean":"2024-04-26T01:37:25.491542+0000","last_became_active":"2024-04-26T01:35:13.919834+0000","last_became_peered":"2024-04-26T01:35:13.919834+0000","last_unstale":"2024-04-26T01:37:25.491542+0000","last_undegraded":"2024-04-26T01:37:25.491542+0000","last_fullsized":"2024-04-26T01:37:25.491542+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_clean_scrub_stamp":"2024-04-26T01:31:12.563114+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:53.935491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.490631+0000","last_change":"2024-04-26T01:36:54.270553+0000","last_active":"2024-04-26T01:37:25.490631+0000","last_peered":"2024-04-26T01:37:25.490631+0000","last_clean":"2024-04-26T01:37:25.490631+0000","last_became_active":"2024-04-26T01:36:54.269986+0000","last_became_peered":"2024-04-26T01:36:54.269986+0000","last_unstale":"2024-04-26T01:37:25.490631+0000","last_undegraded":"2024-04-26T01:37:25.490631+0000","last_fullsized":"2024-04-26T01:37:25.490631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:19.597337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491547+0000","last_change":"2024-04-26T01:36:50.235583+0000","last_active":"2024-04-26T01:37:25.491547+0000","last_peered":"2024-04-26T01:37:25.491547+0000","last_clean":"2024-04-26T01:37:25.491547+0000","last_became_active":"2024-04-26T01:36:50.235214+0000","last_became_peered":"2024-04-26T01:36:50.235214+0000","last_unstale":"2024-04-26T01:37:25.491547+0000","last_undegraded":"2024-04-26T01:37:25.491547+0000","last_fullsized":"2024-04-26T01:37:25.491547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:05.822872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495673+0000","last_change":"2024-04-26T01:36:50.260988+0000","last_active":"2024-04-26T01:37:25.495673+0000","last_peered":"2024-04-26T01:37:25.495673+0000","last_clean":"2024-04-26T01:37:25.495673+0000","last_became_active":"2024-04-26T01:36:50.260615+0000","last_became_peered":"2024-04-26T01:36:50.260615+0000","last_unstale":"2024-04-26T01:37:25.495673+0000","last_undegraded":"2024-04-26T01:37:25.495673+0000","last_fullsized":"2024-04-26T01:37:25.495673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:15.933939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492283+0000","last_change":"2024-04-26T01:36:54.279650+0000","last_active":"2024-04-26T01:37:25.492283+0000","last_peered":"2024-04-26T01:37:25.492283+0000","last_clean":"2024-04-26T01:37:25.492283+0000","last_became_active":"2024-04-26T01:36:54.275814+0000","last_became_peered":"2024-04-26T01:36:54.275814+0000","last_unstale":"2024-04-26T01:37:25.492283+0000","last_undegraded":"2024-04-26T01:37:25.492283+0000","last_fullsized":"2024-04-26T01:37:25.492283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:58:42.369511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:43.307474+0000","last_change":"2024-04-26T01:36:50.248179+0000","last_active":"2024-04-26T01:37:43.307474+0000","last_peered":"2024-04-26T01:37:43.307474+0000","last_clean":"2024-04-26T01:37:43.307474+0000","last_became_active":"2024-04-26T01:36:50.247805+0000","last_became_peered":"2024-04-26T01:36:50.247805+0000","last_unstale":"2024-04-26T01:37:43.307474+0000","last_undegraded":"2024-04-26T01:37:43.307474+0000","last_fullsized":"2024-04-26T01:37:43.307474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:19:23.740920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514830+0000","last_change":"2024-04-26T01:36:54.276626+0000","last_active":"2024-04-26T01:37:25.514830+0000","last_peered":"2024-04-26T01:37:25.514830+0000","last_clean":"2024-04-26T01:37:25.514830+0000","last_became_active":"2024-04-26T01:36:54.276238+0000","last_became_peered":"2024-04-26T01:36:54.276238+0000","last_unstale":"2024-04-26T01:37:25.514830+0000","last_undegraded":"2024-04-26T01:37:25.514830+0000","last_fullsized":"2024-04-26T01:37:25.514830+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:57.306950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497759+0000","last_change":"2024-04-26T01:36:50.278502+0000","last_active":"2024-04-26T01:37:25.497759+0000","last_peered":"2024-04-26T01:37:25.497759+0000","last_clean":"2024-04-26T01:37:25.497759+0000","last_became_active":"2024-04-26T01:36:50.278215+0000","last_became_peered":"2024-04-26T01:36:50.278215+0000","last_unstale":"2024-04-26T01:37:25.497759+0000","last_undegraded":"2024-04-26T01:37:25.497759+0000","last_fullsized":"2024-04-26T01:37:25.497759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:05:46.142902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501473+0000","last_change":"2024-04-26T01:36:54.292600+0000","last_active":"2024-04-26T01:37:25.501473+0000","last_peered":"2024-04-26T01:37:25.501473+0000","last_clean":"2024-04-26T01:37:25.501473+0000","last_became_active":"2024-04-26T01:36:54.292260+0000","last_became_peered":"2024-04-26T01:36:54.292260+0000","last_unstale":"2024-04-26T01:37:25.501473+0000","last_undegraded":"2024-04-26T01:37:25.501473+0000","last_fullsized":"2024-04-26T01:37:25.501473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:10:12.296180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488731+0000","last_change":"2024-04-26T01:36:50.262479+0000","last_active":"2024-04-26T01:37:25.488731+0000","last_peered":"2024-04-26T01:37:25.488731+0000","last_clean":"2024-04-26T01:37:25.488731+0000","last_became_active":"2024-04-26T01:36:50.262109+0000","last_became_peered":"2024-04-26T01:36:50.262109+0000","last_unstale":"2024-04-26T01:37:25.488731+0000","last_undegraded":"2024-04-26T01:37:25.488731+0000","last_fullsized":"2024-04-26T01:37:25.488731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:19.805321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494347+0000","last_change":"2024-04-26T01:36:54.270051+0000","last_active":"2024-04-26T01:37:25.494347+0000","last_peered":"2024-04-26T01:37:25.494347+0000","last_clean":"2024-04-26T01:37:25.494347+0000","last_became_active":"2024-04-26T01:36:54.269670+0000","last_became_peered":"2024-04-26T01:36:54.269670+0000","last_unstale":"2024-04-26T01:37:25.494347+0000","last_undegraded":"2024-04-26T01:37:25.494347+0000","last_fullsized":"2024-04-26T01:37:25.494347+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:44.137454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514311+0000","last_change":"2024-04-26T01:36:50.248107+0000","last_active":"2024-04-26T01:37:25.514311+0000","last_peered":"2024-04-26T01:37:25.514311+0000","last_clean":"2024-04-26T01:37:25.514311+0000","last_became_active":"2024-04-26T01:36:50.247695+0000","last_became_peered":"2024-04-26T01:36:50.247695+0000","last_unstale":"2024-04-26T01:37:25.514311+0000","last_undegraded":"2024-04-26T01:37:25.514311+0000","last_fullsized":"2024-04-26T01:37:25.514311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:52.702512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491254+0000","last_change":"2024-04-26T01:36:54.278686+0000","last_active":"2024-04-26T01:37:25.491254+0000","last_peered":"2024-04-26T01:37:25.491254+0000","last_clean":"2024-04-26T01:37:25.491254+0000","last_became_active":"2024-04-26T01:36:54.278312+0000","last_became_peered":"2024-04-26T01:36:54.278312+0000","last_unstale":"2024-04-26T01:37:25.491254+0000","last_undegraded":"2024-04-26T01:37:25.491254+0000","last_fullsized":"2024-04-26T01:37:25.491254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:46.193032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491497+0000","last_change":"2024-04-26T01:36:50.236715+0000","last_active":"2024-04-26T01:37:25.491497+0000","last_peered":"2024-04-26T01:37:25.491497+0000","last_clean":"2024-04-26T01:37:25.491497+0000","last_became_active":"2024-04-26T01:36:50.236184+0000","last_became_peered":"2024-04-26T01:36:50.236184+0000","last_unstale":"2024-04-26T01:37:25.491497+0000","last_undegraded":"2024-04-26T01:37:25.491497+0000","last_fullsized":"2024-04-26T01:37:25.491497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:24:16.063401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514621+0000","last_change":"2024-04-26T01:36:54.290601+0000","last_active":"2024-04-26T01:37:25.514621+0000","last_peered":"2024-04-26T01:37:25.514621+0000","last_clean":"2024-04-26T01:37:25.514621+0000","last_became_active":"2024-04-26T01:36:54.290168+0000","last_became_peered":"2024-04-26T01:36:54.290168+0000","last_unstale":"2024-04-26T01:37:25.514621+0000","last_undegraded":"2024-04-26T01:37:25.514621+0000","last_fullsized":"2024-04-26T01:37:25.514621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:48.770431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494639+0000","last_change":"2024-04-26T01:36:50.235394+0000","last_active":"2024-04-26T01:37:25.494639+0000","last_peered":"2024-04-26T01:37:25.494639+0000","last_clean":"2024-04-26T01:37:25.494639+0000","last_became_active":"2024-04-26T01:36:50.235012+0000","last_became_peered":"2024-04-26T01:36:50.235012+0000","last_unstale":"2024-04-26T01:37:25.494639+0000","last_undegraded":"2024-04-26T01:37:25.494639+0000","last_fullsized":"2024-04-26T01:37:25.494639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:24:17.218945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491498+0000","last_change":"2024-04-26T01:36:54.280490+0000","last_active":"2024-04-26T01:37:25.491498+0000","last_peered":"2024-04-26T01:37:25.491498+0000","last_clean":"2024-04-26T01:37:25.491498+0000","last_became_active":"2024-04-26T01:36:54.280047+0000","last_became_peered":"2024-04-26T01:36:54.280047+0000","last_unstale":"2024-04-26T01:37:25.491498+0000","last_undegraded":"2024-04-26T01:37:25.491498+0000","last_fullsized":"2024-04-26T01:37:25.491498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:32.610972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492704+0000","last_change":"2024-04-26T01:36:50.278325+0000","last_active":"2024-04-26T01:37:25.492704+0000","last_peered":"2024-04-26T01:37:25.492704+0000","last_clean":"2024-04-26T01:37:25.492704+0000","last_became_active":"2024-04-26T01:36:50.277882+0000","last_became_peered":"2024-04-26T01:36:50.277882+0000","last_unstale":"2024-04-26T01:37:25.492704+0000","last_undegraded":"2024-04-26T01:37:25.492704+0000","last_fullsized":"2024-04-26T01:37:25.492704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:06.082223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491947+0000","last_change":"2024-04-26T01:36:54.278106+0000","last_active":"2024-04-26T01:37:25.491947+0000","last_peered":"2024-04-26T01:37:25.491947+0000","last_clean":"2024-04-26T01:37:25.491947+0000","last_became_active":"2024-04-26T01:36:54.277644+0000","last_became_peered":"2024-04-26T01:36:54.277644+0000","last_unstale":"2024-04-26T01:37:25.491947+0000","last_undegraded":"2024-04-26T01:37:25.491947+0000","last_fullsized":"2024-04-26T01:37:25.491947+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:10.619640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489417+0000","last_change":"2024-04-26T01:36:50.237072+0000","last_active":"2024-04-26T01:37:25.489417+0000","last_peered":"2024-04-26T01:37:25.489417+0000","last_clean":"2024-04-26T01:37:25.489417+0000","last_became_active":"2024-04-26T01:36:50.236682+0000","last_became_peered":"2024-04-26T01:36:50.236682+0000","last_unstale":"2024-04-26T01:37:25.489417+0000","last_undegraded":"2024-04-26T01:37:25.489417+0000","last_fullsized":"2024-04-26T01:37:25.489417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:58:17.553492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491759+0000","last_change":"2024-04-26T01:36:54.289997+0000","last_active":"2024-04-26T01:37:25.491759+0000","last_peered":"2024-04-26T01:37:25.491759+0000","last_clean":"2024-04-26T01:37:25.491759+0000","last_became_active":"2024-04-26T01:36:54.289505+0000","last_became_peered":"2024-04-26T01:36:54.289505+0000","last_unstale":"2024-04-26T01:37:25.491759+0000","last_undegraded":"2024-04-26T01:37:25.491759+0000","last_fullsized":"2024-04-26T01:37:25.491759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:09.340231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491177+0000","last_change":"2024-04-26T01:36:50.245231+0000","last_active":"2024-04-26T01:37:25.491177+0000","last_peered":"2024-04-26T01:37:25.491177+0000","last_clean":"2024-04-26T01:37:25.491177+0000","last_became_active":"2024-04-26T01:36:50.244811+0000","last_became_peered":"2024-04-26T01:36:50.244811+0000","last_unstale":"2024-04-26T01:37:25.491177+0000","last_undegraded":"2024-04-26T01:37:25.491177+0000","last_fullsized":"2024-04-26T01:37:25.491177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:12:08.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492015+0000","last_change":"2024-04-26T01:36:54.272197+0000","last_active":"2024-04-26T01:37:25.492015+0000","last_peered":"2024-04-26T01:37:25.492015+0000","last_clean":"2024-04-26T01:37:25.492015+0000","last_became_active":"2024-04-26T01:36:54.271776+0000","last_became_peered":"2024-04-26T01:36:54.271776+0000","last_unstale":"2024-04-26T01:37:25.492015+0000","last_undegraded":"2024-04-26T01:37:25.492015+0000","last_fullsized":"2024-04-26T01:37:25.492015+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:20.024890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491123+0000","last_change":"2024-04-26T01:36:54.271120+0000","last_active":"2024-04-26T01:37:25.491123+0000","last_peered":"2024-04-26T01:37:25.491123+0000","last_clean":"2024-04-26T01:37:25.491123+0000","last_became_active":"2024-04-26T01:36:54.270796+0000","last_became_peered":"2024-04-26T01:36:54.270796+0000","last_unstale":"2024-04-26T01:37:25.491123+0000","last_undegraded":"2024-04-26T01:37:25.491123+0000","last_fullsized":"2024-04-26T01:37:25.491123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:04:35.093847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492321+0000","last_change":"2024-04-26T01:36:54.273013+0000","last_active":"2024-04-26T01:37:25.492321+0000","last_peered":"2024-04-26T01:37:25.492321+0000","last_clean":"2024-04-26T01:37:25.492321+0000","last_became_active":"2024-04-26T01:36:54.272666+0000","last_became_peered":"2024-04-26T01:36:54.272666+0000","last_unstale":"2024-04-26T01:37:25.492321+0000","last_undegraded":"2024-04-26T01:37:25.492321+0000","last_fullsized":"2024-04-26T01:37:25.492321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:45.203185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491591+0000","last_change":"2024-04-26T01:36:54.289427+0000","last_active":"2024-04-26T01:37:25.491591+0000","last_peered":"2024-04-26T01:37:25.491591+0000","last_clean":"2024-04-26T01:37:25.491591+0000","last_became_active":"2024-04-26T01:36:54.289090+0000","last_became_peered":"2024-04-26T01:36:54.289090+0000","last_unstale":"2024-04-26T01:37:25.491591+0000","last_undegraded":"2024-04-26T01:37:25.491591+0000","last_fullsized":"2024-04-26T01:37:25.491591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:18:18.544777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501913+0000","last_change":"2024-04-26T01:36:54.294308+0000","last_active":"2024-04-26T01:37:25.501913+0000","last_peered":"2024-04-26T01:37:25.501913+0000","last_clean":"2024-04-26T01:37:25.501913+0000","last_became_active":"2024-04-26T01:36:54.291910+0000","last_became_peered":"2024-04-26T01:36:54.291910+0000","last_unstale":"2024-04-26T01:37:25.501913+0000","last_undegraded":"2024-04-26T01:37:25.501913+0000","last_fullsized":"2024-04-26T01:37:25.501913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:35:16.683265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491389+0000","last_change":"2024-04-26T01:36:54.285479+0000","last_active":"2024-04-26T01:37:25.491389+0000","last_peered":"2024-04-26T01:37:25.491389+0000","last_clean":"2024-04-26T01:37:25.491389+0000","last_became_active":"2024-04-26T01:36:54.284139+0000","last_became_peered":"2024-04-26T01:36:54.284139+0000","last_unstale":"2024-04-26T01:37:25.491389+0000","last_undegraded":"2024-04-26T01:37:25.491389+0000","last_fullsized":"2024-04-26T01:37:25.491389+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:55.506496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491623+0000","last_change":"2024-04-26T01:36:54.270567+0000","last_active":"2024-04-26T01:37:25.491623+0000","last_peered":"2024-04-26T01:37:25.491623+0000","last_clean":"2024-04-26T01:37:25.491623+0000","last_became_active":"2024-04-26T01:36:54.270218+0000","last_became_peered":"2024-04-26T01:36:54.270218+0000","last_unstale":"2024-04-26T01:37:25.491623+0000","last_undegraded":"2024-04-26T01:37:25.491623+0000","last_fullsized":"2024-04-26T01:37:25.491623+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:50:43.685086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492144+0000","last_change":"2024-04-26T01:36:54.269746+0000","last_active":"2024-04-26T01:37:25.492144+0000","last_peered":"2024-04-26T01:37:25.492144+0000","last_clean":"2024-04-26T01:37:25.492144+0000","last_became_active":"2024-04-26T01:36:54.269143+0000","last_became_peered":"2024-04-26T01:36:54.269143+0000","last_unstale":"2024-04-26T01:37:25.492144+0000","last_undegraded":"2024-04-26T01:37:25.492144+0000","last_fullsized":"2024-04-26T01:37:25.492144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:07:02.911645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514724+0000","last_change":"2024-04-26T01:36:54.279194+0000","last_active":"2024-04-26T01:37:25.514724+0000","last_peered":"2024-04-26T01:37:25.514724+0000","last_clean":"2024-04-26T01:37:25.514724+0000","last_became_active":"2024-04-26T01:36:54.278709+0000","last_became_peered":"2024-04-26T01:36:54.278709+0000","last_unstale":"2024-04-26T01:37:25.514724+0000","last_undegraded":"2024-04-26T01:37:25.514724+0000","last_fullsized":"2024-04-26T01:37:25.514724+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:19.604884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488990+0000","last_change":"2024-04-26T01:36:54.277540+0000","last_active":"2024-04-26T01:37:25.488990+0000","last_peered":"2024-04-26T01:37:25.488990+0000","last_clean":"2024-04-26T01:37:25.488990+0000","last_became_active":"2024-04-26T01:36:54.277193+0000","last_became_peered":"2024-04-26T01:36:54.277193+0000","last_unstale":"2024-04-26T01:37:25.488990+0000","last_undegraded":"2024-04-26T01:37:25.488990+0000","last_fullsized":"2024-04-26T01:37:25.488990+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:34.079094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514261+0000","last_change":"2024-04-26T01:36:54.294593+0000","last_active":"2024-04-26T01:37:25.514261+0000","last_peered":"2024-04-26T01:37:25.514261+0000","last_clean":"2024-04-26T01:37:25.514261+0000","last_became_active":"2024-04-26T01:36:54.291894+0000","last_became_peered":"2024-04-26T01:36:54.291894+0000","last_unstale":"2024-04-26T01:37:25.514261+0000","last_undegraded":"2024-04-26T01:37:25.514261+0000","last_fullsized":"2024-04-26T01:37:25.514261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:16.715108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488955+0000","last_change":"2024-04-26T01:36:54.292433+0000","last_active":"2024-04-26T01:37:25.488955+0000","last_peered":"2024-04-26T01:37:25.488955+0000","last_clean":"2024-04-26T01:37:25.488955+0000","last_became_active":"2024-04-26T01:36:54.291445+0000","last_became_peered":"2024-04-26T01:36:54.291445+0000","last_unstale":"2024-04-26T01:37:25.488955+0000","last_undegraded":"2024-04-26T01:37:25.488955+0000","last_fullsized":"2024-04-26T01:37:25.488955+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:49.081871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492668+0000","last_change":"2024-04-26T01:36:54.291892+0000","last_active":"2024-04-26T01:37:25.492668+0000","last_peered":"2024-04-26T01:37:25.492668+0000","last_clean":"2024-04-26T01:37:25.492668+0000","last_became_active":"2024-04-26T01:36:54.291549+0000","last_became_peered":"2024-04-26T01:36:54.291549+0000","last_unstale":"2024-04-26T01:37:25.492668+0000","last_undegraded":"2024-04-26T01:37:25.492668+0000","last_fullsized":"2024-04-26T01:37:25.492668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:09:22.954651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.498138+0000","last_change":"2024-04-26T01:37:25.498138+0000","last_active":"2024-04-26T01:37:25.498138+0000","last_peered":"2024-04-26T01:37:25.498138+0000","last_clean":"2024-04-26T01:37:25.498138+0000","last_became_active":"2024-04-26T01:37:25.497681+0000","last_became_peered":"2024-04-26T01:37:25.497681+0000","last_unstale":"2024-04-26T01:37:25.498138+0000","last_undegraded":"2024-04-26T01:37:25.498138+0000","last_fullsized":"2024-04-26T01:37:25.498138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:03:25.016623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282972,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67154,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":216,"num_read_kb":181,"num_write":166,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":192752,"data_compressed":5484,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":479903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1130496,"data_stored":1134515,"data_compressed":37816,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":480117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":480015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1071967,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":479506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":543465,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":480777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":534358,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":480055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071651,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64166,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":55059,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64731,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:37:54.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:37:54.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:54.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:54 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:54.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:54 smithi169 ceph-mon[32851]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:37:54.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:54.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:54 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:54.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:54 smithi053 ceph-mon[32593]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:37:54.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:54.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:54 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:54.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:54 smithi028 ceph-mon[28247]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:37:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:55.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:55 smithi169 ceph-mon[32851]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-26T01:37:55.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:55.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:55.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:55.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:55 smithi053 ceph-mon[32593]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-26T01:37:55.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:55.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:55.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:55.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:55 smithi028 ceph-mon[28247]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-26T01:37:55.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:55.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:55.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:37:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 -- 172.21.15.28:0/4275754077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8109fc0 msgr2=0x7fcbd8111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 --2- 172.21.15.28:0/4275754077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8109fc0 0x7fcbd8111b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcbd00099c0 tx=0x7fcbd0009cd0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 -- 172.21.15.28:0/4275754077 shutdown_connections 2024-04-26T01:37:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 --2- 172.21.15.28:0/4275754077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8109fc0 0x7fcbd8111b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 --2- 172.21.15.28:0/4275754077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd8104ee0 0x7fcbd81052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 --2- 172.21.15.28:0/4275754077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd8105890 0x7fcbd81098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 -- 172.21.15.28:0/4275754077 >> 172.21.15.28:0/4275754077 conn(0x7fcbd8100770 msgr2=0x7fcbd8102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 -- 172.21.15.28:0/4275754077 shutdown_connections 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.319+0000 7fcbde600700 1 -- 172.21.15.28:0/4275754077 wait complete. 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.320+0000 7fcbde600700 1 Processor -- start 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbde600700 1 -- start start 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbde600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8104ee0 0x7fcbd819b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbde600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd8105890 0x7fcbd819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbde600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd8109fc0 0x7fcbd819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbde600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd8114e60 con 0x7fcbd8105890 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbde600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd8114ce0 con 0x7fcbd8109fc0 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbde600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd8114fe0 con 0x7fcbd8104ee0 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbd7400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8104ee0 0x7fcbd819b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbd7400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8104ee0 0x7fcbd819b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:38224/0 (socket says 172.21.15.28:38224) 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.321+0000 7fcbd7400700 1 -- 172.21.15.28:0/1479776567 learned_addr learned my addr 172.21.15.28:0/1479776567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbd7e00700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd8109fc0 0x7fcbd819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbcea00700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd8105890 0x7fcbd819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbd7400700 1 -- 172.21.15.28:0/1479776567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd8109fc0 msgr2=0x7fcbd819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbd7400700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd8109fc0 0x7fcbd819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbd7400700 1 -- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd8105890 msgr2=0x7fcbd819b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbd7400700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd8105890 0x7fcbd819b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbd7400700 1 -- 172.21.15.28:0/1479776567 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd81a02c0 con 0x7fcbd8104ee0 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbcea00700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd8105890 0x7fcbd819b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbd7400700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8104ee0 0x7fcbd819b2b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800c370 tx=0x7fcbc800c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.322+0000 7fcbd4c00700 1 -- 172.21.15.28:0/1479776567 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc800cea0 con 0x7fcbd8104ee0 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.323+0000 7fcbd4c00700 1 -- 172.21.15.28:0/1479776567 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbc80090d0 con 0x7fcbd8104ee0 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.323+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd81a0550 con 0x7fcbd8104ee0 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.323+0000 7fcbd4c00700 1 -- 172.21.15.28:0/1479776567 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8004770 con 0x7fcbd8104ee0 2024-04-26T01:37:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.323+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd81a7e30 con 0x7fcbd8104ee0 2024-04-26T01:37:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.324+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd8069730 con 0x7fcbd8104ee0 2024-04-26T01:37:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.326+0000 7fcbd4c00700 1 -- 172.21.15.28:0/1479776567 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcbc801a020 con 0x7fcbd8104ee0 2024-04-26T01:37:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.327+0000 7fcbd4c00700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcbbc0a3d50 0x7fcbbc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.328+0000 7fcbd4c00700 1 -- 172.21.15.28:0/1479776567 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fcbc80d0130 con 0x7fcbd8104ee0 2024-04-26T01:37:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.328+0000 7fcbcea00700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcbbc0a3d50 0x7fcbbc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.328+0000 7fcbcea00700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcbbc0a3d50 0x7fcbbc0a6210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8103e60 tx=0x7fcbc0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:57.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.330+0000 7fcbd4c00700 1 -- 172.21.15.28:0/1479776567 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbc8010040 con 0x7fcbd8104ee0 2024-04-26T01:37:57.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.562+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcbd8062e80 con 0x7fcbbc0a3d50 2024-04-26T01:37:57.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.586+0000 7fcbd4c00700 1 -- 172.21.15.28:0/1479776567 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258951 (secure 0 0 0) 0x7fcbd8062e80 con 0x7fcbbc0a3d50 2024-04-26T01:37:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:37:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.592+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcbbc0a3d50 msgr2=0x7fcbbc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.592+0000 7fcbde600700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcbbc0a3d50 0x7fcbbc0a6210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8103e60 tx=0x7fcbc0009260 comp rx=0 tx=0).stop 2024-04-26T01:37:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.592+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8104ee0 msgr2=0x7fcbd819b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.592+0000 7fcbde600700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8104ee0 0x7fcbd819b2b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800c370 tx=0x7fcbc800c680 comp rx=0 tx=0).stop 2024-04-26T01:37:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.593+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 shutdown_connections 2024-04-26T01:37:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.593+0000 7fcbde600700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcbbc0a3d50 0x7fcbbc0a6210 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.593+0000 7fcbde600700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd8109fc0 0x7fcbd819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.593+0000 7fcbde600700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbd8104ee0 0x7fcbd819b2b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.593+0000 7fcbde600700 1 --2- 172.21.15.28:0/1479776567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd8105890 0x7fcbd819b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.593+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 >> 172.21.15.28:0/1479776567 conn(0x7fcbd8100770 msgr2=0x7fcbd8101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.593+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 shutdown_connections 2024-04-26T01:37:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:37:57.593+0000 7fcbde600700 1 -- 172.21.15.28:0/1479776567 wait complete. 2024-04-26T01:37:57.590 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-26T01:37:57.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:57 smithi169 ceph-mon[32851]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:37:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:57 smithi053 ceph-mon[32593]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:37:57.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:57 smithi028 ceph-mon[28247]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:37:58.550 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-04-26T01:37:55.783031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657756,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":382,"num_read_kb":315,"num_write":349,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181492,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377812,"kb_used_data":12000,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564012,"statfs":{"total":1151940427776,"available":1151553548288,"internally_reserved":0,"allocated":12288000,"data_stored":7724856,"data_compressed":113448,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":374453564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17566,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":63,"num_write":10,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005871"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492333+0000","last_change":"2024-04-26T01:36:54.288460+0000","last_active":"2024-04-26T01:37:25.492333+0000","last_peered":"2024-04-26T01:37:25.492333+0000","last_clean":"2024-04-26T01:37:25.492333+0000","last_became_active":"2024-04-26T01:36:54.288033+0000","last_became_peered":"2024-04-26T01:36:54.288033+0000","last_unstale":"2024-04-26T01:37:25.492333+0000","last_undegraded":"2024-04-26T01:37:25.492333+0000","last_fullsized":"2024-04-26T01:37:25.492333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:33.737622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501153+0000","last_change":"2024-04-26T01:36:54.287367+0000","last_active":"2024-04-26T01:37:25.501153+0000","last_peered":"2024-04-26T01:37:25.501153+0000","last_clean":"2024-04-26T01:37:25.501153+0000","last_became_active":"2024-04-26T01:36:54.287024+0000","last_became_peered":"2024-04-26T01:36:54.287024+0000","last_unstale":"2024-04-26T01:37:25.501153+0000","last_undegraded":"2024-04-26T01:37:25.501153+0000","last_fullsized":"2024-04-26T01:37:25.501153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:31.014627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.502368+0000","last_change":"2024-04-26T01:36:54.288433+0000","last_active":"2024-04-26T01:37:25.502368+0000","last_peered":"2024-04-26T01:37:25.502368+0000","last_clean":"2024-04-26T01:37:25.502368+0000","last_became_active":"2024-04-26T01:36:54.288002+0000","last_became_peered":"2024-04-26T01:36:54.288002+0000","last_unstale":"2024-04-26T01:37:25.502368+0000","last_undegraded":"2024-04-26T01:37:25.502368+0000","last_fullsized":"2024-04-26T01:37:25.502368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:37.885179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.498038+0000","last_change":"2024-04-26T01:36:54.273857+0000","last_active":"2024-04-26T01:37:25.498038+0000","last_peered":"2024-04-26T01:37:25.498038+0000","last_clean":"2024-04-26T01:37:25.498038+0000","last_became_active":"2024-04-26T01:36:54.272376+0000","last_became_peered":"2024-04-26T01:36:54.272376+0000","last_unstale":"2024-04-26T01:37:25.498038+0000","last_undegraded":"2024-04-26T01:37:25.498038+0000","last_fullsized":"2024-04-26T01:37:25.498038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:30.110044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494496+0000","last_change":"2024-04-26T01:36:54.288606+0000","last_active":"2024-04-26T01:37:25.494496+0000","last_peered":"2024-04-26T01:37:25.494496+0000","last_clean":"2024-04-26T01:37:25.494496+0000","last_became_active":"2024-04-26T01:36:54.288247+0000","last_became_peered":"2024-04-26T01:36:54.288247+0000","last_unstale":"2024-04-26T01:37:25.494496+0000","last_undegraded":"2024-04-26T01:37:25.494496+0000","last_fullsized":"2024-04-26T01:37:25.494496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:47:56.944164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494229+0000","last_change":"2024-04-26T01:36:54.277701+0000","last_active":"2024-04-26T01:37:25.494229+0000","last_peered":"2024-04-26T01:37:25.494229+0000","last_clean":"2024-04-26T01:37:25.494229+0000","last_became_active":"2024-04-26T01:36:54.277306+0000","last_became_peered":"2024-04-26T01:36:54.277306+0000","last_unstale":"2024-04-26T01:37:25.494229+0000","last_undegraded":"2024-04-26T01:37:25.494229+0000","last_fullsized":"2024-04-26T01:37:25.494229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:42:05.058525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491329+0000","last_change":"2024-04-26T01:36:54.274090+0000","last_active":"2024-04-26T01:37:25.491329+0000","last_peered":"2024-04-26T01:37:25.491329+0000","last_clean":"2024-04-26T01:37:25.491329+0000","last_became_active":"2024-04-26T01:36:54.273700+0000","last_became_peered":"2024-04-26T01:36:54.273700+0000","last_unstale":"2024-04-26T01:37:25.491329+0000","last_undegraded":"2024-04-26T01:37:25.491329+0000","last_fullsized":"2024-04-26T01:37:25.491329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:32.207534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501321+0000","last_change":"2024-04-26T01:37:25.499759+0000","last_active":"2024-04-26T01:37:25.501321+0000","last_peered":"2024-04-26T01:37:25.501321+0000","last_clean":"2024-04-26T01:37:25.501321+0000","last_became_active":"2024-04-26T01:37:25.499317+0000","last_became_peered":"2024-04-26T01:37:25.499317+0000","last_unstale":"2024-04-26T01:37:25.501321+0000","last_undegraded":"2024-04-26T01:37:25.501321+0000","last_fullsized":"2024-04-26T01:37:25.501321+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:39.972315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489354+0000","last_change":"2024-04-26T01:36:54.277457+0000","last_active":"2024-04-26T01:37:25.489354+0000","last_peered":"2024-04-26T01:37:25.489354+0000","last_clean":"2024-04-26T01:37:25.489354+0000","last_became_active":"2024-04-26T01:36:54.277118+0000","last_became_peered":"2024-04-26T01:36:54.277118+0000","last_unstale":"2024-04-26T01:37:25.489354+0000","last_undegraded":"2024-04-26T01:37:25.489354+0000","last_fullsized":"2024-04-26T01:37:25.489354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:05.628027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495077+0000","last_change":"2024-04-26T01:36:54.278494+0000","last_active":"2024-04-26T01:37:25.495077+0000","last_peered":"2024-04-26T01:37:25.495077+0000","last_clean":"2024-04-26T01:37:25.495077+0000","last_became_active":"2024-04-26T01:36:54.278188+0000","last_became_peered":"2024-04-26T01:36:54.278188+0000","last_unstale":"2024-04-26T01:37:25.495077+0000","last_undegraded":"2024-04-26T01:37:25.495077+0000","last_fullsized":"2024-04-26T01:37:25.495077+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:59.150771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491742+0000","last_change":"2024-04-26T01:36:54.277424+0000","last_active":"2024-04-26T01:37:25.491742+0000","last_peered":"2024-04-26T01:37:25.491742+0000","last_clean":"2024-04-26T01:37:25.491742+0000","last_became_active":"2024-04-26T01:36:54.275752+0000","last_became_peered":"2024-04-26T01:36:54.275752+0000","last_unstale":"2024-04-26T01:37:25.491742+0000","last_undegraded":"2024-04-26T01:37:25.491742+0000","last_fullsized":"2024-04-26T01:37:25.491742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:22.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.503692+0000","last_change":"2024-04-26T01:37:25.502448+0000","last_active":"2024-04-26T01:37:25.503692+0000","last_peered":"2024-04-26T01:37:25.503692+0000","last_clean":"2024-04-26T01:37:25.503692+0000","last_became_active":"2024-04-26T01:37:25.501161+0000","last_became_peered":"2024-04-26T01:37:25.501161+0000","last_unstale":"2024-04-26T01:37:25.503692+0000","last_undegraded":"2024-04-26T01:37:25.503692+0000","last_fullsized":"2024-04-26T01:37:25.503692+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:41:40.443780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492930+0000","last_change":"2024-04-26T01:36:54.268721+0000","last_active":"2024-04-26T01:37:25.492930+0000","last_peered":"2024-04-26T01:37:25.492930+0000","last_clean":"2024-04-26T01:37:25.492930+0000","last_became_active":"2024-04-26T01:36:54.268007+0000","last_became_peered":"2024-04-26T01:36:54.268007+0000","last_unstale":"2024-04-26T01:37:25.492930+0000","last_undegraded":"2024-04-26T01:37:25.492930+0000","last_fullsized":"2024-04-26T01:37:25.492930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:55.120130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.500807+0000","last_change":"2024-04-26T01:36:54.286595+0000","last_active":"2024-04-26T01:37:25.500807+0000","last_peered":"2024-04-26T01:37:25.500807+0000","last_clean":"2024-04-26T01:37:25.500807+0000","last_became_active":"2024-04-26T01:36:54.286236+0000","last_became_peered":"2024-04-26T01:36:54.286236+0000","last_unstale":"2024-04-26T01:37:25.500807+0000","last_undegraded":"2024-04-26T01:37:25.500807+0000","last_fullsized":"2024-04-26T01:37:25.500807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:59.443467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.500156+0000","last_change":"2024-04-26T01:36:54.293254+0000","last_active":"2024-04-26T01:37:25.500156+0000","last_peered":"2024-04-26T01:37:25.500156+0000","last_clean":"2024-04-26T01:37:25.500156+0000","last_became_active":"2024-04-26T01:36:54.292853+0000","last_became_peered":"2024-04-26T01:36:54.292853+0000","last_unstale":"2024-04-26T01:37:25.500156+0000","last_undegraded":"2024-04-26T01:37:25.500156+0000","last_fullsized":"2024-04-26T01:37:25.500156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:19.844006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488715+0000","last_change":"2024-04-26T01:36:54.285346+0000","last_active":"2024-04-26T01:37:25.488715+0000","last_peered":"2024-04-26T01:37:25.488715+0000","last_clean":"2024-04-26T01:37:25.488715+0000","last_became_active":"2024-04-26T01:36:54.284949+0000","last_became_peered":"2024-04-26T01:36:54.284949+0000","last_unstale":"2024-04-26T01:37:25.488715+0000","last_undegraded":"2024-04-26T01:37:25.488715+0000","last_fullsized":"2024-04-26T01:37:25.488715+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:33:29.276240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492076+0000","last_change":"2024-04-26T01:36:54.283161+0000","last_active":"2024-04-26T01:37:25.492076+0000","last_peered":"2024-04-26T01:37:25.492076+0000","last_clean":"2024-04-26T01:37:25.492076+0000","last_became_active":"2024-04-26T01:36:54.282908+0000","last_became_peered":"2024-04-26T01:36:54.282908+0000","last_unstale":"2024-04-26T01:37:25.492076+0000","last_undegraded":"2024-04-26T01:37:25.492076+0000","last_fullsized":"2024-04-26T01:37:25.492076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:26.503964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497700+0000","last_change":"2024-04-26T01:36:54.282516+0000","last_active":"2024-04-26T01:37:25.497700+0000","last_peered":"2024-04-26T01:37:25.497700+0000","last_clean":"2024-04-26T01:37:25.497700+0000","last_became_active":"2024-04-26T01:36:54.278634+0000","last_became_peered":"2024-04-26T01:36:54.278634+0000","last_unstale":"2024-04-26T01:37:25.497700+0000","last_undegraded":"2024-04-26T01:37:25.497700+0000","last_fullsized":"2024-04-26T01:37:25.497700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:10:38.275720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491997+0000","last_change":"2024-04-26T01:36:54.289502+0000","last_active":"2024-04-26T01:37:25.491997+0000","last_peered":"2024-04-26T01:37:25.491997+0000","last_clean":"2024-04-26T01:37:25.491997+0000","last_became_active":"2024-04-26T01:36:54.289079+0000","last_became_peered":"2024-04-26T01:36:54.289079+0000","last_unstale":"2024-04-26T01:37:25.491997+0000","last_undegraded":"2024-04-26T01:37:25.491997+0000","last_fullsized":"2024-04-26T01:37:25.491997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:51.412032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.498019+0000","last_change":"2024-04-26T01:36:50.243164+0000","last_active":"2024-04-26T01:37:25.498019+0000","last_peered":"2024-04-26T01:37:25.498019+0000","last_clean":"2024-04-26T01:37:25.498019+0000","last_became_active":"2024-04-26T01:36:50.242508+0000","last_became_peered":"2024-04-26T01:36:50.242508+0000","last_unstale":"2024-04-26T01:37:25.498019+0000","last_undegraded":"2024-04-26T01:37:25.498019+0000","last_fullsized":"2024-04-26T01:37:25.498019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:06:01.662698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514971+0000","last_change":"2024-04-26T01:36:54.279809+0000","last_active":"2024-04-26T01:37:25.514971+0000","last_peered":"2024-04-26T01:37:25.514971+0000","last_clean":"2024-04-26T01:37:25.514971+0000","last_became_active":"2024-04-26T01:36:54.279509+0000","last_became_peered":"2024-04-26T01:36:54.279509+0000","last_unstale":"2024-04-26T01:37:25.514971+0000","last_undegraded":"2024-04-26T01:37:25.514971+0000","last_fullsized":"2024-04-26T01:37:25.514971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:54:26.734136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.515137+0000","last_change":"2024-04-26T01:36:50.261942+0000","last_active":"2024-04-26T01:37:25.515137+0000","last_peered":"2024-04-26T01:37:25.515137+0000","last_clean":"2024-04-26T01:37:25.515137+0000","last_became_active":"2024-04-26T01:36:50.261564+0000","last_became_peered":"2024-04-26T01:36:50.261564+0000","last_unstale":"2024-04-26T01:37:25.515137+0000","last_undegraded":"2024-04-26T01:37:25.515137+0000","last_fullsized":"2024-04-26T01:37:25.515137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:25.102557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501782+0000","last_change":"2024-04-26T01:36:54.288011+0000","last_active":"2024-04-26T01:37:25.501782+0000","last_peered":"2024-04-26T01:37:25.501782+0000","last_clean":"2024-04-26T01:37:25.501782+0000","last_became_active":"2024-04-26T01:36:54.286853+0000","last_became_peered":"2024-04-26T01:36:54.286853+0000","last_unstale":"2024-04-26T01:37:25.501782+0000","last_undegraded":"2024-04-26T01:37:25.501782+0000","last_fullsized":"2024-04-26T01:37:25.501782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:53.422470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495834+0000","last_change":"2024-04-26T01:36:50.235291+0000","last_active":"2024-04-26T01:37:25.495834+0000","last_peered":"2024-04-26T01:37:25.495834+0000","last_clean":"2024-04-26T01:37:25.495834+0000","last_became_active":"2024-04-26T01:36:50.234939+0000","last_became_peered":"2024-04-26T01:36:50.234939+0000","last_unstale":"2024-04-26T01:37:25.495834+0000","last_undegraded":"2024-04-26T01:37:25.495834+0000","last_fullsized":"2024-04-26T01:37:25.495834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:13.048185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488964+0000","last_change":"2024-04-26T01:36:54.282731+0000","last_active":"2024-04-26T01:37:25.488964+0000","last_peered":"2024-04-26T01:37:25.488964+0000","last_clean":"2024-04-26T01:37:25.488964+0000","last_became_active":"2024-04-26T01:36:54.282384+0000","last_became_peered":"2024-04-26T01:36:54.282384+0000","last_unstale":"2024-04-26T01:37:25.488964+0000","last_undegraded":"2024-04-26T01:37:25.488964+0000","last_fullsized":"2024-04-26T01:37:25.488964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:09.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495596+0000","last_change":"2024-04-26T01:36:50.233581+0000","last_active":"2024-04-26T01:37:25.495596+0000","last_peered":"2024-04-26T01:37:25.495596+0000","last_clean":"2024-04-26T01:37:25.495596+0000","last_became_active":"2024-04-26T01:36:50.233210+0000","last_became_peered":"2024-04-26T01:36:50.233210+0000","last_unstale":"2024-04-26T01:37:25.495596+0000","last_undegraded":"2024-04-26T01:37:25.495596+0000","last_fullsized":"2024-04-26T01:37:25.495596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:01.223954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489527+0000","last_change":"2024-04-26T01:36:54.274657+0000","last_active":"2024-04-26T01:37:25.489527+0000","last_peered":"2024-04-26T01:37:25.489527+0000","last_clean":"2024-04-26T01:37:25.489527+0000","last_became_active":"2024-04-26T01:36:54.274333+0000","last_became_peered":"2024-04-26T01:36:54.274333+0000","last_unstale":"2024-04-26T01:37:25.489527+0000","last_undegraded":"2024-04-26T01:37:25.489527+0000","last_fullsized":"2024-04-26T01:37:25.489527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:37.243307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:50.560508+0000","last_change":"2024-04-26T01:37:25.502311+0000","last_active":"2024-04-26T01:37:50.560508+0000","last_peered":"2024-04-26T01:37:50.560508+0000","last_clean":"2024-04-26T01:37:50.560508+0000","last_became_active":"2024-04-26T01:37:25.493259+0000","last_became_peered":"2024-04-26T01:37:25.493259+0000","last_unstale":"2024-04-26T01:37:50.560508+0000","last_undegraded":"2024-04-26T01:37:50.560508+0000","last_fullsized":"2024-04-26T01:37:50.560508+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:46:49.224905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494982+0000","last_change":"2024-04-26T01:36:54.275116+0000","last_active":"2024-04-26T01:37:25.494982+0000","last_peered":"2024-04-26T01:37:25.494982+0000","last_clean":"2024-04-26T01:37:25.494982+0000","last_became_active":"2024-04-26T01:36:54.274815+0000","last_became_peered":"2024-04-26T01:36:54.274815+0000","last_unstale":"2024-04-26T01:37:25.494982+0000","last_undegraded":"2024-04-26T01:37:25.494982+0000","last_fullsized":"2024-04-26T01:37:25.494982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:04.427670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.505327+0000","last_change":"2024-04-26T01:37:25.505327+0000","last_active":"2024-04-26T01:37:25.505327+0000","last_peered":"2024-04-26T01:37:25.505327+0000","last_clean":"2024-04-26T01:37:25.505327+0000","last_became_active":"2024-04-26T01:37:25.497881+0000","last_became_peered":"2024-04-26T01:37:25.497881+0000","last_unstale":"2024-04-26T01:37:25.505327+0000","last_undegraded":"2024-04-26T01:37:25.505327+0000","last_fullsized":"2024-04-26T01:37:25.505327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:09.181372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489215+0000","last_change":"2024-04-26T01:36:54.276415+0000","last_active":"2024-04-26T01:37:25.489215+0000","last_peered":"2024-04-26T01:37:25.489215+0000","last_clean":"2024-04-26T01:37:25.489215+0000","last_became_active":"2024-04-26T01:36:54.276080+0000","last_became_peered":"2024-04-26T01:36:54.276080+0000","last_unstale":"2024-04-26T01:37:25.489215+0000","last_undegraded":"2024-04-26T01:37:25.489215+0000","last_fullsized":"2024-04-26T01:37:25.489215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:08:17.171620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491798+0000","last_change":"2024-04-26T01:36:50.241552+0000","last_active":"2024-04-26T01:37:25.491798+0000","last_peered":"2024-04-26T01:37:25.491798+0000","last_clean":"2024-04-26T01:37:25.491798+0000","last_became_active":"2024-04-26T01:36:50.241022+0000","last_became_peered":"2024-04-26T01:36:50.241022+0000","last_unstale":"2024-04-26T01:37:25.491798+0000","last_undegraded":"2024-04-26T01:37:25.491798+0000","last_fullsized":"2024-04-26T01:37:25.491798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:55.503831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514256+0000","last_change":"2024-04-26T01:36:54.276641+0000","last_active":"2024-04-26T01:37:25.514256+0000","last_peered":"2024-04-26T01:37:25.514256+0000","last_clean":"2024-04-26T01:37:25.514256+0000","last_became_active":"2024-04-26T01:36:54.276309+0000","last_became_peered":"2024-04-26T01:36:54.276309+0000","last_unstale":"2024-04-26T01:37:25.514256+0000","last_undegraded":"2024-04-26T01:37:25.514256+0000","last_fullsized":"2024-04-26T01:37:25.514256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:53.725584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488938+0000","last_change":"2024-04-26T01:36:50.235504+0000","last_active":"2024-04-26T01:37:25.488938+0000","last_peered":"2024-04-26T01:37:25.488938+0000","last_clean":"2024-04-26T01:37:25.488938+0000","last_became_active":"2024-04-26T01:36:50.235161+0000","last_became_peered":"2024-04-26T01:36:50.235161+0000","last_unstale":"2024-04-26T01:37:25.488938+0000","last_undegraded":"2024-04-26T01:37:25.488938+0000","last_fullsized":"2024-04-26T01:37:25.488938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:05.951521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494601+0000","last_change":"2024-04-26T01:36:54.298236+0000","last_active":"2024-04-26T01:37:25.494601+0000","last_peered":"2024-04-26T01:37:25.494601+0000","last_clean":"2024-04-26T01:37:25.494601+0000","last_became_active":"2024-04-26T01:36:54.297850+0000","last_became_peered":"2024-04-26T01:36:54.297850+0000","last_unstale":"2024-04-26T01:37:25.494601+0000","last_undegraded":"2024-04-26T01:37:25.494601+0000","last_fullsized":"2024-04-26T01:37:25.494601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:35:35.244502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:53.566254+0000","last_change":"2024-04-26T01:36:50.241418+0000","last_active":"2024-04-26T01:37:53.566254+0000","last_peered":"2024-04-26T01:37:53.566254+0000","last_clean":"2024-04-26T01:37:53.566254+0000","last_became_active":"2024-04-26T01:36:50.241044+0000","last_became_peered":"2024-04-26T01:36:50.241044+0000","last_unstale":"2024-04-26T01:37:53.566254+0000","last_undegraded":"2024-04-26T01:37:53.566254+0000","last_fullsized":"2024-04-26T01:37:53.566254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:13.317953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489357+0000","last_change":"2024-04-26T01:36:54.284410+0000","last_active":"2024-04-26T01:37:25.489357+0000","last_peered":"2024-04-26T01:37:25.489357+0000","last_clean":"2024-04-26T01:37:25.489357+0000","last_became_active":"2024-04-26T01:36:54.283629+0000","last_became_peered":"2024-04-26T01:36:54.283629+0000","last_unstale":"2024-04-26T01:37:25.489357+0000","last_undegraded":"2024-04-26T01:37:25.489357+0000","last_fullsized":"2024-04-26T01:37:25.489357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:15.380006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.500348+0000","last_change":"2024-04-26T01:37:25.499874+0000","last_active":"2024-04-26T01:37:25.500348+0000","last_peered":"2024-04-26T01:37:25.500348+0000","last_clean":"2024-04-26T01:37:25.500348+0000","last_became_active":"2024-04-26T01:37:25.494878+0000","last_became_peered":"2024-04-26T01:37:25.494878+0000","last_unstale":"2024-04-26T01:37:25.500348+0000","last_undegraded":"2024-04-26T01:37:25.500348+0000","last_fullsized":"2024-04-26T01:37:25.500348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:48:12.678331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.502056+0000","last_change":"2024-04-26T01:36:54.285118+0000","last_active":"2024-04-26T01:37:25.502056+0000","last_peered":"2024-04-26T01:37:25.502056+0000","last_clean":"2024-04-26T01:37:25.502056+0000","last_became_active":"2024-04-26T01:36:54.284554+0000","last_became_peered":"2024-04-26T01:36:54.284554+0000","last_unstale":"2024-04-26T01:37:25.502056+0000","last_undegraded":"2024-04-26T01:37:25.502056+0000","last_fullsized":"2024-04-26T01:37:25.502056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:13.721684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489680+0000","last_change":"2024-04-26T01:36:50.262451+0000","last_active":"2024-04-26T01:37:25.489680+0000","last_peered":"2024-04-26T01:37:25.489680+0000","last_clean":"2024-04-26T01:37:25.489680+0000","last_became_active":"2024-04-26T01:36:50.262056+0000","last_became_peered":"2024-04-26T01:36:50.262056+0000","last_unstale":"2024-04-26T01:37:25.489680+0000","last_undegraded":"2024-04-26T01:37:25.489680+0000","last_fullsized":"2024-04-26T01:37:25.489680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:50.813675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488943+0000","last_change":"2024-04-26T01:36:54.283975+0000","last_active":"2024-04-26T01:37:25.488943+0000","last_peered":"2024-04-26T01:37:25.488943+0000","last_clean":"2024-04-26T01:37:25.488943+0000","last_became_active":"2024-04-26T01:36:54.283564+0000","last_became_peered":"2024-04-26T01:36:54.283564+0000","last_unstale":"2024-04-26T01:37:25.488943+0000","last_undegraded":"2024-04-26T01:37:25.488943+0000","last_fullsized":"2024-04-26T01:37:25.488943+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:53.881763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:54.498115+0000","last_change":"2024-04-26T01:36:50.238839+0000","last_active":"2024-04-26T01:37:54.498115+0000","last_peered":"2024-04-26T01:37:54.498115+0000","last_clean":"2024-04-26T01:37:54.498115+0000","last_became_active":"2024-04-26T01:36:50.238452+0000","last_became_peered":"2024-04-26T01:36:50.238452+0000","last_unstale":"2024-04-26T01:37:54.498115+0000","last_undegraded":"2024-04-26T01:37:54.498115+0000","last_fullsized":"2024-04-26T01:37:54.498115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:05:40.426825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63153,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":84,"num_read_kb":103,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495641+0000","last_change":"2024-04-26T01:36:54.271446+0000","last_active":"2024-04-26T01:37:25.495641+0000","last_peered":"2024-04-26T01:37:25.495641+0000","last_clean":"2024-04-26T01:37:25.495641+0000","last_became_active":"2024-04-26T01:36:54.271122+0000","last_became_peered":"2024-04-26T01:36:54.271122+0000","last_unstale":"2024-04-26T01:37:25.495641+0000","last_undegraded":"2024-04-26T01:37:25.495641+0000","last_fullsized":"2024-04-26T01:37:25.495641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:28:24.275481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491672+0000","last_change":"2024-04-26T01:36:50.237726+0000","last_active":"2024-04-26T01:37:25.491672+0000","last_peered":"2024-04-26T01:37:25.491672+0000","last_clean":"2024-04-26T01:37:25.491672+0000","last_became_active":"2024-04-26T01:36:50.237215+0000","last_became_peered":"2024-04-26T01:36:50.237215+0000","last_unstale":"2024-04-26T01:37:25.491672+0000","last_undegraded":"2024-04-26T01:37:25.491672+0000","last_fullsized":"2024-04-26T01:37:25.491672+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:41.781876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491099+0000","last_change":"2024-04-26T01:36:54.290616+0000","last_active":"2024-04-26T01:37:25.491099+0000","last_peered":"2024-04-26T01:37:25.491099+0000","last_clean":"2024-04-26T01:37:25.491099+0000","last_became_active":"2024-04-26T01:36:54.290293+0000","last_became_peered":"2024-04-26T01:36:54.290293+0000","last_unstale":"2024-04-26T01:37:25.491099+0000","last_undegraded":"2024-04-26T01:37:25.491099+0000","last_fullsized":"2024-04-26T01:37:25.491099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:07.439333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489132+0000","last_change":"2024-04-26T01:36:50.237578+0000","last_active":"2024-04-26T01:37:25.489132+0000","last_peered":"2024-04-26T01:37:25.489132+0000","last_clean":"2024-04-26T01:37:25.489132+0000","last_became_active":"2024-04-26T01:36:50.237224+0000","last_became_peered":"2024-04-26T01:36:50.237224+0000","last_unstale":"2024-04-26T01:37:25.489132+0000","last_undegraded":"2024-04-26T01:37:25.489132+0000","last_fullsized":"2024-04-26T01:37:25.489132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:46:38.078631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491672+0000","last_change":"2024-04-26T01:36:54.280818+0000","last_active":"2024-04-26T01:37:25.491672+0000","last_peered":"2024-04-26T01:37:25.491672+0000","last_clean":"2024-04-26T01:37:25.491672+0000","last_became_active":"2024-04-26T01:36:54.280440+0000","last_became_peered":"2024-04-26T01:36:54.280440+0000","last_unstale":"2024-04-26T01:37:25.491672+0000","last_undegraded":"2024-04-26T01:37:25.491672+0000","last_fullsized":"2024-04-26T01:37:25.491672+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:47.185219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:52.518580+0000","last_change":"2024-04-26T01:36:50.234602+0000","last_active":"2024-04-26T01:37:52.518580+0000","last_peered":"2024-04-26T01:37:52.518580+0000","last_clean":"2024-04-26T01:37:52.518580+0000","last_became_active":"2024-04-26T01:36:50.234175+0000","last_became_peered":"2024-04-26T01:36:50.234175+0000","last_unstale":"2024-04-26T01:37:52.518580+0000","last_undegraded":"2024-04-26T01:37:52.518580+0000","last_fullsized":"2024-04-26T01:37:52.518580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:17:14.236069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491239+0000","last_change":"2024-04-26T01:36:54.271466+0000","last_active":"2024-04-26T01:37:25.491239+0000","last_peered":"2024-04-26T01:37:25.491239+0000","last_clean":"2024-04-26T01:37:25.491239+0000","last_became_active":"2024-04-26T01:36:54.271086+0000","last_became_peered":"2024-04-26T01:36:54.271086+0000","last_unstale":"2024-04-26T01:37:25.491239+0000","last_undegraded":"2024-04-26T01:37:25.491239+0000","last_fullsized":"2024-04-26T01:37:25.491239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:41.052895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.518730+0000","last_change":"2024-04-26T01:37:25.518730+0000","last_active":"2024-04-26T01:37:25.518730+0000","last_peered":"2024-04-26T01:37:25.518730+0000","last_clean":"2024-04-26T01:37:25.518730+0000","last_became_active":"2024-04-26T01:37:25.504464+0000","last_became_peered":"2024-04-26T01:37:25.504464+0000","last_unstale":"2024-04-26T01:37:25.518730+0000","last_undegraded":"2024-04-26T01:37:25.518730+0000","last_fullsized":"2024-04-26T01:37:25.518730+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:07.578917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488951+0000","last_change":"2024-04-26T01:36:54.273621+0000","last_active":"2024-04-26T01:37:25.488951+0000","last_peered":"2024-04-26T01:37:25.488951+0000","last_clean":"2024-04-26T01:37:25.488951+0000","last_became_active":"2024-04-26T01:36:54.273218+0000","last_became_peered":"2024-04-26T01:36:54.273218+0000","last_unstale":"2024-04-26T01:37:25.488951+0000","last_undegraded":"2024-04-26T01:37:25.488951+0000","last_fullsized":"2024-04-26T01:37:25.488951+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:55.543943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.509030+0000","last_change":"2024-04-26T01:37:25.508575+0000","last_active":"2024-04-26T01:37:25.509030+0000","last_peered":"2024-04-26T01:37:25.509030+0000","last_clean":"2024-04-26T01:37:25.509030+0000","last_became_active":"2024-04-26T01:37:25.496407+0000","last_became_peered":"2024-04-26T01:37:25.496407+0000","last_unstale":"2024-04-26T01:37:25.509030+0000","last_undegraded":"2024-04-26T01:37:25.509030+0000","last_fullsized":"2024-04-26T01:37:25.509030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:05.878252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492135+0000","last_change":"2024-04-26T01:36:54.272147+0000","last_active":"2024-04-26T01:37:25.492135+0000","last_peered":"2024-04-26T01:37:25.492135+0000","last_clean":"2024-04-26T01:37:25.492135+0000","last_became_active":"2024-04-26T01:36:54.271737+0000","last_became_peered":"2024-04-26T01:36:54.271737+0000","last_unstale":"2024-04-26T01:37:25.492135+0000","last_undegraded":"2024-04-26T01:37:25.492135+0000","last_fullsized":"2024-04-26T01:37:25.492135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:41.445073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.496852+0000","last_change":"2024-04-26T01:36:50.264444+0000","last_active":"2024-04-26T01:37:25.496852+0000","last_peered":"2024-04-26T01:37:25.496852+0000","last_clean":"2024-04-26T01:37:25.496852+0000","last_became_active":"2024-04-26T01:36:50.264058+0000","last_became_peered":"2024-04-26T01:36:50.264058+0000","last_unstale":"2024-04-26T01:37:25.496852+0000","last_undegraded":"2024-04-26T01:37:25.496852+0000","last_fullsized":"2024-04-26T01:37:25.496852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:56:15.428588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.493923+0000","last_change":"2024-04-26T01:36:54.289537+0000","last_active":"2024-04-26T01:37:25.493923+0000","last_peered":"2024-04-26T01:37:25.493923+0000","last_clean":"2024-04-26T01:37:25.493923+0000","last_became_active":"2024-04-26T01:36:54.289149+0000","last_became_peered":"2024-04-26T01:36:54.289149+0000","last_unstale":"2024-04-26T01:37:25.493923+0000","last_undegraded":"2024-04-26T01:37:25.493923+0000","last_fullsized":"2024-04-26T01:37:25.493923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:07.801551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491498+0000","last_change":"2024-04-26T01:36:50.242190+0000","last_active":"2024-04-26T01:37:25.491498+0000","last_peered":"2024-04-26T01:37:25.491498+0000","last_clean":"2024-04-26T01:37:25.491498+0000","last_became_active":"2024-04-26T01:36:50.241890+0000","last_became_peered":"2024-04-26T01:36:50.241890+0000","last_unstale":"2024-04-26T01:37:25.491498+0000","last_undegraded":"2024-04-26T01:37:25.491498+0000","last_fullsized":"2024-04-26T01:37:25.491498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:29.472502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494503+0000","last_change":"2024-04-26T01:36:54.294699+0000","last_active":"2024-04-26T01:37:25.494503+0000","last_peered":"2024-04-26T01:37:25.494503+0000","last_clean":"2024-04-26T01:37:25.494503+0000","last_became_active":"2024-04-26T01:36:54.294313+0000","last_became_peered":"2024-04-26T01:36:54.294313+0000","last_unstale":"2024-04-26T01:37:25.494503+0000","last_undegraded":"2024-04-26T01:37:25.494503+0000","last_fullsized":"2024-04-26T01:37:25.494503+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:15.026357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489176+0000","last_change":"2024-04-26T01:36:54.276798+0000","last_active":"2024-04-26T01:37:25.489176+0000","last_peered":"2024-04-26T01:37:25.489176+0000","last_clean":"2024-04-26T01:37:25.489176+0000","last_became_active":"2024-04-26T01:36:54.275955+0000","last_became_peered":"2024-04-26T01:36:54.275955+0000","last_unstale":"2024-04-26T01:37:25.489176+0000","last_undegraded":"2024-04-26T01:37:25.489176+0000","last_fullsized":"2024-04-26T01:37:25.489176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:17.020909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:52.517635+0000","last_change":"2024-04-26T01:36:50.263356+0000","last_active":"2024-04-26T01:37:52.517635+0000","last_peered":"2024-04-26T01:37:52.517635+0000","last_clean":"2024-04-26T01:37:52.517635+0000","last_became_active":"2024-04-26T01:36:50.261596+0000","last_became_peered":"2024-04-26T01:36:50.261596+0000","last_unstale":"2024-04-26T01:37:52.517635+0000","last_undegraded":"2024-04-26T01:37:52.517635+0000","last_fullsized":"2024-04-26T01:37:52.517635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:10.773347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514523+0000","last_change":"2024-04-26T01:36:54.288135+0000","last_active":"2024-04-26T01:37:25.514523+0000","last_peered":"2024-04-26T01:37:25.514523+0000","last_clean":"2024-04-26T01:37:25.514523+0000","last_became_active":"2024-04-26T01:36:54.287728+0000","last_became_peered":"2024-04-26T01:36:54.287728+0000","last_unstale":"2024-04-26T01:37:25.514523+0000","last_undegraded":"2024-04-26T01:37:25.514523+0000","last_fullsized":"2024-04-26T01:37:25.514523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:18.428151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497273+0000","last_change":"2024-04-26T01:36:50.243522+0000","last_active":"2024-04-26T01:37:25.497273+0000","last_peered":"2024-04-26T01:37:25.497273+0000","last_clean":"2024-04-26T01:37:25.497273+0000","last_became_active":"2024-04-26T01:36:50.243084+0000","last_became_peered":"2024-04-26T01:36:50.243084+0000","last_unstale":"2024-04-26T01:37:25.497273+0000","last_undegraded":"2024-04-26T01:37:25.497273+0000","last_fullsized":"2024-04-26T01:37:25.497273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:03.159750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491542+0000","last_change":"2024-04-26T01:35:13.920486+0000","last_active":"2024-04-26T01:37:25.491542+0000","last_peered":"2024-04-26T01:37:25.491542+0000","last_clean":"2024-04-26T01:37:25.491542+0000","last_became_active":"2024-04-26T01:35:13.919834+0000","last_became_peered":"2024-04-26T01:35:13.919834+0000","last_unstale":"2024-04-26T01:37:25.491542+0000","last_undegraded":"2024-04-26T01:37:25.491542+0000","last_fullsized":"2024-04-26T01:37:25.491542+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_clean_scrub_stamp":"2024-04-26T01:31:12.563114+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:53.935491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.490631+0000","last_change":"2024-04-26T01:36:54.270553+0000","last_active":"2024-04-26T01:37:25.490631+0000","last_peered":"2024-04-26T01:37:25.490631+0000","last_clean":"2024-04-26T01:37:25.490631+0000","last_became_active":"2024-04-26T01:36:54.269986+0000","last_became_peered":"2024-04-26T01:36:54.269986+0000","last_unstale":"2024-04-26T01:37:25.490631+0000","last_undegraded":"2024-04-26T01:37:25.490631+0000","last_fullsized":"2024-04-26T01:37:25.490631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:19.597337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491547+0000","last_change":"2024-04-26T01:36:50.235583+0000","last_active":"2024-04-26T01:37:25.491547+0000","last_peered":"2024-04-26T01:37:25.491547+0000","last_clean":"2024-04-26T01:37:25.491547+0000","last_became_active":"2024-04-26T01:36:50.235214+0000","last_became_peered":"2024-04-26T01:36:50.235214+0000","last_unstale":"2024-04-26T01:37:25.491547+0000","last_undegraded":"2024-04-26T01:37:25.491547+0000","last_fullsized":"2024-04-26T01:37:25.491547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:05.822872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495673+0000","last_change":"2024-04-26T01:36:50.260988+0000","last_active":"2024-04-26T01:37:25.495673+0000","last_peered":"2024-04-26T01:37:25.495673+0000","last_clean":"2024-04-26T01:37:25.495673+0000","last_became_active":"2024-04-26T01:36:50.260615+0000","last_became_peered":"2024-04-26T01:36:50.260615+0000","last_unstale":"2024-04-26T01:37:25.495673+0000","last_undegraded":"2024-04-26T01:37:25.495673+0000","last_fullsized":"2024-04-26T01:37:25.495673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:15.933939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492283+0000","last_change":"2024-04-26T01:36:54.279650+0000","last_active":"2024-04-26T01:37:25.492283+0000","last_peered":"2024-04-26T01:37:25.492283+0000","last_clean":"2024-04-26T01:37:25.492283+0000","last_became_active":"2024-04-26T01:36:54.275814+0000","last_became_peered":"2024-04-26T01:36:54.275814+0000","last_unstale":"2024-04-26T01:37:25.492283+0000","last_undegraded":"2024-04-26T01:37:25.492283+0000","last_fullsized":"2024-04-26T01:37:25.492283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:58:42.369511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:43.307474+0000","last_change":"2024-04-26T01:36:50.248179+0000","last_active":"2024-04-26T01:37:43.307474+0000","last_peered":"2024-04-26T01:37:43.307474+0000","last_clean":"2024-04-26T01:37:43.307474+0000","last_became_active":"2024-04-26T01:36:50.247805+0000","last_became_peered":"2024-04-26T01:36:50.247805+0000","last_unstale":"2024-04-26T01:37:43.307474+0000","last_undegraded":"2024-04-26T01:37:43.307474+0000","last_fullsized":"2024-04-26T01:37:43.307474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:19:23.740920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514830+0000","last_change":"2024-04-26T01:36:54.276626+0000","last_active":"2024-04-26T01:37:25.514830+0000","last_peered":"2024-04-26T01:37:25.514830+0000","last_clean":"2024-04-26T01:37:25.514830+0000","last_became_active":"2024-04-26T01:36:54.276238+0000","last_became_peered":"2024-04-26T01:36:54.276238+0000","last_unstale":"2024-04-26T01:37:25.514830+0000","last_undegraded":"2024-04-26T01:37:25.514830+0000","last_fullsized":"2024-04-26T01:37:25.514830+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:57.306950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497759+0000","last_change":"2024-04-26T01:36:50.278502+0000","last_active":"2024-04-26T01:37:25.497759+0000","last_peered":"2024-04-26T01:37:25.497759+0000","last_clean":"2024-04-26T01:37:25.497759+0000","last_became_active":"2024-04-26T01:36:50.278215+0000","last_became_peered":"2024-04-26T01:36:50.278215+0000","last_unstale":"2024-04-26T01:37:25.497759+0000","last_undegraded":"2024-04-26T01:37:25.497759+0000","last_fullsized":"2024-04-26T01:37:25.497759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:05:46.142902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501473+0000","last_change":"2024-04-26T01:36:54.292600+0000","last_active":"2024-04-26T01:37:25.501473+0000","last_peered":"2024-04-26T01:37:25.501473+0000","last_clean":"2024-04-26T01:37:25.501473+0000","last_became_active":"2024-04-26T01:36:54.292260+0000","last_became_peered":"2024-04-26T01:36:54.292260+0000","last_unstale":"2024-04-26T01:37:25.501473+0000","last_undegraded":"2024-04-26T01:37:25.501473+0000","last_fullsized":"2024-04-26T01:37:25.501473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:10:12.296180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488731+0000","last_change":"2024-04-26T01:36:50.262479+0000","last_active":"2024-04-26T01:37:25.488731+0000","last_peered":"2024-04-26T01:37:25.488731+0000","last_clean":"2024-04-26T01:37:25.488731+0000","last_became_active":"2024-04-26T01:36:50.262109+0000","last_became_peered":"2024-04-26T01:36:50.262109+0000","last_unstale":"2024-04-26T01:37:25.488731+0000","last_undegraded":"2024-04-26T01:37:25.488731+0000","last_fullsized":"2024-04-26T01:37:25.488731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:19.805321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494347+0000","last_change":"2024-04-26T01:36:54.270051+0000","last_active":"2024-04-26T01:37:25.494347+0000","last_peered":"2024-04-26T01:37:25.494347+0000","last_clean":"2024-04-26T01:37:25.494347+0000","last_became_active":"2024-04-26T01:36:54.269670+0000","last_became_peered":"2024-04-26T01:36:54.269670+0000","last_unstale":"2024-04-26T01:37:25.494347+0000","last_undegraded":"2024-04-26T01:37:25.494347+0000","last_fullsized":"2024-04-26T01:37:25.494347+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:44.137454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514311+0000","last_change":"2024-04-26T01:36:50.248107+0000","last_active":"2024-04-26T01:37:25.514311+0000","last_peered":"2024-04-26T01:37:25.514311+0000","last_clean":"2024-04-26T01:37:25.514311+0000","last_became_active":"2024-04-26T01:36:50.247695+0000","last_became_peered":"2024-04-26T01:36:50.247695+0000","last_unstale":"2024-04-26T01:37:25.514311+0000","last_undegraded":"2024-04-26T01:37:25.514311+0000","last_fullsized":"2024-04-26T01:37:25.514311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:52.702512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491254+0000","last_change":"2024-04-26T01:36:54.278686+0000","last_active":"2024-04-26T01:37:25.491254+0000","last_peered":"2024-04-26T01:37:25.491254+0000","last_clean":"2024-04-26T01:37:25.491254+0000","last_became_active":"2024-04-26T01:36:54.278312+0000","last_became_peered":"2024-04-26T01:36:54.278312+0000","last_unstale":"2024-04-26T01:37:25.491254+0000","last_undegraded":"2024-04-26T01:37:25.491254+0000","last_fullsized":"2024-04-26T01:37:25.491254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:46.193032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491497+0000","last_change":"2024-04-26T01:36:50.236715+0000","last_active":"2024-04-26T01:37:25.491497+0000","last_peered":"2024-04-26T01:37:25.491497+0000","last_clean":"2024-04-26T01:37:25.491497+0000","last_became_active":"2024-04-26T01:36:50.236184+0000","last_became_peered":"2024-04-26T01:36:50.236184+0000","last_unstale":"2024-04-26T01:37:25.491497+0000","last_undegraded":"2024-04-26T01:37:25.491497+0000","last_fullsized":"2024-04-26T01:37:25.491497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:24:16.063401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514621+0000","last_change":"2024-04-26T01:36:54.290601+0000","last_active":"2024-04-26T01:37:25.514621+0000","last_peered":"2024-04-26T01:37:25.514621+0000","last_clean":"2024-04-26T01:37:25.514621+0000","last_became_active":"2024-04-26T01:36:54.290168+0000","last_became_peered":"2024-04-26T01:36:54.290168+0000","last_unstale":"2024-04-26T01:37:25.514621+0000","last_undegraded":"2024-04-26T01:37:25.514621+0000","last_fullsized":"2024-04-26T01:37:25.514621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:48.770431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494639+0000","last_change":"2024-04-26T01:36:50.235394+0000","last_active":"2024-04-26T01:37:25.494639+0000","last_peered":"2024-04-26T01:37:25.494639+0000","last_clean":"2024-04-26T01:37:25.494639+0000","last_became_active":"2024-04-26T01:36:50.235012+0000","last_became_peered":"2024-04-26T01:36:50.235012+0000","last_unstale":"2024-04-26T01:37:25.494639+0000","last_undegraded":"2024-04-26T01:37:25.494639+0000","last_fullsized":"2024-04-26T01:37:25.494639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:24:17.218945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491498+0000","last_change":"2024-04-26T01:36:54.280490+0000","last_active":"2024-04-26T01:37:25.491498+0000","last_peered":"2024-04-26T01:37:25.491498+0000","last_clean":"2024-04-26T01:37:25.491498+0000","last_became_active":"2024-04-26T01:36:54.280047+0000","last_became_peered":"2024-04-26T01:36:54.280047+0000","last_unstale":"2024-04-26T01:37:25.491498+0000","last_undegraded":"2024-04-26T01:37:25.491498+0000","last_fullsized":"2024-04-26T01:37:25.491498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:32.610972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492704+0000","last_change":"2024-04-26T01:36:50.278325+0000","last_active":"2024-04-26T01:37:25.492704+0000","last_peered":"2024-04-26T01:37:25.492704+0000","last_clean":"2024-04-26T01:37:25.492704+0000","last_became_active":"2024-04-26T01:36:50.277882+0000","last_became_peered":"2024-04-26T01:36:50.277882+0000","last_unstale":"2024-04-26T01:37:25.492704+0000","last_undegraded":"2024-04-26T01:37:25.492704+0000","last_fullsized":"2024-04-26T01:37:25.492704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:06.082223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491947+0000","last_change":"2024-04-26T01:36:54.278106+0000","last_active":"2024-04-26T01:37:25.491947+0000","last_peered":"2024-04-26T01:37:25.491947+0000","last_clean":"2024-04-26T01:37:25.491947+0000","last_became_active":"2024-04-26T01:36:54.277644+0000","last_became_peered":"2024-04-26T01:36:54.277644+0000","last_unstale":"2024-04-26T01:37:25.491947+0000","last_undegraded":"2024-04-26T01:37:25.491947+0000","last_fullsized":"2024-04-26T01:37:25.491947+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:10.619640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489417+0000","last_change":"2024-04-26T01:36:50.237072+0000","last_active":"2024-04-26T01:37:25.489417+0000","last_peered":"2024-04-26T01:37:25.489417+0000","last_clean":"2024-04-26T01:37:25.489417+0000","last_became_active":"2024-04-26T01:36:50.236682+0000","last_became_peered":"2024-04-26T01:36:50.236682+0000","last_unstale":"2024-04-26T01:37:25.489417+0000","last_undegraded":"2024-04-26T01:37:25.489417+0000","last_fullsized":"2024-04-26T01:37:25.489417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:58:17.553492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491759+0000","last_change":"2024-04-26T01:36:54.289997+0000","last_active":"2024-04-26T01:37:25.491759+0000","last_peered":"2024-04-26T01:37:25.491759+0000","last_clean":"2024-04-26T01:37:25.491759+0000","last_became_active":"2024-04-26T01:36:54.289505+0000","last_became_peered":"2024-04-26T01:36:54.289505+0000","last_unstale":"2024-04-26T01:37:25.491759+0000","last_undegraded":"2024-04-26T01:37:25.491759+0000","last_fullsized":"2024-04-26T01:37:25.491759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:09.340231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491177+0000","last_change":"2024-04-26T01:36:50.245231+0000","last_active":"2024-04-26T01:37:25.491177+0000","last_peered":"2024-04-26T01:37:25.491177+0000","last_clean":"2024-04-26T01:37:25.491177+0000","last_became_active":"2024-04-26T01:36:50.244811+0000","last_became_peered":"2024-04-26T01:36:50.244811+0000","last_unstale":"2024-04-26T01:37:25.491177+0000","last_undegraded":"2024-04-26T01:37:25.491177+0000","last_fullsized":"2024-04-26T01:37:25.491177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:12:08.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492015+0000","last_change":"2024-04-26T01:36:54.272197+0000","last_active":"2024-04-26T01:37:25.492015+0000","last_peered":"2024-04-26T01:37:25.492015+0000","last_clean":"2024-04-26T01:37:25.492015+0000","last_became_active":"2024-04-26T01:36:54.271776+0000","last_became_peered":"2024-04-26T01:36:54.271776+0000","last_unstale":"2024-04-26T01:37:25.492015+0000","last_undegraded":"2024-04-26T01:37:25.492015+0000","last_fullsized":"2024-04-26T01:37:25.492015+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:20.024890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491123+0000","last_change":"2024-04-26T01:36:54.271120+0000","last_active":"2024-04-26T01:37:25.491123+0000","last_peered":"2024-04-26T01:37:25.491123+0000","last_clean":"2024-04-26T01:37:25.491123+0000","last_became_active":"2024-04-26T01:36:54.270796+0000","last_became_peered":"2024-04-26T01:36:54.270796+0000","last_unstale":"2024-04-26T01:37:25.491123+0000","last_undegraded":"2024-04-26T01:37:25.491123+0000","last_fullsized":"2024-04-26T01:37:25.491123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:04:35.093847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492321+0000","last_change":"2024-04-26T01:36:54.273013+0000","last_active":"2024-04-26T01:37:25.492321+0000","last_peered":"2024-04-26T01:37:25.492321+0000","last_clean":"2024-04-26T01:37:25.492321+0000","last_became_active":"2024-04-26T01:36:54.272666+0000","last_became_peered":"2024-04-26T01:36:54.272666+0000","last_unstale":"2024-04-26T01:37:25.492321+0000","last_undegraded":"2024-04-26T01:37:25.492321+0000","last_fullsized":"2024-04-26T01:37:25.492321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:45.203185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491591+0000","last_change":"2024-04-26T01:36:54.289427+0000","last_active":"2024-04-26T01:37:25.491591+0000","last_peered":"2024-04-26T01:37:25.491591+0000","last_clean":"2024-04-26T01:37:25.491591+0000","last_became_active":"2024-04-26T01:36:54.289090+0000","last_became_peered":"2024-04-26T01:36:54.289090+0000","last_unstale":"2024-04-26T01:37:25.491591+0000","last_undegraded":"2024-04-26T01:37:25.491591+0000","last_fullsized":"2024-04-26T01:37:25.491591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:18:18.544777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.501913+0000","last_change":"2024-04-26T01:36:54.294308+0000","last_active":"2024-04-26T01:37:25.501913+0000","last_peered":"2024-04-26T01:37:25.501913+0000","last_clean":"2024-04-26T01:37:25.501913+0000","last_became_active":"2024-04-26T01:36:54.291910+0000","last_became_peered":"2024-04-26T01:36:54.291910+0000","last_unstale":"2024-04-26T01:37:25.501913+0000","last_undegraded":"2024-04-26T01:37:25.501913+0000","last_fullsized":"2024-04-26T01:37:25.501913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:35:16.683265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491389+0000","last_change":"2024-04-26T01:36:54.285479+0000","last_active":"2024-04-26T01:37:25.491389+0000","last_peered":"2024-04-26T01:37:25.491389+0000","last_clean":"2024-04-26T01:37:25.491389+0000","last_became_active":"2024-04-26T01:36:54.284139+0000","last_became_peered":"2024-04-26T01:36:54.284139+0000","last_unstale":"2024-04-26T01:37:25.491389+0000","last_undegraded":"2024-04-26T01:37:25.491389+0000","last_fullsized":"2024-04-26T01:37:25.491389+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:55.506496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491623+0000","last_change":"2024-04-26T01:36:54.270567+0000","last_active":"2024-04-26T01:37:25.491623+0000","last_peered":"2024-04-26T01:37:25.491623+0000","last_clean":"2024-04-26T01:37:25.491623+0000","last_became_active":"2024-04-26T01:36:54.270218+0000","last_became_peered":"2024-04-26T01:36:54.270218+0000","last_unstale":"2024-04-26T01:37:25.491623+0000","last_undegraded":"2024-04-26T01:37:25.491623+0000","last_fullsized":"2024-04-26T01:37:25.491623+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:50:43.685086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492144+0000","last_change":"2024-04-26T01:36:54.269746+0000","last_active":"2024-04-26T01:37:25.492144+0000","last_peered":"2024-04-26T01:37:25.492144+0000","last_clean":"2024-04-26T01:37:25.492144+0000","last_became_active":"2024-04-26T01:36:54.269143+0000","last_became_peered":"2024-04-26T01:36:54.269143+0000","last_unstale":"2024-04-26T01:37:25.492144+0000","last_undegraded":"2024-04-26T01:37:25.492144+0000","last_fullsized":"2024-04-26T01:37:25.492144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:07:02.911645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514724+0000","last_change":"2024-04-26T01:36:54.279194+0000","last_active":"2024-04-26T01:37:25.514724+0000","last_peered":"2024-04-26T01:37:25.514724+0000","last_clean":"2024-04-26T01:37:25.514724+0000","last_became_active":"2024-04-26T01:36:54.278709+0000","last_became_peered":"2024-04-26T01:36:54.278709+0000","last_unstale":"2024-04-26T01:37:25.514724+0000","last_undegraded":"2024-04-26T01:37:25.514724+0000","last_fullsized":"2024-04-26T01:37:25.514724+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:19.604884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488990+0000","last_change":"2024-04-26T01:36:54.277540+0000","last_active":"2024-04-26T01:37:25.488990+0000","last_peered":"2024-04-26T01:37:25.488990+0000","last_clean":"2024-04-26T01:37:25.488990+0000","last_became_active":"2024-04-26T01:36:54.277193+0000","last_became_peered":"2024-04-26T01:36:54.277193+0000","last_unstale":"2024-04-26T01:37:25.488990+0000","last_undegraded":"2024-04-26T01:37:25.488990+0000","last_fullsized":"2024-04-26T01:37:25.488990+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:34.079094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514261+0000","last_change":"2024-04-26T01:36:54.294593+0000","last_active":"2024-04-26T01:37:25.514261+0000","last_peered":"2024-04-26T01:37:25.514261+0000","last_clean":"2024-04-26T01:37:25.514261+0000","last_became_active":"2024-04-26T01:36:54.291894+0000","last_became_peered":"2024-04-26T01:36:54.291894+0000","last_unstale":"2024-04-26T01:37:25.514261+0000","last_undegraded":"2024-04-26T01:37:25.514261+0000","last_fullsized":"2024-04-26T01:37:25.514261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:16.715108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488955+0000","last_change":"2024-04-26T01:36:54.292433+0000","last_active":"2024-04-26T01:37:25.488955+0000","last_peered":"2024-04-26T01:37:25.488955+0000","last_clean":"2024-04-26T01:37:25.488955+0000","last_became_active":"2024-04-26T01:36:54.291445+0000","last_became_peered":"2024-04-26T01:36:54.291445+0000","last_unstale":"2024-04-26T01:37:25.488955+0000","last_undegraded":"2024-04-26T01:37:25.488955+0000","last_fullsized":"2024-04-26T01:37:25.488955+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:49.081871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492668+0000","last_change":"2024-04-26T01:36:54.291892+0000","last_active":"2024-04-26T01:37:25.492668+0000","last_peered":"2024-04-26T01:37:25.492668+0000","last_clean":"2024-04-26T01:37:25.492668+0000","last_became_active":"2024-04-26T01:36:54.291549+0000","last_became_peered":"2024-04-26T01:36:54.291549+0000","last_unstale":"2024-04-26T01:37:25.492668+0000","last_undegraded":"2024-04-26T01:37:25.492668+0000","last_fullsized":"2024-04-26T01:37:25.492668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:09:22.954651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.498138+0000","last_change":"2024-04-26T01:37:25.498138+0000","last_active":"2024-04-26T01:37:25.498138+0000","last_peered":"2024-04-26T01:37:25.498138+0000","last_clean":"2024-04-26T01:37:25.498138+0000","last_became_active":"2024-04-26T01:37:25.497681+0000","last_became_peered":"2024-04-26T01:37:25.497681+0000","last_unstale":"2024-04-26T01:37:25.498138+0000","last_undegraded":"2024-04-26T01:37:25.498138+0000","last_fullsized":"2024-04-26T01:37:25.498138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:03:25.016623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282972,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67154,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":246,"num_read_kb":196,"num_write":166,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":201462,"data_compressed":5484,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":479903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1130496,"data_stored":1134515,"data_compressed":37816,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":480117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":480015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1071967,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":479506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":543465,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":480777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":543068,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":480055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071651,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64166,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63769,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64731,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:37:58.563 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:37:58.564 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T01:37:58.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T01:37:58.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:58 smithi169 ceph-mon[32851]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:37:58.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:58 smithi028 ceph-mon[28247]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:37:58.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:58 smithi053 ceph-mon[32593]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:37:59.073 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:59.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:37:59 smithi028 ceph-mon[28247]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:37:59.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:37:59 smithi053 ceph-mon[32593]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:37:59.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:37:59 smithi169 ceph-mon[32851]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:38:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.313+0000 7fc257a00700 1 -- 172.21.15.28:0/1809440189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250076a10 msgr2=0x7fc250074e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.313+0000 7fc257a00700 1 --2- 172.21.15.28:0/1809440189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250076a10 0x7fc250074e70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc240009990 tx=0x7fc240009ca0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.314+0000 7fc257a00700 1 -- 172.21.15.28:0/1809440189 shutdown_connections 2024-04-26T01:38:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.314+0000 7fc257a00700 1 --2- 172.21.15.28:0/1809440189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc25010a9d0 0x7fc25010ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.314+0000 7fc257a00700 1 --2- 172.21.15.28:0/1809440189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250076a10 0x7fc250074e70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.314+0000 7fc257a00700 1 --2- 172.21.15.28:0/1809440189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc250075440 0x7fc2500758c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.314+0000 7fc257a00700 1 -- 172.21.15.28:0/1809440189 >> 172.21.15.28:0/1809440189 conn(0x7fc2500fe090 msgr2=0x7fc2501004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:01.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.314+0000 7fc257a00700 1 -- 172.21.15.28:0/1809440189 shutdown_connections 2024-04-26T01:38:01.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.314+0000 7fc257a00700 1 -- 172.21.15.28:0/1809440189 wait complete. 2024-04-26T01:38:01.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.315+0000 7fc257a00700 1 Processor -- start 2024-04-26T01:38:01.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.315+0000 7fc257a00700 1 -- start start 2024-04-26T01:38:01.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.315+0000 7fc257a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250075440 0x7fc25019ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:01.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.315+0000 7fc257a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250076a10 0x7fc25019b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:01.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.315+0000 7fc257a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25010a9d0 0x7fc2501a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc257a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc250110740 con 0x7fc25010a9d0 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc257a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc2501105c0 con 0x7fc250075440 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc257a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc2501108c0 con 0x7fc250076a10 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25010a9d0 0x7fc2501a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250075440 0x7fc25019ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25010a9d0 0x7fc2501a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57804/0 (socket says 172.21.15.28:57804) 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255a00700 1 -- 172.21.15.28:0/1482468689 learned_addr learned my addr 172.21.15.28:0/1482468689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc247e00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250076a10 0x7fc25019b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255000700 1 -- 172.21.15.28:0/1482468689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250076a10 msgr2=0x7fc25019b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255000700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250076a10 0x7fc25019b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255000700 1 -- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25010a9d0 msgr2=0x7fc2501a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255000700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25010a9d0 0x7fc2501a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255000700 1 -- 172.21.15.28:0/1482468689 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2501a2970 con 0x7fc250075440 2024-04-26T01:38:01.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.316+0000 7fc255a00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25010a9d0 0x7fc2501a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:01.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.317+0000 7fc255000700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250075440 0x7fc25019ad20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2400098e0 tx=0x7fc240004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:01.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.317+0000 7fc24e000700 1 -- 172.21.15.28:0/1482468689 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24000fb10 con 0x7fc250075440 2024-04-26T01:38:01.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.317+0000 7fc24e000700 1 -- 172.21.15.28:0/1482468689 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc24000fcb0 con 0x7fc250075440 2024-04-26T01:38:01.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.317+0000 7fc24e000700 1 -- 172.21.15.28:0/1482468689 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2400055f0 con 0x7fc250075440 2024-04-26T01:38:01.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.317+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2501a2c00 con 0x7fc250075440 2024-04-26T01:38:01.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.317+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2501a2fa0 con 0x7fc250075440 2024-04-26T01:38:01.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.318+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc25002df80 con 0x7fc250075440 2024-04-26T01:38:01.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.321+0000 7fc24e000700 1 -- 172.21.15.28:0/1482468689 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc240028060 con 0x7fc250075440 2024-04-26T01:38:01.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.322+0000 7fc24e000700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2340a3e00 0x7fc2340a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.322+0000 7fc24e000700 1 -- 172.21.15.28:0/1482468689 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fc2400d13c0 con 0x7fc250075440 2024-04-26T01:38:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.322+0000 7fc247e00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2340a3e00 0x7fc2340a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.323+0000 7fc247e00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2340a3e00 0x7fc2340a62c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc238009680 tx=0x7fc238009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:01.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.324+0000 7fc24e000700 1 -- 172.21.15.28:0/1482468689 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc24005cff0 con 0x7fc250075440 2024-04-26T01:38:01.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.649+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fc250063650 con 0x7fc250075440 2024-04-26T01:38:01.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.658+0000 7fc24e000700 1 -- 172.21.15.28:0/1482468689 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fc240013070 con 0x7fc250075440 2024-04-26T01:38:01.655 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-26T01:38:01.655 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQB5BStm2QvgJhAAuFFr+GNVkaLIUGkhw0JJPg== 2024-04-26T01:38:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.660+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2340a3e00 msgr2=0x7fc2340a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.660+0000 7fc257a00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2340a3e00 0x7fc2340a62c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc238009680 tx=0x7fc238009260 comp rx=0 tx=0).stop 2024-04-26T01:38:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250075440 msgr2=0x7fc25019ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250075440 0x7fc25019ad20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2400098e0 tx=0x7fc240004790 comp rx=0 tx=0).stop 2024-04-26T01:38:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 shutdown_connections 2024-04-26T01:38:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2340a3e00 0x7fc2340a62c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25010a9d0 0x7fc2501a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc250075440 0x7fc25019ad20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 --2- 172.21.15.28:0/1482468689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250076a10 0x7fc25019b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 >> 172.21.15.28:0/1482468689 conn(0x7fc2500fe090 msgr2=0x7fc250106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 shutdown_connections 2024-04-26T01:38:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:01.661+0000 7fc257a00700 1 -- 172.21.15.28:0/1482468689 wait complete. 2024-04-26T01:38:01.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:01 smithi028 ceph-mon[28247]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:38:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:01 smithi053 ceph-mon[32593]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:38:01.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:01 smithi169 ceph-mon[32851]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:38:02.473 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:38:02.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T01:38:02.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T01:38:02.605 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T01:38:02.605 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T01:38:02.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T01:38:02.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:02 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1482468689' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:38:02.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:02 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:38:02.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:02 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:38:02.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:02 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1482468689' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:38:02.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:02 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:38:02.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:02 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:38:02.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:02 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1482468689' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:38:02.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:02 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:38:02.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:02 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:38:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:03.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:03 smithi028 ceph-mon[28247]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T01:38:03.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:03 smithi053 ceph-mon[32593]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T01:38:03.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:03 smithi169 ceph-mon[32851]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T01:38:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.447+0000 7f371c000700 1 -- 172.21.15.28:0/3946835308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371410f4f0 msgr2=0x7f37141118e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.447+0000 7f371c000700 1 --2- 172.21.15.28:0/3946835308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371410f4f0 0x7f37141118e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f370c00b0a0 tx=0x7f370c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.447+0000 7f371c000700 1 -- 172.21.15.28:0/3946835308 shutdown_connections 2024-04-26T01:38:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.447+0000 7f371c000700 1 --2- 172.21.15.28:0/3946835308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371410f4f0 0x7f37141118e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.447+0000 7f371c000700 1 --2- 172.21.15.28:0/3946835308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f371406a750 0x7f371406ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.447+0000 7f371c000700 1 --2- 172.21.15.28:0/3946835308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371406b070 0x7f371410efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.447+0000 7f371c000700 1 -- 172.21.15.28:0/3946835308 >> 172.21.15.28:0/3946835308 conn(0x7f37140788f0 msgr2=0x7f3714078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:05.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.448+0000 7f371c000700 1 -- 172.21.15.28:0/3946835308 shutdown_connections 2024-04-26T01:38:05.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.448+0000 7f371c000700 1 -- 172.21.15.28:0/3946835308 wait complete. 2024-04-26T01:38:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.449+0000 7f371c000700 1 Processor -- start 2024-04-26T01:38:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.449+0000 7f371c000700 1 -- start start 2024-04-26T01:38:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f371c000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f371406a750 0x7f371419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f371c000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371406b070 0x7f371419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f371c000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371410f4f0 0x7f371419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f371c000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3714114e10 con 0x7f371410f4f0 2024-04-26T01:38:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f371c000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3714114c90 con 0x7f371406a750 2024-04-26T01:38:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f371c000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3714114f90 con 0x7f371406b070 2024-04-26T01:38:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f3718c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371406b070 0x7f371419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f3719600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f371406a750 0x7f371419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f3718c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371406b070 0x7f371419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58584/0 (socket says 172.21.15.28:58584) 2024-04-26T01:38:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f3718c00700 1 -- 172.21.15.28:0/2760068908 learned_addr learned my addr 172.21.15.28:0/2760068908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f3719600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f371406a750 0x7f371419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:42244/0 (socket says 172.21.15.28:42244) 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.450+0000 7f371a000700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371410f4f0 0x7f371419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3718c00700 1 -- 172.21.15.28:0/2760068908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f371406a750 msgr2=0x7f371419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3718c00700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f371406a750 0x7f371419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3718c00700 1 -- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371410f4f0 msgr2=0x7f371419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3718c00700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371410f4f0 0x7f371419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3718c00700 1 -- 172.21.15.28:0/2760068908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37141a0330 con 0x7f371406b070 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3718c00700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371406b070 0x7f371419b860 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f370000e9c0 tx=0x7f370000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3712000700 1 -- 172.21.15.28:0/2760068908 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370000cbd0 con 0x7f371406b070 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3712000700 1 -- 172.21.15.28:0/2760068908 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f370000ee60 con 0x7f371406b070 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f3712000700 1 -- 172.21.15.28:0/2760068908 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000055c0 con 0x7f371406b070 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.451+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37141a0620 con 0x7f371406b070 2024-04-26T01:38:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.452+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37141a7ea0 con 0x7f371406b070 2024-04-26T01:38:05.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.453+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3714069730 con 0x7f371406b070 2024-04-26T01:38:05.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.455+0000 7f3712000700 1 -- 172.21.15.28:0/2760068908 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3700004750 con 0x7f371406b070 2024-04-26T01:38:05.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.456+0000 7f3712000700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f36fc0a3d50 0x7f36fc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:05.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.456+0000 7f3712000700 1 -- 172.21.15.28:0/2760068908 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f37000d0730 con 0x7f371406b070 2024-04-26T01:38:05.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.457+0000 7f3719600700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f36fc0a3d50 0x7f36fc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.457+0000 7f3719600700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f36fc0a3d50 0x7f36fc0a6210 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3708009680 tx=0x7f3708009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.461+0000 7f3712000700 1 -- 172.21.15.28:0/2760068908 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f370009cf60 con 0x7f371406b070 2024-04-26T01:38:05.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:05 smithi053 ceph-mon[32593]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.758+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f371402df80 con 0x7f371406b070 2024-04-26T01:38:05.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:05 smithi028 ceph-mon[28247]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.764+0000 7f3712000700 1 -- 172.21.15.28:0/2760068908 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264997 (secure 0 0 0) 0x7f370009c860 con 0x7f371406b070 2024-04-26T01:38:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:38:05.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f36fc0a3d50 msgr2=0x7f36fc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:05.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f36fc0a3d50 0x7f36fc0a6210 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3708009680 tx=0x7f3708009260 comp rx=0 tx=0).stop 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371406b070 msgr2=0x7f371419b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371406b070 0x7f371419b860 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f370000e9c0 tx=0x7f370000ecd0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 shutdown_connections 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f36fc0a3d50 0x7f36fc0a6210 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371410f4f0 0x7f371419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f371406a750 0x7f371419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 --2- 172.21.15.28:0/2760068908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371406b070 0x7f371419b860 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 >> 172.21.15.28:0/2760068908 conn(0x7f37140788f0 msgr2=0x7f3714110940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 shutdown_connections 2024-04-26T01:38:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:05.769+0000 7f371c000700 1 -- 172.21.15.28:0/2760068908 wait complete. 2024-04-26T01:38:05.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:05 smithi169 ceph-mon[32851]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:06.580 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3431203710},{"type":"v1","addr":"172.21.15.28:6801","nonce":3431203710}]},"active_addr":"172.21.15.28:6801/3431203710","active_change":"2024-04-26T01:28:23.601551+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":218486566}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3010978828}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3441292878}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1295766352}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1153275446}]}]} 2024-04-26T01:38:06.585 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T01:38:06.585 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T01:38:06.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:38:06.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:06 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2760068908' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:38:06.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:06 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2760068908' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:38:06.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:06 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2760068908' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:38:07.033 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:07 smithi053 ceph-mon[32593]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:07 smithi028 ceph-mon[28247]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:07.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:07 smithi169 ceph-mon[32851]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.369+0000 7f0a31c00700 1 -- 172.21.15.28:0/846888166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 msgr2=0x7f0a2c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.369+0000 7f0a31c00700 1 --2- 172.21.15.28:0/846888166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 0x7f0a2c108e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0098f0 tx=0x7f0a1c009c00 comp rx=0 tx=0).stop 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.370+0000 7f0a31c00700 1 -- 172.21.15.28:0/846888166 shutdown_connections 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.370+0000 7f0a31c00700 1 --2- 172.21.15.28:0/846888166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c103450 0x7f0a2c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.370+0000 7f0a31c00700 1 --2- 172.21.15.28:0/846888166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 0x7f0a2c108e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.370+0000 7f0a31c00700 1 --2- 172.21.15.28:0/846888166 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a2c102a90 0x7f0a2c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.370+0000 7f0a31c00700 1 -- 172.21.15.28:0/846888166 >> 172.21.15.28:0/846888166 conn(0x7f0a2c079ba0 msgr2=0x7f0a2c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.370+0000 7f0a31c00700 1 -- 172.21.15.28:0/846888166 shutdown_connections 2024-04-26T01:38:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.370+0000 7f0a31c00700 1 -- 172.21.15.28:0/846888166 wait complete. 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.371+0000 7f0a31c00700 1 Processor -- start 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.371+0000 7f0a31c00700 1 -- start start 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.371+0000 7f0a31c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102a90 0x7f0a2c19b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.371+0000 7f0a31c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a2c103450 0x7f0a2c19b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.371+0000 7f0a31c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 0x7f0a2c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.371+0000 7f0a31c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c112be0 con 0x7f0a2c102a90 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.371+0000 7f0a31c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c112a60 con 0x7f0a2c108a50 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.371+0000 7f0a31c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c112d60 con 0x7f0a2c103450 2024-04-26T01:38:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a2c103450 0x7f0a2c19b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a2c103450 0x7f0a2c19b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:56862/0 (socket says 172.21.15.28:56862) 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 -- 172.21.15.28:0/2725599581 learned_addr learned my addr 172.21.15.28:0/2725599581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a2aa00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102a90 0x7f0a2c19b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a2b400700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 0x7f0a2c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 -- 172.21.15.28:0/2725599581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 msgr2=0x7f0a2c19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 0x7f0a2c19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 -- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102a90 msgr2=0x7f0a2c19b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102a90 0x7f0a2c19b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 -- 172.21.15.28:0/2725599581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2c1a03c0 con 0x7f0a2c103450 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a2b400700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 0x7f0a2c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:38:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.372+0000 7f0a22000700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a2c103450 0x7f0a2c19b8f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400e9c0 tx=0x7f0a1400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.373+0000 7f0a23e00700 1 -- 172.21.15.28:0/2725599581 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400cbd0 con 0x7f0a2c103450 2024-04-26T01:38:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.373+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2c1a06b0 con 0x7f0a2c103450 2024-04-26T01:38:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.373+0000 7f0a23e00700 1 -- 172.21.15.28:0/2725599581 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a1400cd70 con 0x7f0a2c103450 2024-04-26T01:38:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.373+0000 7f0a23e00700 1 -- 172.21.15.28:0/2725599581 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a14005580 con 0x7f0a2c103450 2024-04-26T01:38:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.373+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a2c1a7f30 con 0x7f0a2c103450 2024-04-26T01:38:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.374+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2c069730 con 0x7f0a2c103450 2024-04-26T01:38:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.376+0000 7f0a23e00700 1 -- 172.21.15.28:0/2725599581 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0a14004750 con 0x7f0a2c103450 2024-04-26T01:38:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.377+0000 7f0a23e00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f0a0c0a3cd0 0x7f0a0c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.378+0000 7f0a23e00700 1 -- 172.21.15.28:0/2725599581 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f0a140d07e0 con 0x7f0a2c103450 2024-04-26T01:38:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.378+0000 7f0a2aa00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f0a0c0a3cd0 0x7f0a0c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.378+0000 7f0a2aa00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f0a0c0a3cd0 0x7f0a0c0a6190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009fd0 tx=0x7f0a1c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.382+0000 7f0a23e00700 1 -- 172.21.15.28:0/2725599581 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a1409d010 con 0x7f0a2c103450 2024-04-26T01:38:09.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.617+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a2c02df80 con 0x7f0a2c103450 2024-04-26T01:38:09.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.619+0000 7f0a23e00700 1 -- 172.21.15.28:0/2725599581 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22630 (secure 0 0 0) 0x7f0a1409c910 con 0x7f0a2c103450 2024-04-26T01:38:09.618 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:38:09.618 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:37:25.472266+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":7,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:38:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f0a0c0a3cd0 msgr2=0x7f0a0c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f0a0c0a3cd0 0x7f0a0c0a6190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009fd0 tx=0x7f0a1c019040 comp rx=0 tx=0).stop 2024-04-26T01:38:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a2c103450 msgr2=0x7f0a2c19b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a2c103450 0x7f0a2c19b8f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400e9c0 tx=0x7f0a1400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 shutdown_connections 2024-04-26T01:38:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f0a0c0a3cd0 0x7f0a0c0a6190 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a2c108a50 0x7f0a2c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102a90 0x7f0a2c19b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 --2- 172.21.15.28:0/2725599581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a2c103450 0x7f0a2c19b8f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 >> 172.21.15.28:0/2725599581 conn(0x7f0a2c079ba0 msgr2=0x7f0a2c10df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.622+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 shutdown_connections 2024-04-26T01:38:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:09.623+0000 7f0a31c00700 1 -- 172.21.15.28:0/2725599581 wait complete. 2024-04-26T01:38:09.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:09 smithi053 ceph-mon[32593]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T01:38:09.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:09 smithi028 ceph-mon[28247]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T01:38:09.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:09 smithi169 ceph-mon[32851]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T01:38:10.565 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T01:38:10.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:38:10.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:10 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2725599581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:38:10.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:10 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2725599581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:38:10.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:10 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2725599581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:38:11.028 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:11.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:11 smithi053 ceph-mon[32593]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:11.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:11 smithi028 ceph-mon[28247]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:11.811 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:11 smithi169 ceph-mon[32851]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 -- 172.21.15.28:0/3143169454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88108970 msgr2=0x7f8d88108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 --2- 172.21.15.28:0/3143169454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88108970 0x7f8d88108d50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8d780098f0 tx=0x7f8d78009c00 comp rx=0 tx=0).stop 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 -- 172.21.15.28:0/3143169454 shutdown_connections 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 --2- 172.21.15.28:0/3143169454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d88103330 0x7f8d8810f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 --2- 172.21.15.28:0/3143169454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88108970 0x7f8d88108d50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 --2- 172.21.15.28:0/3143169454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d88102970 0x7f8d88102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 -- 172.21.15.28:0/3143169454 >> 172.21.15.28:0/3143169454 conn(0x7f8d880fe4c0 msgr2=0x7f8d881008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 -- 172.21.15.28:0/3143169454 shutdown_connections 2024-04-26T01:38:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.475+0000 7f8d87e00700 1 -- 172.21.15.28:0/3143169454 wait complete. 2024-04-26T01:38:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.477+0000 7f8d87e00700 1 Processor -- start 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.477+0000 7f8d87e00700 1 -- start start 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88102970 0x7f8d8819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d88103330 0x7f8d8819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d88108970 0x7f8d8819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d88112bb0 con 0x7f8d88102970 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8d88112a30 con 0x7f8d88103330 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8d88112d30 con 0x7f8d88108970 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d86a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88102970 0x7f8d8819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d88108970 0x7f8d8819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d88108970 0x7f8d8819fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:56880/0 (socket says 172.21.15.28:56880) 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d86a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88102970 0x7f8d8819b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51518/0 (socket says 172.21.15.28:51518) 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d87400700 1 -- 172.21.15.28:0/1485195673 learned_addr learned my addr 172.21.15.28:0/1485195673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.478+0000 7f8d86000700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d88103330 0x7f8d8819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d86000700 1 -- 172.21.15.28:0/1485195673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d88108970 msgr2=0x7f8d8819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d86000700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d88108970 0x7f8d8819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d86000700 1 -- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88102970 msgr2=0x7f8d8819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d86000700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88102970 0x7f8d8819b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d86000700 1 -- 172.21.15.28:0/1485195673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d881a0300 con 0x7f8d88103330 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d87400700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d88108970 0x7f8d8819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d86000700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d88103330 0x7f8d8819b880 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400eb90 tx=0x7f8d7400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d7f400700 1 -- 172.21.15.28:0/1485195673 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7400cc90 con 0x7f8d88103330 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d881a05f0 con 0x7f8d88103330 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d7f400700 1 -- 172.21.15.28:0/1485195673 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d7400ce30 con 0x7f8d88103330 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d7f400700 1 -- 172.21.15.28:0/1485195673 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d74010430 con 0x7f8d88103330 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.479+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d881a7e70 con 0x7f8d88103330 2024-04-26T01:38:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.480+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d88069730 con 0x7f8d88103330 2024-04-26T01:38:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.482+0000 7f8d7f400700 1 -- 172.21.15.28:0/1485195673 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8d74003ef0 con 0x7f8d88103330 2024-04-26T01:38:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.484+0000 7f8d7f400700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8d6c0a3d50 0x7f8d6c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.484+0000 7f8d7f400700 1 -- 172.21.15.28:0/1485195673 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f8d740d0840 con 0x7f8d88103330 2024-04-26T01:38:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.484+0000 7f8d86a00700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8d6c0a3d50 0x7f8d6c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.485+0000 7f8d86a00700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8d6c0a3d50 0x7f8d6c0a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8d780098c0 tx=0x7f8d78019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:13.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.489+0000 7f8d7f400700 1 -- 172.21.15.28:0/1485195673 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d7409d070 con 0x7f8d88103330 2024-04-26T01:38:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:13 smithi053 ceph-mon[32593]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.721+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d881a80a0 con 0x7f8d88103330 2024-04-26T01:38:13.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:13 smithi028 ceph-mon[28247]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:13.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.723+0000 7f8d7f400700 1 -- 172.21.15.28:0/1485195673 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22630 (secure 0 0 0) 0x7f8d7409c970 con 0x7f8d88103330 2024-04-26T01:38:13.724 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:38:13.724 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:37:25.472266+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":7,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:38:13.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.726+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8d6c0a3d50 msgr2=0x7f8d6c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:13.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.726+0000 7f8d87e00700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8d6c0a3d50 0x7f8d6c0a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8d780098c0 tx=0x7f8d78019040 comp rx=0 tx=0).stop 2024-04-26T01:38:13.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d88103330 msgr2=0x7f8d8819b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:13.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d88103330 0x7f8d8819b880 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400eb90 tx=0x7f8d7400eea0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 shutdown_connections 2024-04-26T01:38:13.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f8d6c0a3d50 0x7f8d6c0a6210 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d88108970 0x7f8d8819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88102970 0x7f8d8819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 --2- 172.21.15.28:0/1485195673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d88103330 0x7f8d8819b880 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:13.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 >> 172.21.15.28:0/1485195673 conn(0x7f8d880fe4c0 msgr2=0x7f8d880fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:13.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 shutdown_connections 2024-04-26T01:38:13.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:13.727+0000 7f8d87e00700 1 -- 172.21.15.28:0/1485195673 wait complete. 2024-04-26T01:38:13.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:13 smithi169 ceph-mon[32851]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:14.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T01:38:14.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T01:38:14.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T01:38:14.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T01:38:14.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T01:38:14.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T01:38:14.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T01:38:14.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T01:38:14.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T01:38:14.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T01:38:14.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T01:38:14.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T01:38:14.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:14 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1485195673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:38:14.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:14 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1485195673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:38:14.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:14 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1485195673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:38:15.732 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:15.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:15 smithi028 ceph-mon[28247]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:15.811 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:15.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:15 smithi169 ceph-mon[32851]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:15.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:15 smithi053 ceph-mon[32593]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:16.866 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:16.866 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:17.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:17 smithi028 ceph-mon[28247]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:17.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:17 smithi169 ceph-mon[32851]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:17.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:17 smithi053 ceph-mon[32593]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:19.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:19 smithi028 ceph-mon[28247]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:19.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:19 smithi169 ceph-mon[32851]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:19.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:19 smithi053 ceph-mon[32593]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:21.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:21 smithi028 ceph-mon[28247]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:21.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:21 smithi169 ceph-mon[32851]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:21 smithi053 ceph-mon[32593]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 -- 172.21.15.28:0/2476784511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 msgr2=0x7f2ed006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 --2- 172.21.15.28:0/2476784511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 0x7f2ed006abb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098f0 tx=0x7f2ebc009c00 comp rx=0 tx=0).stop 2024-04-26T01:38:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 -- 172.21.15.28:0/2476784511 shutdown_connections 2024-04-26T01:38:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 --2- 172.21.15.28:0/2476784511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed006b0f0 0x7f2ed010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 --2- 172.21.15.28:0/2476784511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ed0108a30 0x7f2ed0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 --2- 172.21.15.28:0/2476784511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 0x7f2ed006abb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 -- 172.21.15.28:0/2476784511 >> 172.21.15.28:0/2476784511 conn(0x7f2ed00788f0 msgr2=0x7f2ed0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 -- 172.21.15.28:0/2476784511 shutdown_connections 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.139+0000 7f2ed5600700 1 -- 172.21.15.28:0/2476784511 wait complete. 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.141+0000 7f2ed5600700 1 Processor -- start 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.141+0000 7f2ed5600700 1 -- start start 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.141+0000 7f2ed5600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 0x7f2ed019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.141+0000 7f2ed5600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed006b0f0 0x7f2ed019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.141+0000 7f2ed5600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ed0108a30 0x7f2ed019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.141+0000 7f2ed5600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed0112c20 con 0x7f2ed0108a30 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.141+0000 7f2ed5600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed0112aa0 con 0x7f2ed006b0f0 2024-04-26T01:38:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.141+0000 7f2ed5600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed0112da0 con 0x7f2ed006a750 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ece000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed006b0f0 0x7f2ed019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 0x7f2ed019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 0x7f2ed019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:49154/0 (socket says 172.21.15.28:49154) 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ece000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed006b0f0 0x7f2ed019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:34772/0 (socket says 172.21.15.28:34772) 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 -- 172.21.15.28:0/375824365 learned_addr learned my addr 172.21.15.28:0/375824365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecf400700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ed0108a30 0x7f2ed019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 -- 172.21.15.28:0/375824365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed006b0f0 msgr2=0x7f2ed019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed006b0f0 0x7f2ed019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 -- 172.21.15.28:0/375824365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ed0108a30 msgr2=0x7f2ed019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ed0108a30 0x7f2ed019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 -- 172.21.15.28:0/375824365 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed01a03a0 con 0x7f2ed006a750 2024-04-26T01:38:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.142+0000 7f2ecea00700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 0x7f2ed019b390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000e9c0 tx=0x7f2ec000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.143+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec000cbd0 con 0x7f2ed006a750 2024-04-26T01:38:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.143+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec000cd70 con 0x7f2ed006a750 2024-04-26T01:38:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.143+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0005580 con 0x7f2ed006a750 2024-04-26T01:38:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.143+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed01a0690 con 0x7f2ed006a750 2024-04-26T01:38:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.143+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed01a7f10 con 0x7f2ed006a750 2024-04-26T01:38:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.144+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ed0063150 con 0x7f2ed006a750 2024-04-26T01:38:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.146+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2ec0004750 con 0x7f2ed006a750 2024-04-26T01:38:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.148+0000 7f2ec7400700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2eb40a3d50 0x7f2eb40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.148+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f2ec00d0ae0 con 0x7f2ed006a750 2024-04-26T01:38:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.148+0000 7f2ece000700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2eb40a3d50 0x7f2eb40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.148+0000 7f2ec7400700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] conn(0x7f2eb40ad890 0x7f2eb40afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.148+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 --> [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2eb40b0400 con 0x7f2eb40ad890 2024-04-26T01:38:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.148+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f2ec00d0e90 con 0x7f2ed006a750 2024-04-26T01:38:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.149+0000 7f2ecf400700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] conn(0x7f2eb40ad890 0x7f2eb40afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.149+0000 7f2ece000700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2eb40a3d50 0x7f2eb40a6210 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098c0 tx=0x7f2ebc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.149+0000 7f2ecf400700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] conn(0x7f2eb40ad890 0x7f2eb40afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.150+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 <== osd.4 v2:172.21.15.53:6808/3129809581 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2eb40b0400 con 0x7f2eb40ad890 2024-04-26T01:38:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.168+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 --> [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2ed0063620 con 0x7f2eb40ad890 2024-04-26T01:38:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.169+0000 7f2ec7400700 1 -- 172.21.15.28:0/375824365 <== osd.4 v2:172.21.15.53:6808/3129809581 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2ed0063620 con 0x7f2eb40ad890 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.169+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 >> [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] conn(0x7f2eb40ad890 msgr2=0x7f2eb40afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.169+0000 7f2ed5600700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] conn(0x7f2eb40ad890 0x7f2eb40afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.169+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2eb40a3d50 msgr2=0x7f2eb40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.169+0000 7f2ed5600700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2eb40a3d50 0x7f2eb40a6210 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098c0 tx=0x7f2ebc019040 comp rx=0 tx=0).stop 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.169+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 msgr2=0x7f2ed019b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.169+0000 7f2ed5600700 1 --2- 172.21.15.28:0/375824365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ed006a750 0x7f2ed019b390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000e9c0 tx=0x7f2ec000ecd0 comp rx=0 tx=0).stop 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.169+0000 7f2ecf400700 1 -- 172.21.15.28:0/375824365 reap_dead start 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.170+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 shutdown_connections 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.170+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 >> 172.21.15.28:0/375824365 conn(0x7f2ed00788f0 msgr2=0x7f2ed010dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.170+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 shutdown_connections 2024-04-26T01:38:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:23.170+0000 7f2ed5600700 1 -- 172.21.15.28:0/375824365 wait complete. 2024-04-26T01:38:23.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:23 smithi028 ceph-mon[28247]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:23.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:23 smithi169 ceph-mon[32851]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:23.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:23 smithi053 ceph-mon[32593]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:25.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:25 smithi028 ceph-mon[28247]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:25.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 -- 172.21.15.28:0/3538183134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406a750 msgr2=0x7f221406ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:25.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 --2- 172.21.15.28:0/3538183134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406a750 0x7f221406ab30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2208009990 tx=0x7f2208009ca0 comp rx=0 tx=0).stop 2024-04-26T01:38:25.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 -- 172.21.15.28:0/3538183134 shutdown_connections 2024-04-26T01:38:25.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 --2- 172.21.15.28:0/3538183134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f221410f4f0 0x7f22141118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 --2- 172.21.15.28:0/3538183134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406a750 0x7f221406ab30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 --2- 172.21.15.28:0/3538183134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f221406b070 0x7f221410efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 -- 172.21.15.28:0/3538183134 >> 172.21.15.28:0/3538183134 conn(0x7f22140788f0 msgr2=0x7f2214078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 -- 172.21.15.28:0/3538183134 shutdown_connections 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.749+0000 7f221c600700 1 -- 172.21.15.28:0/3538183134 wait complete. 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.751+0000 7f221c600700 1 Processor -- start 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.751+0000 7f221c600700 1 -- start start 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.751+0000 7f221c600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f221406a750 0x7f221419b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.751+0000 7f221c600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406b070 0x7f221419b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.751+0000 7f221c600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f221410f4f0 0x7f221419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.751+0000 7f221c600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2214114e70 con 0x7f221406b070 2024-04-26T01:38:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.751+0000 7f221c600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2214114cf0 con 0x7f221406a750 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.751+0000 7f221c600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2214114ff0 con 0x7f221410f4f0 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406b070 0x7f221419b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406b070 0x7f221419b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46534/0 (socket says 172.21.15.28:46534) 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219200700 1 -- 172.21.15.28:0/3520678927 learned_addr learned my addr 172.21.15.28:0/3520678927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f221a600700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f221410f4f0 0x7f221419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219c00700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f221406a750 0x7f221419b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219c00700 1 -- 172.21.15.28:0/3520678927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f221410f4f0 msgr2=0x7f221419fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219c00700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f221410f4f0 0x7f221419fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219c00700 1 -- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406b070 msgr2=0x7f221419b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219c00700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406b070 0x7f221419b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219c00700 1 -- 172.21.15.28:0/3520678927 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22141a0390 con 0x7f221406a750 2024-04-26T01:38:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f2219200700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f221406b070 0x7f221419b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.752+0000 7f221a600700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f221410f4f0 0x7f221419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.753+0000 7f2219c00700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f221406a750 0x7f221419b380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f220800b2e0 tx=0x7f22080047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.753+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220800fa50 con 0x7f221406a750 2024-04-26T01:38:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.753+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22141a0620 con 0x7f221406a750 2024-04-26T01:38:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.753+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2208004ce0 con 0x7f221406a750 2024-04-26T01:38:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.753+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22080054c0 con 0x7f221406a750 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.753+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22141a7f00 con 0x7f221406a750 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.754+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2214063150 con 0x7f221406a750 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.756+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2208028060 con 0x7f221406a750 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.757+0000 7f2212000700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f21fc0a3cd0 0x7f21fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.758+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f22080d1420 con 0x7f221406a750 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.758+0000 7f2219200700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f21fc0a3cd0 0x7f21fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.758+0000 7f2212000700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] conn(0x7f21fc0ad810 0x7f21fc0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.758+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 --> [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2208024ec0 con 0x7f21fc0ad810 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.758+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f22080d17d0 con 0x7f221406a750 2024-04-26T01:38:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.758+0000 7f221a600700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] conn(0x7f21fc0ad810 0x7f21fc0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.759+0000 7f2219200700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f21fc0a3cd0 0x7f21fc0a6190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2214102ea0 tx=0x7f2200007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.759+0000 7f221a600700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] conn(0x7f21fc0ad810 0x7f21fc0afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.760+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 <== osd.6 v2:172.21.15.28:6818/4283307315 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2208024ec0 con 0x7f21fc0ad810 2024-04-26T01:38:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.779+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 --> [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2214063580 con 0x7f21fc0ad810 2024-04-26T01:38:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.779+0000 7f2212000700 1 -- 172.21.15.28:0/3520678927 <== osd.6 v2:172.21.15.28:6818/4283307315 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2214063580 con 0x7f21fc0ad810 2024-04-26T01:38:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.780+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] conn(0x7f21fc0ad810 msgr2=0x7f21fc0afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.780+0000 7f221c600700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] conn(0x7f21fc0ad810 0x7f21fc0afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.780+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f21fc0a3cd0 msgr2=0x7f21fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.780+0000 7f221c600700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f21fc0a3cd0 0x7f21fc0a6190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2214102ea0 tx=0x7f2200007480 comp rx=0 tx=0).stop 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.780+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f221406a750 msgr2=0x7f221419b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.780+0000 7f221c600700 1 --2- 172.21.15.28:0/3520678927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f221406a750 0x7f221419b380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f220800b2e0 tx=0x7f22080047b0 comp rx=0 tx=0).stop 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.781+0000 7f221a600700 1 -- 172.21.15.28:0/3520678927 reap_dead start 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.781+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 shutdown_connections 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.781+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 >> 172.21.15.28:0/3520678927 conn(0x7f22140788f0 msgr2=0x7f22141109a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.781+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 shutdown_connections 2024-04-26T01:38:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:25.781+0000 7f221c600700 1 -- 172.21.15.28:0/3520678927 wait complete. 2024-04-26T01:38:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:25 smithi169 ceph-mon[32851]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:25 smithi053 ceph-mon[32593]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout:141733920844 2024-04-26T01:38:27.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T01:38:27.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:27 smithi028 ceph-mon[28247]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T01:38:27.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:27 smithi053 ceph-mon[32593]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T01:38:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:27 smithi169 ceph-mon[32851]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T01:38:29.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:29 smithi028 ceph-mon[28247]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:38:29.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:29 smithi169 ceph-mon[32851]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:38:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:29 smithi053 ceph-mon[32593]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:38:31.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:31 smithi028 ceph-mon[28247]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.449+0000 7fc93d200700 1 -- 172.21.15.28:0/4128063132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc938113920 msgr2=0x7fc938115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.449+0000 7fc93d200700 1 --2- 172.21.15.28:0/4128063132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc938113920 0x7fc938115d10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc93000b0a0 tx=0x7fc93000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:38:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.449+0000 7fc93d200700 1 -- 172.21.15.28:0/4128063132 shutdown_connections 2024-04-26T01:38:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.449+0000 7fc93d200700 1 --2- 172.21.15.28:0/4128063132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc938113920 0x7fc938115d10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.449+0000 7fc93d200700 1 --2- 172.21.15.28:0/4128063132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc938076f70 0x7fc938077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.449+0000 7fc93d200700 1 --2- 172.21.15.28:0/4128063132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc938077890 0x7fc9381133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.449+0000 7fc93d200700 1 -- 172.21.15.28:0/4128063132 >> 172.21.15.28:0/4128063132 conn(0x7fc938100770 msgr2=0x7fc938102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.450+0000 7fc93d200700 1 -- 172.21.15.28:0/4128063132 shutdown_connections 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.450+0000 7fc93d200700 1 -- 172.21.15.28:0/4128063132 wait complete. 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.451+0000 7fc93d200700 1 Processor -- start 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.451+0000 7fc93d200700 1 -- start start 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.451+0000 7fc93d200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc938076f70 0x7fc93819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.451+0000 7fc93d200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc938077890 0x7fc93819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.451+0000 7fc93d200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc938113920 0x7fc9381a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.451+0000 7fc93d200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9381192a0 con 0x7fc938077890 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.451+0000 7fc93d200700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc938119120 con 0x7fc938113920 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.451+0000 7fc93d200700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc938119420 con 0x7fc938076f70 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc935e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc938076f70 0x7fc93819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc938113920 0x7fc9381a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc935400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc938077890 0x7fc93819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc938113920 0x7fc9381a4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:40078/0 (socket says 172.21.15.28:40078) 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 -- 172.21.15.28:0/682520713 learned_addr learned my addr 172.21.15.28:0/682520713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc935e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc938076f70 0x7fc93819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:35262/0 (socket says 172.21.15.28:35262) 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 -- 172.21.15.28:0/682520713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc938076f70 msgr2=0x7fc93819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc938076f70 0x7fc93819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 -- 172.21.15.28:0/682520713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc938077890 msgr2=0x7fc93819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc938077890 0x7fc93819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 -- 172.21.15.28:0/682520713 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9381a4750 con 0x7fc938113920 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.452+0000 7fc936800700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc938113920 0x7fc9381a4010 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc93000b070 tx=0x7fc930004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.453+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930021070 con 0x7fc938113920 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.453+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9381a49e0 con 0x7fc938113920 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.453+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc930007c20 con 0x7fc938113920 2024-04-26T01:38:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.453+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9300045e0 con 0x7fc938113920 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.453+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9381ac2c0 con 0x7fc938113920 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.454+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9380631d0 con 0x7fc938113920 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.456+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc9300074e0 con 0x7fc938113920 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.457+0000 7fc92ea00700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc91c0a3cd0 0x7fc91c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.457+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fc9300d17e0 con 0x7fc938113920 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.458+0000 7fc935e00700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc91c0a3cd0 0x7fc91c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.458+0000 7fc92ea00700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] conn(0x7fc91c0ad810 0x7fc91c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.458+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 --> [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc93000ce60 con 0x7fc91c0ad810 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.458+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fc9300d1b90 con 0x7fc938113920 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.458+0000 7fc935400700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] conn(0x7fc91c0ad810 0x7fc91c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.458+0000 7fc935e00700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc91c0a3cd0 0x7fc91c0a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc928005950 tx=0x7fc92800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.459+0000 7fc935400700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] conn(0x7fc91c0ad810 0x7fc91c0afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.460+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 <== osd.8 v2:172.21.15.169:6816/3120946551 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc93000ce60 con 0x7fc91c0ad810 2024-04-26T01:38:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.478+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 --> [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc938063650 con 0x7fc91c0ad810 2024-04-26T01:38:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.479+0000 7fc92ea00700 1 -- 172.21.15.28:0/682520713 <== osd.8 v2:172.21.15.169:6816/3120946551 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc938063650 con 0x7fc91c0ad810 2024-04-26T01:38:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.479+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 >> [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] conn(0x7fc91c0ad810 msgr2=0x7fc91c0afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] conn(0x7fc91c0ad810 0x7fc91c0afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc91c0a3cd0 msgr2=0x7fc91c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc91c0a3cd0 0x7fc91c0a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc928005950 tx=0x7fc92800b410 comp rx=0 tx=0).stop 2024-04-26T01:38:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc938113920 msgr2=0x7fc9381a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 --2- 172.21.15.28:0/682520713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc938113920 0x7fc9381a4010 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc93000b070 tx=0x7fc930004c80 comp rx=0 tx=0).stop 2024-04-26T01:38:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc936800700 1 -- 172.21.15.28:0/682520713 reap_dead start 2024-04-26T01:38:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 shutdown_connections 2024-04-26T01:38:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 >> 172.21.15.28:0/682520713 conn(0x7fc938100770 msgr2=0x7fc938102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 shutdown_connections 2024-04-26T01:38:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:31.480+0000 7fc93d200700 1 -- 172.21.15.28:0/682520713 wait complete. 2024-04-26T01:38:31.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:31 smithi169 ceph-mon[32851]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:31.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:31 smithi053 ceph-mon[32593]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 -- 172.21.15.28:0/2128325583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c076f70 msgr2=0x7fe29c077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 --2- 172.21.15.28:0/2128325583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c076f70 0x7fe29c077350 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2900098f0 tx=0x7fe290009c00 comp rx=0 tx=0).stop 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 -- 172.21.15.28:0/2128325583 shutdown_connections 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 --2- 172.21.15.28:0/2128325583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe29c1138d0 0x7fe29c115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 --2- 172.21.15.28:0/2128325583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c076f70 0x7fe29c077350 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 --2- 172.21.15.28:0/2128325583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe29c077890 0x7fe29c113390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 -- 172.21.15.28:0/2128325583 >> 172.21.15.28:0/2128325583 conn(0x7fe29c100720 msgr2=0x7fe29c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 -- 172.21.15.28:0/2128325583 shutdown_connections 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.372+0000 7fe2a3a00700 1 -- 172.21.15.28:0/2128325583 wait complete. 2024-04-26T01:38:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.373+0000 7fe2a3a00700 1 Processor -- start 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.373+0000 7fe2a3a00700 1 -- start start 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a3a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe29c076f70 0x7fe29c1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a3a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c077890 0x7fe29c1a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a3a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe29c1138d0 0x7fe29c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a3a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c119240 con 0x7fe29c076f70 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a3a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c1190c0 con 0x7fe29c1138d0 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a3a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c1193c0 con 0x7fe29c077890 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe29be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c077890 0x7fe29c1a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a1a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe29c1138d0 0x7fe29c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a1a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe29c1138d0 0x7fe29c19f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:40104/0 (socket says 172.21.15.28:40104) 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe2a1a00700 1 -- 172.21.15.28:0/1265747655 learned_addr learned my addr 172.21.15.28:0/1265747655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.374+0000 7fe29be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c077890 0x7fe29c1a5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:35286/0 (socket says 172.21.15.28:35286) 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.375+0000 7fe2a1000700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe29c076f70 0x7fe29c1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.375+0000 7fe29be00700 1 -- 172.21.15.28:0/1265747655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe29c1138d0 msgr2=0x7fe29c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.375+0000 7fe29be00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe29c1138d0 0x7fe29c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.375+0000 7fe29be00700 1 -- 172.21.15.28:0/1265747655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe29c076f70 msgr2=0x7fe29c1a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.375+0000 7fe29be00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe29c076f70 0x7fe29c1a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.375+0000 7fe29be00700 1 -- 172.21.15.28:0/1265747655 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c19fd40 con 0x7fe29c077890 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.375+0000 7fe2a1a00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe29c1138d0 0x7fe29c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.375+0000 7fe29be00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c077890 0x7fe29c1a5960 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe28800eb90 tx=0x7fe28800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.376+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28800cc90 con 0x7fe29c077890 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.376+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe28800ce30 con 0x7fe29c077890 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.376+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288010430 con 0x7fe29c077890 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.376+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c1a0030 con 0x7fe29c077890 2024-04-26T01:38:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.376+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c1ac2f0 con 0x7fe29c077890 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.377+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe29c063190 con 0x7fe29c077890 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.379+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe288003ea0 con 0x7fe29c077890 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.380+0000 7fe299600700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe2840a3d50 0x7fe2840a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.381+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fe2880d0b80 con 0x7fe29c077890 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.381+0000 7fe2a1000700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe2840a3d50 0x7fe2840a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.381+0000 7fe299600700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] conn(0x7fe2840ad7c0 0x7fe2840afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.381+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 --> [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe288023e50 con 0x7fe2840ad7c0 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.381+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe2880d0f30 con 0x7fe29c077890 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.381+0000 7fe2a1a00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] conn(0x7fe2840ad7c0 0x7fe2840afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.381+0000 7fe2a1000700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe2840a3d50 0x7fe2840a6210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe290009fd0 tx=0x7fe290019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.382+0000 7fe2a1a00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] conn(0x7fe2840ad7c0 0x7fe2840afc60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.383+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 <== osd.1 v2:172.21.15.53:6800/1246761036 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe288023e50 con 0x7fe2840ad7c0 2024-04-26T01:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.401+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 --> [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe29c063660 con 0x7fe2840ad7c0 2024-04-26T01:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.402+0000 7fe299600700 1 -- 172.21.15.28:0/1265747655 <== osd.1 v2:172.21.15.53:6800/1246761036 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe29c063660 con 0x7fe2840ad7c0 2024-04-26T01:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.402+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 >> [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] conn(0x7fe2840ad7c0 msgr2=0x7fe2840afc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.402+0000 7fe2a3a00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] conn(0x7fe2840ad7c0 0x7fe2840afc60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe2840a3d50 msgr2=0x7fe2840a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a3a00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe2840a3d50 0x7fe2840a6210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe290009fd0 tx=0x7fe290019040 comp rx=0 tx=0).stop 2024-04-26T01:38:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c077890 msgr2=0x7fe29c1a5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a3a00700 1 --2- 172.21.15.28:0/1265747655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe29c077890 0x7fe29c1a5960 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe28800eb90 tx=0x7fe28800eea0 comp rx=0 tx=0).stop 2024-04-26T01:38:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a1a00700 1 -- 172.21.15.28:0/1265747655 reap_dead start 2024-04-26T01:38:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 shutdown_connections 2024-04-26T01:38:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 >> 172.21.15.28:0/1265747655 conn(0x7fe29c100720 msgr2=0x7fe29c102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 shutdown_connections 2024-04-26T01:38:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:32.403+0000 7fe2a3a00700 1 -- 172.21.15.28:0/1265747655 wait complete. 2024-04-26T01:38:33.665 INFO:teuthology.orchestra.run.smithi028.stdout:197568495681 2024-04-26T01:38:33.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T01:38:33.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:33 smithi028 ceph-mon[28247]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:33.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:33 smithi169 ceph-mon[32851]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:33.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:33 smithi053 ceph-mon[32593]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:35.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:35 smithi028 ceph-mon[28247]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:35.576 INFO:teuthology.orchestra.run.smithi028.stdout:253403070518 2024-04-26T01:38:35.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T01:38:35.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:35 smithi169 ceph-mon[32851]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:35.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:35 smithi053 ceph-mon[32593]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.203+0000 7f4c5ce00700 1 -- 172.21.15.28:0/2387547238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806a750 msgr2=0x7f4c5806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.203+0000 7f4c5ce00700 1 --2- 172.21.15.28:0/2387547238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806a750 0x7f4c5806abb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009990 tx=0x7f4c44009ca0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.203+0000 7f4c5ce00700 1 -- 172.21.15.28:0/2387547238 shutdown_connections 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.203+0000 7f4c5ce00700 1 --2- 172.21.15.28:0/2387547238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c5806b0f0 0x7f4c5810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.203+0000 7f4c5ce00700 1 --2- 172.21.15.28:0/2387547238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c58108a30 0x7f4c58108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.203+0000 7f4c5ce00700 1 --2- 172.21.15.28:0/2387547238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806a750 0x7f4c5806abb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.203+0000 7f4c5ce00700 1 -- 172.21.15.28:0/2387547238 >> 172.21.15.28:0/2387547238 conn(0x7f4c580788f0 msgr2=0x7f4c58078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.204+0000 7f4c5ce00700 1 -- 172.21.15.28:0/2387547238 shutdown_connections 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.204+0000 7f4c5ce00700 1 -- 172.21.15.28:0/2387547238 wait complete. 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.205+0000 7f4c5ce00700 1 Processor -- start 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.205+0000 7f4c5ce00700 1 -- start start 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.205+0000 7f4c5ce00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c5806a750 0x7f4c5819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.205+0000 7f4c5ce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b0f0 0x7f4c5819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.205+0000 7f4c5ce00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c58108a30 0x7f4c5819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.205+0000 7f4c5ce00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58112c20 con 0x7f4c5806b0f0 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.205+0000 7f4c5ce00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58112aa0 con 0x7f4c58108a30 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.205+0000 7f4c5ce00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58112da0 con 0x7f4c5806a750 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c5806a750 0x7f4c5819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b0f0 0x7f4c5819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c5806a750 0x7f4c5819b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:35332/0 (socket says 172.21.15.28:35332) 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 -- 172.21.15.28:0/1483421249 learned_addr learned my addr 172.21.15.28:0/1483421249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c56800700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c58108a30 0x7f4c5819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 -- 172.21.15.28:0/1483421249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c58108a30 msgr2=0x7f4c5819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c58108a30 0x7f4c5819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 -- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b0f0 msgr2=0x7f4c5819b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b0f0 0x7f4c5819b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 -- 172.21.15.28:0/1483421249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c581a03a0 con 0x7f4c5806a750 2024-04-26T01:38:36.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.206+0000 7f4c55e00700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c5806a750 0x7f4c5819b390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800eb30 tx=0x7f4c4800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:36.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.207+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4800cbd0 con 0x7f4c5806a750 2024-04-26T01:38:36.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.207+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c480044d0 con 0x7f4c5806a750 2024-04-26T01:38:36.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.207+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48003e30 con 0x7f4c5806a750 2024-04-26T01:38:36.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.207+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c581a0690 con 0x7f4c5806a750 2024-04-26T01:38:36.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.207+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c581a7f10 con 0x7f4c5806a750 2024-04-26T01:38:36.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.208+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4c58063150 con 0x7f4c5806a750 2024-04-26T01:38:36.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.210+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4c48004000 con 0x7f4c5806a750 2024-04-26T01:38:36.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.212+0000 7f4c4ea00700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f4c3c0a3cd0 0x7f4c3c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.212+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f4c480d0cc0 con 0x7f4c5806a750 2024-04-26T01:38:36.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.212+0000 7f4c55400700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f4c3c0a3cd0 0x7f4c3c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.213+0000 7f4c4ea00700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] conn(0x7f4c3c0ad810 0x7f4c3c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.213+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 --> [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c48017e50 con 0x7f4c3c0ad810 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.213+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f4c480d1070 con 0x7f4c5806a750 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.213+0000 7f4c56800700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] conn(0x7f4c3c0ad810 0x7f4c3c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.213+0000 7f4c55400700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f4c3c0a3cd0 0x7f4c3c0a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4c58078f40 tx=0x7f4c4400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.214+0000 7f4c56800700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] conn(0x7f4c3c0ad810 0x7f4c3c0afcb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.215+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 <== osd.9 v2:172.21.15.28:6826/2760989310 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c48017e50 con 0x7f4c3c0ad810 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.233+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 --> [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4c580635d0 con 0x7f4c3c0ad810 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.234+0000 7f4c4ea00700 1 -- 172.21.15.28:0/1483421249 <== osd.9 v2:172.21.15.28:6826/2760989310 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4c580635d0 con 0x7f4c3c0ad810 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.234+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] conn(0x7f4c3c0ad810 msgr2=0x7f4c3c0afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.234+0000 7f4c5ce00700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] conn(0x7f4c3c0ad810 0x7f4c3c0afcb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.234+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f4c3c0a3cd0 msgr2=0x7f4c3c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.234+0000 7f4c5ce00700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f4c3c0a3cd0 0x7f4c3c0a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4c58078f40 tx=0x7f4c4400b260 comp rx=0 tx=0).stop 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.234+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c5806a750 msgr2=0x7f4c5819b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.234+0000 7f4c5ce00700 1 --2- 172.21.15.28:0/1483421249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c5806a750 0x7f4c5819b390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800eb30 tx=0x7f4c4800ee40 comp rx=0 tx=0).stop 2024-04-26T01:38:36.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.234+0000 7f4c56800700 1 -- 172.21.15.28:0/1483421249 reap_dead start 2024-04-26T01:38:36.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.235+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 shutdown_connections 2024-04-26T01:38:36.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.235+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 >> 172.21.15.28:0/1483421249 conn(0x7f4c580788f0 msgr2=0x7f4c5810dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:36.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.235+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 shutdown_connections 2024-04-26T01:38:36.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.235+0000 7f4c5ce00700 1 -- 172.21.15.28:0/1483421249 wait complete. 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 -- 172.21.15.28:0/245249364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c4102ac0 msgr2=0x7f01c4102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 --2- 172.21.15.28:0/245249364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c4102ac0 0x7f01c4102f20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009990 tx=0x7f01b0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 -- 172.21.15.28:0/245249364 shutdown_connections 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 --2- 172.21.15.28:0/245249364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01c4103460 0x7f01c410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 --2- 172.21.15.28:0/245249364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01c4108a70 0x7f01c4108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 --2- 172.21.15.28:0/245249364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c4102ac0 0x7f01c4102f20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 -- 172.21.15.28:0/245249364 >> 172.21.15.28:0/245249364 conn(0x7f01c4079b10 msgr2=0x7f01c4079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 -- 172.21.15.28:0/245249364 shutdown_connections 2024-04-26T01:38:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.343+0000 7f01cba00700 1 -- 172.21.15.28:0/245249364 wait complete. 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.344+0000 7f01cba00700 1 Processor -- start 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01cba00700 1 -- start start 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01cba00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c4102ac0 0x7f01c419b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01cba00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01c4103460 0x7f01c419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01cba00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01c4108a70 0x7f01c419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01cba00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01c4112c60 con 0x7f01c4108a70 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01cba00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f01c4112ae0 con 0x7f01c4103460 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01cba00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f01c4112de0 con 0x7f01c4102ac0 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01c9000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c4102ac0 0x7f01c419b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.345+0000 7f01c3e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01c4103460 0x7f01c419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c9a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01c4108a70 0x7f01c419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c3e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01c4103460 0x7f01c419b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:40184/0 (socket says 172.21.15.28:40184) 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c3e00700 1 -- 172.21.15.28:0/4182743367 learned_addr learned my addr 172.21.15.28:0/4182743367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c9000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c4102ac0 0x7f01c419b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:35358/0 (socket says 172.21.15.28:35358) 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c3e00700 1 -- 172.21.15.28:0/4182743367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c4102ac0 msgr2=0x7f01c419b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c3e00700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c4102ac0 0x7f01c419b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c3e00700 1 -- 172.21.15.28:0/4182743367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01c4108a70 msgr2=0x7f01c419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c3e00700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01c4108a70 0x7f01c419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c3e00700 1 -- 172.21.15.28:0/4182743367 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c41a03e0 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.346+0000 7f01c3e00700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01c4103460 0x7f01c419b910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f01b0005910 tx=0x7f01b000b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.347+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0027070 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.347+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c41a0670 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.347+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c41a7f50 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.347+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01b0022470 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.347+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0017650 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.348+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f01c4101350 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.350+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f01b002a070 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.351+0000 7f01c1600700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f01ac0a3d50 0x7f01ac0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.352+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f01b00d1680 con 0x7f01c4103460 2024-04-26T01:38:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.352+0000 7f01c9000700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f01ac0a3d50 0x7f01ac0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.352+0000 7f01c1600700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] conn(0x7f01ac0ad890 0x7f01ac0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.352+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 --> [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01b00095d0 con 0x7f01ac0ad890 2024-04-26T01:38:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.352+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f01b00d1a30 con 0x7f01c4103460 2024-04-26T01:38:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.352+0000 7f01c9000700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f01ac0a3d50 0x7f01ac0a6210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f01b800f540 tx=0x7f01b800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.352+0000 7f01c9a00700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] conn(0x7f01ac0ad890 0x7f01ac0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.353+0000 7f01c9a00700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] conn(0x7f01ac0ad890 0x7f01ac0afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.354+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 <== osd.7 v2:172.21.15.53:6816/1268967758 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b00095d0 con 0x7f01ac0ad890 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.372+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 --> [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f01c41041b0 con 0x7f01ac0ad890 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.373+0000 7f01c1600700 1 -- 172.21.15.28:0/4182743367 <== osd.7 v2:172.21.15.53:6816/1268967758 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f01c41041b0 con 0x7f01ac0ad890 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 >> [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] conn(0x7f01ac0ad890 msgr2=0x7f01ac0afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] conn(0x7f01ac0ad890 0x7f01ac0afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f01ac0a3d50 msgr2=0x7f01ac0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f01ac0a3d50 0x7f01ac0a6210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f01b800f540 tx=0x7f01b800b040 comp rx=0 tx=0).stop 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01c4103460 msgr2=0x7f01c419b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 --2- 172.21.15.28:0/4182743367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01c4103460 0x7f01c419b910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f01b0005910 tx=0x7f01b000b670 comp rx=0 tx=0).stop 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01c9a00700 1 -- 172.21.15.28:0/4182743367 reap_dead start 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 shutdown_connections 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 >> 172.21.15.28:0/4182743367 conn(0x7f01c4079b10 msgr2=0x7f01c4100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.374+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 shutdown_connections 2024-04-26T01:38:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:36.375+0000 7f01cba00700 1 -- 172.21.15.28:0/4182743367 wait complete. 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.882+0000 7f20dbc00700 1 -- 172.21.15.28:0/1911511209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d410f440 msgr2=0x7f20d4111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.882+0000 7f20dbc00700 1 --2- 172.21.15.28:0/1911511209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d410f440 0x7f20d4111830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00b0a0 tx=0x7f20cc00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.882+0000 7f20dbc00700 1 -- 172.21.15.28:0/1911511209 shutdown_connections 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.882+0000 7f20dbc00700 1 --2- 172.21.15.28:0/1911511209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d410f440 0x7f20d4111830 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.882+0000 7f20dbc00700 1 --2- 172.21.15.28:0/1911511209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20d4074c70 0x7f20d4075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.882+0000 7f20dbc00700 1 --2- 172.21.15.28:0/1911511209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d4075590 0x7f20d410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.882+0000 7f20dbc00700 1 -- 172.21.15.28:0/1911511209 >> 172.21.15.28:0/1911511209 conn(0x7f20d4079ba0 msgr2=0x7f20d4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.882+0000 7f20dbc00700 1 -- 172.21.15.28:0/1911511209 shutdown_connections 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.883+0000 7f20dbc00700 1 -- 172.21.15.28:0/1911511209 wait complete. 2024-04-26T01:38:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.884+0000 7f20dbc00700 1 Processor -- start 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.884+0000 7f20dbc00700 1 -- start start 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20dbc00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20d4074c70 0x7f20d419b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20dbc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d4075590 0x7f20d419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20dbc00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d410f440 0x7f20d419fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20dbc00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20d4114e50 con 0x7f20d4075590 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20dbc00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f20d4114cd0 con 0x7f20d4074c70 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20dbc00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f20d4114fd0 con 0x7f20d410f440 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d4075590 0x7f20d419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d4075590 0x7f20d419b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44560/0 (socket says 172.21.15.28:44560) 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d3e00700 1 -- 172.21.15.28:0/3054216894 learned_addr learned my addr 172.21.15.28:0/3054216894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d9c00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d410f440 0x7f20d419fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d9200700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20d4074c70 0x7f20d419b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:37.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d3e00700 1 -- 172.21.15.28:0/3054216894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d410f440 msgr2=0x7f20d419fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d3e00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d410f440 0x7f20d419fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d3e00700 1 -- 172.21.15.28:0/3054216894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20d4074c70 msgr2=0x7f20d419b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d3e00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20d4074c70 0x7f20d419b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.885+0000 7f20d3e00700 1 -- 172.21.15.28:0/3054216894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d41a0280 con 0x7f20d4075590 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.886+0000 7f20d3e00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d4075590 0x7f20d419b800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f20c000e950 tx=0x7f20c000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.886+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0004230 con 0x7f20d4075590 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.886+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20c00043d0 con 0x7f20d4075590 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.886+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0003d60 con 0x7f20d4075590 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.886+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d41a0570 con 0x7f20d4075590 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.886+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d41a7df0 con 0x7f20d4075590 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.887+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f20d4063150 con 0x7f20d4075590 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.890+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f20c000c7e0 con 0x7f20d4075590 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.891+0000 7f20d1600700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f20bc0a3d50 0x7f20bc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.891+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f20c00d09c0 con 0x7f20d4075590 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.891+0000 7f20d9200700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f20bc0a3d50 0x7f20bc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.892+0000 7f20d1600700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] conn(0x7f20bc0ad890 0x7f20bc0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.892+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 --> [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20c001aec0 con 0x7f20bc0ad890 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.892+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f20c00d0d70 con 0x7f20d4075590 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.892+0000 7f20d9c00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] conn(0x7f20bc0ad890 0x7f20bc0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.892+0000 7f20d9200700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f20bc0a3d50 0x7f20bc0a6210 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009c60 tx=0x7f20c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.893+0000 7f20d9c00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] conn(0x7f20bc0ad890 0x7f20bc0afd30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.894+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 <== osd.11 v2:172.21.15.169:6824/2005675057 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f20c001aec0 con 0x7f20bc0ad890 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.912+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 --> [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f20d40635d0 con 0x7f20bc0ad890 2024-04-26T01:38:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.913+0000 7f20d1600700 1 -- 172.21.15.28:0/3054216894 <== osd.11 v2:172.21.15.169:6824/2005675057 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f20d40635d0 con 0x7f20bc0ad890 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.913+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 >> [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] conn(0x7f20bc0ad890 msgr2=0x7f20bc0afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.913+0000 7f20dbc00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] conn(0x7f20bc0ad890 0x7f20bc0afd30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.913+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f20bc0a3d50 msgr2=0x7f20bc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.913+0000 7f20dbc00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f20bc0a3d50 0x7f20bc0a6210 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009c60 tx=0x7f20c8009260 comp rx=0 tx=0).stop 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.913+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d4075590 msgr2=0x7f20d419b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.913+0000 7f20dbc00700 1 --2- 172.21.15.28:0/3054216894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d4075590 0x7f20d419b800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f20c000e950 tx=0x7f20c000ec60 comp rx=0 tx=0).stop 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.913+0000 7f20d9c00700 1 -- 172.21.15.28:0/3054216894 reap_dead start 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.914+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 shutdown_connections 2024-04-26T01:38:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.914+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 >> 172.21.15.28:0/3054216894 conn(0x7f20d4079ba0 msgr2=0x7f20d40ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.914+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 shutdown_connections 2024-04-26T01:38:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.914+0000 7f20dbc00700 1 -- 172.21.15.28:0/3054216894 wait complete. 2024-04-26T01:38:37.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:37 smithi028 ceph-mon[28247]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:37.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:37 smithi169 ceph-mon[32851]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:37 smithi053 ceph-mon[32593]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:38.952 INFO:teuthology.orchestra.run.smithi028.stdout:343597383719 2024-04-26T01:38:38.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T01:38:39.560 INFO:teuthology.orchestra.run.smithi028.stdout:287762808882 2024-04-26T01:38:39.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T01:38:39.766 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:39 smithi028 ceph-mon[28247]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:38:39.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:39 smithi169 ceph-mon[32851]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:38:39.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:39 smithi053 ceph-mon[32593]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:38:40.061 INFO:teuthology.orchestra.run.smithi028.stdout:219043332158 2024-04-26T01:38:40.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.698+0000 7f2a93e00700 1 -- 172.21.15.28:0/116242155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a94102970 msgr2=0x7f2a94102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.698+0000 7f2a93e00700 1 --2- 172.21.15.28:0/116242155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a94102970 0x7f2a94102df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2a800098f0 tx=0x7f2a80009c00 comp rx=0 tx=0).stop 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.698+0000 7f2a93e00700 1 -- 172.21.15.28:0/116242155 shutdown_connections 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.698+0000 7f2a93e00700 1 --2- 172.21.15.28:0/116242155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a94103330 0x7f2a9410f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.698+0000 7f2a93e00700 1 --2- 172.21.15.28:0/116242155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a94108970 0x7f2a94108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.698+0000 7f2a93e00700 1 --2- 172.21.15.28:0/116242155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a94102970 0x7f2a94102df0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.698+0000 7f2a93e00700 1 -- 172.21.15.28:0/116242155 >> 172.21.15.28:0/116242155 conn(0x7f2a940fe4c0 msgr2=0x7f2a941008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.699+0000 7f2a93e00700 1 -- 172.21.15.28:0/116242155 shutdown_connections 2024-04-26T01:38:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.699+0000 7f2a93e00700 1 -- 172.21.15.28:0/116242155 wait complete. 2024-04-26T01:38:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.700+0000 7f2a93e00700 1 Processor -- start 2024-04-26T01:38:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.700+0000 7f2a93e00700 1 -- start start 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a93e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a94102970 0x7f2a9419b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a93e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a94103330 0x7f2a9419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a93e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a94108970 0x7f2a9419faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a93e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a94112b80 con 0x7f2a94102970 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a93e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2a94112a00 con 0x7f2a94108970 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a93e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2a94112d00 con 0x7f2a94103330 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a92a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a94102970 0x7f2a9419b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a92a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a94102970 0x7f2a9419b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42688/0 (socket says 172.21.15.28:42688) 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a92a00700 1 -- 172.21.15.28:0/213618045 learned_addr learned my addr 172.21.15.28:0/213618045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a92000700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a94103330 0x7f2a9419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.701+0000 7f2a93400700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a94108970 0x7f2a9419faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a92a00700 1 -- 172.21.15.28:0/213618045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a94103330 msgr2=0x7f2a9419b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a92a00700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a94103330 0x7f2a9419b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a92a00700 1 -- 172.21.15.28:0/213618045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a94108970 msgr2=0x7f2a9419faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a92a00700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a94108970 0x7f2a9419faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a92a00700 1 -- 172.21.15.28:0/213618045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a941a0230 con 0x7f2a94102970 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a93400700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a94108970 0x7f2a9419faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a92a00700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a94102970 0x7f2a9419b220 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400e9c0 tx=0x7f2a8400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a84004a30 con 0x7f2a94102970 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a8400cc70 con 0x7f2a94102970 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a941a0520 con 0x7f2a94102970 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.702+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a84005490 con 0x7f2a94102970 2024-04-26T01:38:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.703+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a941a7da0 con 0x7f2a94102970 2024-04-26T01:38:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.703+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a940ffc60 con 0x7f2a94102970 2024-04-26T01:38:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.706+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2a84004bd0 con 0x7f2a94102970 2024-04-26T01:38:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.707+0000 7f2a8b400700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2a780a3cd0 0x7f2a780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.707+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f2a840d0a40 con 0x7f2a94102970 2024-04-26T01:38:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.708+0000 7f2a92000700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2a780a3cd0 0x7f2a780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.708+0000 7f2a8b400700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] conn(0x7f2a780ad810 0x7f2a780afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.708+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 --> [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a8401ee50 con 0x7f2a780ad810 2024-04-26T01:38:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.708+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f2a840d0df0 con 0x7f2a94102970 2024-04-26T01:38:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.708+0000 7f2a93400700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] conn(0x7f2a780ad810 0x7f2a780afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.708+0000 7f2a92000700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2a780a3cd0 0x7f2a780a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009fd0 tx=0x7f2a80019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.709+0000 7f2a93400700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] conn(0x7f2a780ad810 0x7f2a780afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.710+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 <== osd.2 v2:172.21.15.169:6800/3360829572 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2a8401ee50 con 0x7f2a780ad810 2024-04-26T01:38:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.728+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 --> [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2a941040b0 con 0x7f2a780ad810 2024-04-26T01:38:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.729+0000 7f2a8b400700 1 -- 172.21.15.28:0/213618045 <== osd.2 v2:172.21.15.169:6800/3360829572 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2a941040b0 con 0x7f2a780ad810 2024-04-26T01:38:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.729+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 >> [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] conn(0x7f2a780ad810 msgr2=0x7f2a780afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.729+0000 7f2a93e00700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] conn(0x7f2a780ad810 0x7f2a780afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.729+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2a780a3cd0 msgr2=0x7f2a780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.729+0000 7f2a93e00700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2a780a3cd0 0x7f2a780a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009fd0 tx=0x7f2a80019040 comp rx=0 tx=0).stop 2024-04-26T01:38:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.729+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a94102970 msgr2=0x7f2a9419b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.729+0000 7f2a93e00700 1 --2- 172.21.15.28:0/213618045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a94102970 0x7f2a9419b220 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400e9c0 tx=0x7f2a8400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.730+0000 7f2a93400700 1 -- 172.21.15.28:0/213618045 reap_dead start 2024-04-26T01:38:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.730+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 shutdown_connections 2024-04-26T01:38:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.730+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 >> 172.21.15.28:0/213618045 conn(0x7f2a940fe4c0 msgr2=0x7f2a9410e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.730+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 shutdown_connections 2024-04-26T01:38:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:40.730+0000 7f2a93e00700 1 -- 172.21.15.28:0/213618045 wait complete. 2024-04-26T01:38:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 -- 172.21.15.28:0/1190856497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 msgr2=0x7f7b4806b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 --2- 172.21.15.28:0/1190856497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 0x7f7b4806b440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009990 tx=0x7f7b34009ca0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 -- 172.21.15.28:0/1190856497 shutdown_connections 2024-04-26T01:38:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 --2- 172.21.15.28:0/1190856497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b4806b980 0x7f7b4810f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 --2- 172.21.15.28:0/1190856497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b48108a20 0x7f7b48108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 --2- 172.21.15.28:0/1190856497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 0x7f7b4806b440 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 -- 172.21.15.28:0/1190856497 >> 172.21.15.28:0/1190856497 conn(0x7f7b480788b0 msgr2=0x7f7b48078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 -- 172.21.15.28:0/1190856497 shutdown_connections 2024-04-26T01:38:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.995+0000 7f7b4f400700 1 -- 172.21.15.28:0/1190856497 wait complete. 2024-04-26T01:38:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.997+0000 7f7b4f400700 1 Processor -- start 2024-04-26T01:38:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.997+0000 7f7b4f400700 1 -- start start 2024-04-26T01:38:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 0x7f7b48199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4f400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b4806b980 0x7f7b48199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4f400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b48108a20 0x7f7b4819da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4f400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48112c10 con 0x7f7b4806afe0 2024-04-26T01:38:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4f400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48112a90 con 0x7f7b4806b980 2024-04-26T01:38:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4f400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48112d90 con 0x7f7b48108a20 2024-04-26T01:38:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 0x7f7b48199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 0x7f7b48199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44548/0 (socket says 172.21.15.28:44548) 2024-04-26T01:38:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4ca00700 1 -- 172.21.15.28:0/3275468998 learned_addr learned my addr 172.21.15.28:0/3275468998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b4d400700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b48108a20 0x7f7b4819da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.998+0000 7f7b47e00700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b4806b980 0x7f7b48199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4ca00700 1 -- 172.21.15.28:0/3275468998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b48108a20 msgr2=0x7f7b4819da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4ca00700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b48108a20 0x7f7b4819da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4ca00700 1 -- 172.21.15.28:0/3275468998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b4806b980 msgr2=0x7f7b48199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4ca00700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b4806b980 0x7f7b48199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4ca00700 1 -- 172.21.15.28:0/3275468998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4819e150 con 0x7f7b4806afe0 2024-04-26T01:38:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4d400700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b48108a20 0x7f7b4819da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4ca00700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 0x7f7b48199140 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00e9c0 tx=0x7f7b3c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00cbd0 con 0x7f7b4806afe0 2024-04-26T01:38:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4819e440 con 0x7f7b4806afe0 2024-04-26T01:38:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b3c00ee60 con 0x7f7b4806afe0 2024-04-26T01:38:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c0055c0 con 0x7f7b4806afe0 2024-04-26T01:38:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:34.999+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b481a5cc0 con 0x7f7b4806afe0 2024-04-26T01:38:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.000+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b480631d0 con 0x7f7b4806afe0 2024-04-26T01:38:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.002+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7b3c004750 con 0x7f7b4806afe0 2024-04-26T01:38:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.004+0000 7f7b45600700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7b300a3d50 0x7f7b300a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.004+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f7b3c0d0b70 con 0x7f7b4806afe0 2024-04-26T01:38:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.004+0000 7f7b47e00700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7b300a3d50 0x7f7b300a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.004+0000 7f7b45600700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] conn(0x7f7b300ad890 0x7f7b300afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.004+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 --> [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b300b0400 con 0x7f7b300ad890 2024-04-26T01:38:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.004+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f7b3c0d0f20 con 0x7f7b4806afe0 2024-04-26T01:38:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.005+0000 7f7b4d400700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] conn(0x7f7b300ad890 0x7f7b300afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.005+0000 7f7b47e00700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7b300a3d50 0x7f7b300a6210 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7b48078f20 tx=0x7f7b34005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.005+0000 7f7b4d400700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] conn(0x7f7b300ad890 0x7f7b300afd30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.006+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 <== osd.0 v2:172.21.15.28:6802/3881623886 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b300b0400 con 0x7f7b300ad890 2024-04-26T01:38:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.025+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 --> [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b48063650 con 0x7f7b300ad890 2024-04-26T01:38:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.025+0000 7f7b45600700 1 -- 172.21.15.28:0/3275468998 <== osd.0 v2:172.21.15.28:6802/3881623886 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7b48063650 con 0x7f7b300ad890 2024-04-26T01:38:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] conn(0x7f7b300ad890 msgr2=0x7f7b300afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] conn(0x7f7b300ad890 0x7f7b300afd30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7b300a3d50 msgr2=0x7f7b300a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7b300a3d50 0x7f7b300a6210 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7b48078f20 tx=0x7f7b34005fd0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 msgr2=0x7f7b48199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 --2- 172.21.15.28:0/3275468998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b4806afe0 0x7f7b48199140 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00e9c0 tx=0x7f7b3c00ecd0 comp rx=0 tx=0).stop 2024-04-26T01:38:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4d400700 1 -- 172.21.15.28:0/3275468998 reap_dead start 2024-04-26T01:38:40.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 shutdown_connections 2024-04-26T01:38:40.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 >> 172.21.15.28:0/3275468998 conn(0x7f7b480788b0 msgr2=0x7f7b480ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:40.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 shutdown_connections 2024-04-26T01:38:40.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:35.026+0000 7f7b4f400700 1 -- 172.21.15.28:0/3275468998 wait complete. 2024-04-26T01:38:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.300+0000 7f65b3000700 1 -- 172.21.15.28:0/3360220388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac06afe0 msgr2=0x7f65ac06b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.300+0000 7f65b3000700 1 --2- 172.21.15.28:0/3360220388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac06afe0 0x7f65ac06b440 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6594009990 tx=0x7f6594009ca0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.301+0000 7f65b3000700 1 -- 172.21.15.28:0/3360220388 shutdown_connections 2024-04-26T01:38:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.301+0000 7f65b3000700 1 --2- 172.21.15.28:0/3360220388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ac06b980 0x7f65ac10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.301+0000 7f65b3000700 1 --2- 172.21.15.28:0/3360220388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65ac108a20 0x7f65ac108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.301+0000 7f65b3000700 1 --2- 172.21.15.28:0/3360220388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac06afe0 0x7f65ac06b440 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.301+0000 7f65b3000700 1 -- 172.21.15.28:0/3360220388 >> 172.21.15.28:0/3360220388 conn(0x7f65ac0788b0 msgr2=0x7f65ac078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.301+0000 7f65b3000700 1 -- 172.21.15.28:0/3360220388 shutdown_connections 2024-04-26T01:38:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.301+0000 7f65b3000700 1 -- 172.21.15.28:0/3360220388 wait complete. 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.302+0000 7f65b3000700 1 Processor -- start 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.302+0000 7f65b3000700 1 -- start start 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.302+0000 7f65b3000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ac06afe0 0x7f65ac19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.302+0000 7f65b3000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65ac06b980 0x7f65ac19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b3000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac108a20 0x7f65ac19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b3000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac112be0 con 0x7f65ac06afe0 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b3000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac112a60 con 0x7f65ac108a20 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b3000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac112d60 con 0x7f65ac06b980 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65abe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ac06afe0 0x7f65ac19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65abe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ac06afe0 0x7f65ac19b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42714/0 (socket says 172.21.15.28:42714) 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65abe00700 1 -- 172.21.15.28:0/4044001630 learned_addr learned my addr 172.21.15.28:0/4044001630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b1000700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac108a20 0x7f65ac19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65a3e00700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65ac06b980 0x7f65ac19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b1000700 1 -- 172.21.15.28:0/4044001630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65ac06b980 msgr2=0x7f65ac19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b1000700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65ac06b980 0x7f65ac19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b1000700 1 -- 172.21.15.28:0/4044001630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ac06afe0 msgr2=0x7f65ac19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b1000700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ac06afe0 0x7f65ac19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65b1000700 1 -- 172.21.15.28:0/4044001630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac1a03a0 con 0x7f65ac108a20 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.303+0000 7f65abe00700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ac06afe0 0x7f65ac19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.304+0000 7f65b1000700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac108a20 0x7f65ac19fc60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f65a400efc0 tx=0x7f65a400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.304+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a40092e0 con 0x7f65ac108a20 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.304+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65a4004bb0 con 0x7f65ac108a20 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.304+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ac1a0630 con 0x7f65ac108a20 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.304+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4004020 con 0x7f65ac108a20 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.304+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ac1a7f10 con 0x7f65ac108a20 2024-04-26T01:38:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.305+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f65ac063110 con 0x7f65ac108a20 2024-04-26T01:38:41.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.308+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f65a401b030 con 0x7f65ac108a20 2024-04-26T01:38:41.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.309+0000 7f65a9600700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f65900a3cd0 0x7f65900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.309+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f65a40d0a50 con 0x7f65ac108a20 2024-04-26T01:38:41.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.309+0000 7f65abe00700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f65900a3cd0 0x7f65900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.309+0000 7f65a9600700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] conn(0x7f65900ad810 0x7f65900afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.309+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 --> [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65900b0380 con 0x7f65900ad810 2024-04-26T01:38:41.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.309+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f65a40d0e00 con 0x7f65ac108a20 2024-04-26T01:38:41.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.310+0000 7f65a3e00700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] conn(0x7f65900ad810 0x7f65900afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.310+0000 7f65abe00700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f65900a3cd0 0x7f65900a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f659c011550 tx=0x7f659c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:41.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.310+0000 7f65a3e00700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] conn(0x7f65900ad810 0x7f65900afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:41.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.311+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 <== osd.10 v2:172.21.15.53:6824/786160308 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f65900b0380 con 0x7f65900ad810 2024-04-26T01:38:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.330+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 --> [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f65ac063590 con 0x7f65900ad810 2024-04-26T01:38:41.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.330+0000 7f65a9600700 1 -- 172.21.15.28:0/4044001630 <== osd.10 v2:172.21.15.53:6824/786160308 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f65ac063590 con 0x7f65900ad810 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] conn(0x7f65900ad810 msgr2=0x7f65900afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b3000700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] conn(0x7f65900ad810 0x7f65900afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f65900a3cd0 msgr2=0x7f65900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b3000700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f65900a3cd0 0x7f65900a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f659c011550 tx=0x7f659c009260 comp rx=0 tx=0).stop 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac108a20 msgr2=0x7f65ac19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b3000700 1 --2- 172.21.15.28:0/4044001630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac108a20 0x7f65ac19fc60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f65a400efc0 tx=0x7f65a400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b1000700 1 -- 172.21.15.28:0/4044001630 reap_dead start 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 shutdown_connections 2024-04-26T01:38:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.331+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 >> 172.21.15.28:0/4044001630 conn(0x7f65ac0788b0 msgr2=0x7f65ac10dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:41.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.332+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 shutdown_connections 2024-04-26T01:38:41.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.332+0000 7f65b3000700 1 -- 172.21.15.28:0/4044001630 wait complete. 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.529+0000 7f425a000700 1 -- 172.21.15.28:0/90344440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4254103450 msgr2=0x7f425410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.529+0000 7f425a000700 1 --2- 172.21.15.28:0/90344440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4254103450 0x7f425410f8b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f424c00b0a0 tx=0x7f424c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.530+0000 7f425a000700 1 -- 172.21.15.28:0/90344440 shutdown_connections 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.530+0000 7f425a000700 1 --2- 172.21.15.28:0/90344440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4254103450 0x7f425410f8b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.530+0000 7f425a000700 1 --2- 172.21.15.28:0/90344440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254108a50 0x7f4254108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.530+0000 7f425a000700 1 --2- 172.21.15.28:0/90344440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4254102a90 0x7f4254102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.530+0000 7f425a000700 1 -- 172.21.15.28:0/90344440 >> 172.21.15.28:0/90344440 conn(0x7f4254079ba0 msgr2=0x7f4254079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.530+0000 7f425a000700 1 -- 172.21.15.28:0/90344440 shutdown_connections 2024-04-26T01:38:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.530+0000 7f425a000700 1 -- 172.21.15.28:0/90344440 wait complete. 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.531+0000 7f425a000700 1 Processor -- start 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f425a000700 1 -- start start 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f425a000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4254102a90 0x7f425419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f425a000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4254103450 0x7f425419b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f425a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254108a50 0x7f425419fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f425a000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4254112c20 con 0x7f4254108a50 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f425a000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4254112aa0 con 0x7f4254102a90 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f425a000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4254112da0 con 0x7f4254103450 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f424aa00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4254103450 0x7f425419b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f4253e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254108a50 0x7f425419fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f4253400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4254102a90 0x7f425419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f424aa00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4254103450 0x7f425419b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:33988/0 (socket says 172.21.15.28:33988) 2024-04-26T01:38:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.532+0000 7f424aa00700 1 -- 172.21.15.28:0/1872306515 learned_addr learned my addr 172.21.15.28:0/1872306515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.533+0000 7f4253400700 1 -- 172.21.15.28:0/1872306515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4254103450 msgr2=0x7f425419b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.533+0000 7f4253400700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4254103450 0x7f425419b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.533+0000 7f4253400700 1 -- 172.21.15.28:0/1872306515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254108a50 msgr2=0x7f425419fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.533+0000 7f4253400700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254108a50 0x7f425419fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.533+0000 7f4253400700 1 -- 172.21.15.28:0/1872306515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42541a0300 con 0x7f4254102a90 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.533+0000 7f4253400700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4254102a90 0x7f425419b2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f424400e9f0 tx=0x7f424400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.533+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424400cb70 con 0x7f4254102a90 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.534+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f424400ee90 con 0x7f4254102a90 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.534+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244003d40 con 0x7f4254102a90 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.534+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42541a05f0 con 0x7f4254102a90 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.534+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42541a7e70 con 0x7f4254102a90 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.535+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4254063150 con 0x7f4254102a90 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.537+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4244003ee0 con 0x7f4254102a90 2024-04-26T01:38:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.538+0000 7f4250c00700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f42380a3d50 0x7f42380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.539+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f42440d0890 con 0x7f4254102a90 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.539+0000 7f424aa00700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f42380a3d50 0x7f42380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.539+0000 7f4250c00700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] conn(0x7f42380ad890 0x7f42380afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.539+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 --> [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42380b0400 con 0x7f42380ad890 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.539+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f42440d0c40 con 0x7f4254102a90 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.539+0000 7f4253e00700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] conn(0x7f42380ad890 0x7f42380afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.539+0000 7f424aa00700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f42380a3d50 0x7f42380a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f425407a4a0 tx=0x7f423c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.540+0000 7f4253e00700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] conn(0x7f42380ad890 0x7f42380afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.541+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 <== osd.5 v2:172.21.15.169:6808/1028981135 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42380b0400 con 0x7f42380ad890 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.560+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 --> [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4254063550 con 0x7f42380ad890 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.561+0000 7f4250c00700 1 -- 172.21.15.28:0/1872306515 <== osd.5 v2:172.21.15.169:6808/1028981135 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4254063550 con 0x7f42380ad890 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 >> [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] conn(0x7f42380ad890 msgr2=0x7f42380afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] conn(0x7f42380ad890 0x7f42380afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f42380a3d50 msgr2=0x7f42380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f42380a3d50 0x7f42380a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f425407a4a0 tx=0x7f423c009260 comp rx=0 tx=0).stop 2024-04-26T01:38:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4254102a90 msgr2=0x7f425419b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 --2- 172.21.15.28:0/1872306515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4254102a90 0x7f425419b2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f424400e9f0 tx=0x7f424400ed00 comp rx=0 tx=0).stop 2024-04-26T01:38:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f4253e00700 1 -- 172.21.15.28:0/1872306515 reap_dead start 2024-04-26T01:38:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 shutdown_connections 2024-04-26T01:38:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 >> 172.21.15.28:0/1872306515 conn(0x7f4254079ba0 msgr2=0x7f425410df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 shutdown_connections 2024-04-26T01:38:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:41.562+0000 7f425a000700 1 -- 172.21.15.28:0/1872306515 wait complete. 2024-04-26T01:38:41.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:41 smithi028 ceph-mon[28247]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:38:41.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:41 smithi169 ceph-mon[32851]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:38:41.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:41 smithi053 ceph-mon[32593]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:38:42.664 INFO:teuthology.orchestra.run.smithi028.stdout:55834574943 2024-04-26T01:38:42.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T01:38:43.739 INFO:teuthology.orchestra.run.smithi028.stdout:163208757323 2024-04-26T01:38:43.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T01:38:43.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:43 smithi028 ceph-mon[28247]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-26T01:38:43.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:43 smithi169 ceph-mon[32851]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-26T01:38:43.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:43 smithi053 ceph-mon[32593]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-26T01:38:44.476 INFO:teuthology.orchestra.run.smithi028.stdout:77309411421 2024-04-26T01:38:44.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T01:38:44.659 INFO:teuthology.orchestra.run.smithi028.stdout:34359738470 2024-04-26T01:38:44.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T01:38:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 -- 172.21.15.28:0/1647811992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb31010e3b0 msgr2=0x7fb310115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 --2- 172.21.15.28:0/1647811992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb31010e3b0 0x7fb310115f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb30800b0d0 tx=0x7fb30800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 -- 172.21.15.28:0/1647811992 shutdown_connections 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 --2- 172.21.15.28:0/1647811992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb31010e3b0 0x7fb310115f80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 --2- 172.21.15.28:0/1647811992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3101092d0 0x7fb3101096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 --2- 172.21.15.28:0/1647811992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb310109c80 0x7fb31010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 -- 172.21.15.28:0/1647811992 >> 172.21.15.28:0/1647811992 conn(0x7fb310076210 msgr2=0x7fb310107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 -- 172.21.15.28:0/1647811992 shutdown_connections 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.435+0000 7fb315a00700 1 -- 172.21.15.28:0/1647811992 wait complete. 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.436+0000 7fb315a00700 1 Processor -- start 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb315a00700 1 -- start start 2024-04-26T01:38:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb315a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3101092d0 0x7fb31019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb315a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb310109c80 0x7fb31019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb315a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb31010e3b0 0x7fb3101a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb315a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb310119250 con 0x7fb3101092d0 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb315a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb3101190d0 con 0x7fb310109c80 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb315a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb3101193d0 con 0x7fb31010e3b0 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb306000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb310109c80 0x7fb31019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb30ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3101092d0 0x7fb31019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb306000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb310109c80 0x7fb31019fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47262/0 (socket says 172.21.15.28:47262) 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb306000700 1 -- 172.21.15.28:0/2811335969 learned_addr learned my addr 172.21.15.28:0/2811335969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.437+0000 7fb30f400700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb31010e3b0 0x7fb3101a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb306000700 1 -- 172.21.15.28:0/2811335969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb31010e3b0 msgr2=0x7fb3101a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb306000700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb31010e3b0 0x7fb3101a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb306000700 1 -- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3101092d0 msgr2=0x7fb31019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb306000700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3101092d0 0x7fb31019f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb306000700 1 -- 172.21.15.28:0/2811335969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3101a4690 con 0x7fb310109c80 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb306000700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb310109c80 0x7fb31019fbc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800ea30 tx=0x7fb2f800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f800cbd0 con 0x7fb310109c80 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2f80044d0 con 0x7fb310109c80 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3101a4980 con 0x7fb310109c80 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.438+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8003dc0 con 0x7fb310109c80 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.439+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3101ac200 con 0x7fb310109c80 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.440+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3100631d0 con 0x7fb310109c80 2024-04-26T01:38:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.442+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb2f8004000 con 0x7fb310109c80 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.443+0000 7fb307e00700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2f00a3cd0 0x7fb2f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.443+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fb2f80d0990 con 0x7fb310109c80 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.443+0000 7fb30ea00700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2f00a3cd0 0x7fb2f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.444+0000 7fb307e00700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] conn(0x7fb2f00ad810 0x7fb2f00afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.444+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 --> [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2f00b0380 con 0x7fb2f00ad810 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.444+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb2f80d0d40 con 0x7fb310109c80 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.444+0000 7fb30f400700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] conn(0x7fb2f00ad810 0x7fb2f00afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.444+0000 7fb30ea00700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2f00a3cd0 0x7fb2f00a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb300009680 tx=0x7fb300009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.445+0000 7fb30f400700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] conn(0x7fb2f00ad810 0x7fb2f00afcb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.446+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 <== osd.3 v2:172.21.15.28:6810/1623836559 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb2f00b0380 con 0x7fb2f00ad810 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.464+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 --> [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb310063650 con 0x7fb2f00ad810 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.464+0000 7fb307e00700 1 -- 172.21.15.28:0/2811335969 <== osd.3 v2:172.21.15.28:6810/1623836559 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb310063650 con 0x7fb2f00ad810 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] conn(0x7fb2f00ad810 msgr2=0x7fb2f00afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb315a00700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] conn(0x7fb2f00ad810 0x7fb2f00afcb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2f00a3cd0 msgr2=0x7fb2f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb315a00700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb2f00a3cd0 0x7fb2f00a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb300009680 tx=0x7fb300009260 comp rx=0 tx=0).stop 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb310109c80 msgr2=0x7fb31019fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb315a00700 1 --2- 172.21.15.28:0/2811335969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb310109c80 0x7fb31019fbc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800ea30 tx=0x7fb2f800ed40 comp rx=0 tx=0).stop 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb30f400700 1 -- 172.21.15.28:0/2811335969 reap_dead start 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 shutdown_connections 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.465+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 >> 172.21.15.28:0/2811335969 conn(0x7fb310076210 msgr2=0x7fb310076730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.466+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 shutdown_connections 2024-04-26T01:38:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:43.466+0000 7fb315a00700 1 -- 172.21.15.28:0/2811335969 wait complete. 2024-04-26T01:38:45.486 INFO:teuthology.orchestra.run.smithi028.stdout:107374182485 2024-04-26T01:38:45.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T01:38:45.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:45 smithi028 ceph-mon[28247]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-04-26T01:38:45.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:45 smithi169 ceph-mon[32851]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-04-26T01:38:45.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:45 smithi053 ceph-mon[32593]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-04-26T01:38:45.980 INFO:teuthology.orchestra.run.smithi028.stdout:309237645357 2024-04-26T01:38:45.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T01:38:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.538 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.566 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:46.569 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:47.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:47 smithi028 ceph-mon[28247]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:47.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:47 smithi169 ceph-mon[32851]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:47.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:47 smithi053 ceph-mon[32593]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:38:49.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:49 smithi028 ceph-mon[28247]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:49.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:49 smithi169 ceph-mon[32851]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:49.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:49 smithi053 ceph-mon[32593]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:38:51.752 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:51 smithi028 ceph-mon[28247]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:51.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:51 smithi169 ceph-mon[32851]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:51.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:51 smithi053 ceph-mon[32593]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:38:53.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:53 smithi169 ceph-mon[32851]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:53 smithi053 ceph-mon[32593]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:53.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:53 smithi028 ceph-mon[28247]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:38:55.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:55 smithi169 ceph-mon[32851]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:38:55.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:55 smithi169 ceph-mon[32851]: mgrmap e18: x(active, since 10m), standbys: y, z 2024-04-26T01:38:55.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:55 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:55.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:55 smithi028 ceph-mon[28247]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:38:55.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:55 smithi028 ceph-mon[28247]: mgrmap e18: x(active, since 10m), standbys: y, z 2024-04-26T01:38:55.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:55 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:55.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:55 smithi053 ceph-mon[32593]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:38:55.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:55 smithi053 ceph-mon[32593]: mgrmap e18: x(active, since 10m), standbys: y, z 2024-04-26T01:38:55.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:55 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.199+0000 7f91aae00700 1 -- 172.21.15.28:0/3828674218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4113c10 msgr2=0x7f91a4116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.199+0000 7f91aae00700 1 --2- 172.21.15.28:0/3828674218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4113c10 0x7f91a4116000 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f919c009f90 tx=0x7f919c00b330 comp rx=0 tx=0).stop 2024-04-26T01:38:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.199+0000 7f91aae00700 1 -- 172.21.15.28:0/3828674218 shutdown_connections 2024-04-26T01:38:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.199+0000 7f91aae00700 1 --2- 172.21.15.28:0/3828674218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4113c10 0x7f91a4116000 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.199+0000 7f91aae00700 1 --2- 172.21.15.28:0/3828674218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a4074c70 0x7f91a4075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.199+0000 7f91aae00700 1 --2- 172.21.15.28:0/3828674218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a4075590 0x7f91a41136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.199+0000 7f91aae00700 1 -- 172.21.15.28:0/3828674218 >> 172.21.15.28:0/3828674218 conn(0x7f91a4079ba0 msgr2=0x7f91a4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.200+0000 7f91aae00700 1 -- 172.21.15.28:0/3828674218 shutdown_connections 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.200+0000 7f91aae00700 1 -- 172.21.15.28:0/3828674218 wait complete. 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.201+0000 7f91aae00700 1 Processor -- start 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.201+0000 7f91aae00700 1 -- start start 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.201+0000 7f91aae00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a4074c70 0x7f91a41a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.201+0000 7f91aae00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4075590 0x7f91a41a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.201+0000 7f91aae00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a4113c10 0x7f91a419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.201+0000 7f91aae00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91a4119220 con 0x7f91a4074c70 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.201+0000 7f91aae00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f91a41190a0 con 0x7f91a4075590 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.201+0000 7f91aae00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f91a41193a0 con 0x7f91a4113c10 2024-04-26T01:38:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4075590 0x7f91a41a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a4074c70 0x7f91a41a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4075590 0x7f91a41a5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43006/0 (socket says 172.21.15.28:43006) 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 -- 172.21.15.28:0/3827824388 learned_addr learned my addr 172.21.15.28:0/3827824388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a8e00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a4113c10 0x7f91a419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 -- 172.21.15.28:0/3827824388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a4113c10 msgr2=0x7f91a419f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a4113c10 0x7f91a419f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 -- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a4074c70 msgr2=0x7f91a41a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a4074c70 0x7f91a41a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 -- 172.21.15.28:0/3827824388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a419fd40 con 0x7f91a4075590 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.202+0000 7f91a3400700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4075590 0x7f91a41a5960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f919000d950 tx=0x7f919000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.203+0000 7f91a0c00700 1 -- 172.21.15.28:0/3827824388 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190016070 con 0x7f91a4075590 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.203+0000 7f91a0c00700 1 -- 172.21.15.28:0/3827824388 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9190009de0 con 0x7f91a4075590 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.203+0000 7f91a0c00700 1 -- 172.21.15.28:0/3827824388 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919000f3c0 con 0x7f91a4075590 2024-04-26T01:38:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.203+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a41a0030 con 0x7f91a4075590 2024-04-26T01:38:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.203+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a41ac2f0 con 0x7f91a4075590 2024-04-26T01:38:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.204+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a4069730 con 0x7f91a4075590 2024-04-26T01:38:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.206+0000 7f91a0c00700 1 -- 172.21.15.28:0/3827824388 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9190019070 con 0x7f91a4075590 2024-04-26T01:38:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.207+0000 7f91a0c00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f91880a3cc0 0x7f91880a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.207+0000 7f91a0c00700 1 -- 172.21.15.28:0/3827824388 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f91900cfe40 con 0x7f91a4075590 2024-04-26T01:38:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.207+0000 7f91a3e00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f91880a3cc0 0x7f91880a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.208+0000 7f91a3e00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f91880a3cc0 0x7f91880a6180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9194005fd0 tx=0x7f9194009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.212+0000 7f91a0c00700 1 -- 172.21.15.28:0/3827824388 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f919009c7d0 con 0x7f91a4075590 2024-04-26T01:38:56.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.443+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f91a41ac4c0 con 0x7f91a4075590 2024-04-26T01:38:56.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.443+0000 7f91a0c00700 1 -- 172.21.15.28:0/3827824388 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f919009c0d0 con 0x7f91a4075590 2024-04-26T01:38:56.440 INFO:teuthology.orchestra.run.smithi028.stdout:107374182487 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.446+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f91880a3cc0 msgr2=0x7f91880a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.446+0000 7f91aae00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f91880a3cc0 0x7f91880a6180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9194005fd0 tx=0x7f9194009310 comp rx=0 tx=0).stop 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.446+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4075590 msgr2=0x7f91a41a5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.446+0000 7f91aae00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4075590 0x7f91a41a5960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f919000d950 tx=0x7f919000dc60 comp rx=0 tx=0).stop 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.447+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 shutdown_connections 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.447+0000 7f91aae00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f91880a3cc0 0x7f91880a6180 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.447+0000 7f91aae00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a4113c10 0x7f91a419f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.447+0000 7f91aae00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a4074c70 0x7f91a41a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.447+0000 7f91aae00700 1 --2- 172.21.15.28:0/3827824388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91a4075590 0x7f91a41a5960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.447+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 >> 172.21.15.28:0/3827824388 conn(0x7f91a4079ba0 msgr2=0x7f91a41109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.447+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 shutdown_connections 2024-04-26T01:38:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:38:56.447+0000 7f91aae00700 1 -- 172.21.15.28:0/3827824388 wait complete. 2024-04-26T01:38:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:56 smithi028 ceph-mon[28247]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:56 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:56 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:57.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:56 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3827824388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:38:57.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:56 smithi169 ceph-mon[32851]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:57.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:56 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:57.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:56 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:57.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:56 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3827824388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:38:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:56 smithi053 ceph-mon[32593]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T01:38:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:56 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:56 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:56 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3827824388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:38:58.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:57 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:57 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:57 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:57 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:57 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:57 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:57 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:57 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:57 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:57 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:57 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:58.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:57 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:38:59.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:38:58 smithi028 ceph-mon[28247]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:38:59.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:38:58 smithi169 ceph-mon[32851]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:38:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:38:58 smithi053 ceph-mon[32593]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.011+0000 7f1820a00700 1 -- 172.21.15.28:0/2565446154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 msgr2=0x7f181c10f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.011+0000 7f1820a00700 1 --2- 172.21.15.28:0/2565446154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 0x7f181c10f410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f180c009990 tx=0x7f180c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.012+0000 7f1820a00700 1 -- 172.21.15.28:0/2565446154 shutdown_connections 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.012+0000 7f1820a00700 1 --2- 172.21.15.28:0/2565446154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f181c109940 0x7f181c115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.012+0000 7f1820a00700 1 --2- 172.21.15.28:0/2565446154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 0x7f181c10f410 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.012+0000 7f1820a00700 1 --2- 172.21.15.28:0/2565446154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c108f80 0x7f181c109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.012+0000 7f1820a00700 1 -- 172.21.15.28:0/2565446154 >> 172.21.15.28:0/2565446154 conn(0x7f181c0787d0 msgr2=0x7f181c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.012+0000 7f1820a00700 1 -- 172.21.15.28:0/2565446154 shutdown_connections 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.012+0000 7f1820a00700 1 -- 172.21.15.28:0/2565446154 wait complete. 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.013+0000 7f1820a00700 1 Processor -- start 2024-04-26T01:39:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.013+0000 7f1820a00700 1 -- start start 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.013+0000 7f1820a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c108f80 0x7f181c19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.013+0000 7f1820a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f181c109940 0x7f181c19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.013+0000 7f1820a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 0x7f181c1a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.013+0000 7f1820a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1190f0 con 0x7f181c10f030 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.013+0000 7f1820a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f181c118f70 con 0x7f181c108f80 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.013+0000 7f1820a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f181c119270 con 0x7f181c109940 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181a800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c108f80 0x7f181c19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181a800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c108f80 0x7f181c19f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38790/0 (socket says 172.21.15.28:38790) 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 0x7f181c1a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181a800700 1 -- 172.21.15.28:0/1284093717 learned_addr learned my addr 172.21.15.28:0/1284093717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f1813e00700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f181c109940 0x7f181c19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181b200700 1 -- 172.21.15.28:0/1284093717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f181c109940 msgr2=0x7f181c19fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181b200700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f181c109940 0x7f181c19fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181b200700 1 -- 172.21.15.28:0/1284093717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c108f80 msgr2=0x7f181c19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181b200700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c108f80 0x7f181c19f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f181b200700 1 -- 172.21.15.28:0/1284093717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181c1a4550 con 0x7f181c10f030 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.014+0000 7f1813e00700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f181c109940 0x7f181c19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.015+0000 7f181b200700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 0x7f181c1a3e10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f181400efc0 tx=0x7f181400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.015+0000 7f1813400700 1 -- 172.21.15.28:0/1284093717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18140092e0 con 0x7f181c10f030 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.015+0000 7f1813400700 1 -- 172.21.15.28:0/1284093717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1814004bb0 con 0x7f181c10f030 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.015+0000 7f1813400700 1 -- 172.21.15.28:0/1284093717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1814004020 con 0x7f181c10f030 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.015+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f181c1a47e0 con 0x7f181c10f030 2024-04-26T01:39:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.015+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f181c1ac0c0 con 0x7f181c10f030 2024-04-26T01:39:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.016+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f181c069730 con 0x7f181c10f030 2024-04-26T01:39:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.018+0000 7f1813400700 1 -- 172.21.15.28:0/1284093717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f181401b030 con 0x7f181c10f030 2024-04-26T01:39:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.020+0000 7f1813400700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f18000a3c40 0x7f18000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.020+0000 7f1813400700 1 -- 172.21.15.28:0/1284093717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f18140cff10 con 0x7f181c10f030 2024-04-26T01:39:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.020+0000 7f181a800700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f18000a3c40 0x7f18000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.021+0000 7f181a800700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f18000a3c40 0x7f18000a6100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f180c005cd0 tx=0x7f180c005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.025+0000 7f1813400700 1 -- 172.21.15.28:0/1284093717 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1814022020 con 0x7f181c10f030 2024-04-26T01:39:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.273+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f181c02df80 con 0x7f181c10f030 2024-04-26T01:39:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.274+0000 7f1813400700 1 -- 172.21.15.28:0/1284093717 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1814010040 con 0x7f181c10f030 2024-04-26T01:39:00.271 INFO:teuthology.orchestra.run.smithi028.stdout:55834574949 2024-04-26T01:39:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f18000a3c40 msgr2=0x7f18000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f18000a3c40 0x7f18000a6100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f180c005cd0 tx=0x7f180c005c60 comp rx=0 tx=0).stop 2024-04-26T01:39:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 msgr2=0x7f181c1a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 0x7f181c1a3e10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f181400efc0 tx=0x7f181400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 shutdown_connections 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f18000a3c40 0x7f18000a6100 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c10f030 0x7f181c1a3e10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c108f80 0x7f181c19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 --2- 172.21.15.28:0/1284093717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f181c109940 0x7f181c19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 >> 172.21.15.28:0/1284093717 conn(0x7f181c0787d0 msgr2=0x7f181c079b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.277+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 shutdown_connections 2024-04-26T01:39:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:00.278+0000 7f1820a00700 1 -- 172.21.15.28:0/1284093717 wait complete. 2024-04-26T01:39:01.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:00 smithi028 ceph-mon[28247]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:01.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:00 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1284093717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:39:01.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:00 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:01.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:00 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:01.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:00 smithi169 ceph-mon[32851]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:01.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:00 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1284093717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:39:01.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:00 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:01.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:00 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 -- 172.21.15.28:0/3631235074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa4075590 msgr2=0x7f3fa410ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3631235074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa4075590 0x7f3fa410ef60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009990 tx=0x7f3f90009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 -- 172.21.15.28:0/3631235074 shutdown_connections 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3631235074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fa410f4a0 0x7f3fa4111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3631235074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa4074c70 0x7f3fa4075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3631235074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa4075590 0x7f3fa410ef60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 -- 172.21.15.28:0/3631235074 >> 172.21.15.28:0/3631235074 conn(0x7f3fa4079ba0 msgr2=0x7f3fa4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 -- 172.21.15.28:0/3631235074 shutdown_connections 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.370+0000 7f3fa9000700 1 -- 172.21.15.28:0/3631235074 wait complete. 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.372+0000 7f3fa9000700 1 Processor -- start 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.372+0000 7f3fa9000700 1 -- start start 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa9000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa4074c70 0x7f3fa419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa9000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fa4075590 0x7f3fa419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa9000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410f4a0 0x7f3fa419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa9000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa4114eb0 con 0x7f3fa410f4a0 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa9000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa4114d30 con 0x7f3fa4075590 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa9000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa4115030 con 0x7f3fa4074c70 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa2800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410f4a0 0x7f3fa419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa4074c70 0x7f3fa419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fa4075590 0x7f3fa419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa4074c70 0x7f3fa419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55382/0 (socket says 172.21.15.28:55382) 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1e00700 1 -- 172.21.15.28:0/3745247516 learned_addr learned my addr 172.21.15.28:0/3745247516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fa4075590 0x7f3fa419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38808/0 (socket says 172.21.15.28:38808) 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa2800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410f4a0 0x7f3fa419fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41916/0 (socket says 172.21.15.28:41916) 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1e00700 1 -- 172.21.15.28:0/3745247516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fa4075590 msgr2=0x7f3fa419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1e00700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fa4075590 0x7f3fa419b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1e00700 1 -- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410f4a0 msgr2=0x7f3fa419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1e00700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410f4a0 0x7f3fa419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.373+0000 7f3fa1e00700 1 -- 172.21.15.28:0/3745247516 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa41a0330 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.374+0000 7f3fa1e00700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa4074c70 0x7f3fa419b320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400e9a0 tx=0x7f3f9400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.374+0000 7f3f9aa00700 1 -- 172.21.15.28:0/3745247516 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9400ca10 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.374+0000 7f3f9aa00700 1 -- 172.21.15.28:0/3745247516 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9400cbb0 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.374+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa41a0620 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.374+0000 7f3f9aa00700 1 -- 172.21.15.28:0/3745247516 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94018680 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.374+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa41a7ea0 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.375+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa4069730 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.377+0000 7f3f9aa00700 1 -- 172.21.15.28:0/3745247516 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3f94010430 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.379+0000 7f3f9aa00700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f880a3c40 0x7f3f880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.379+0000 7f3f9aa00700 1 -- 172.21.15.28:0/3745247516 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f3f940d05d0 con 0x7f3fa4074c70 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.379+0000 7f3fa1400700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f880a3c40 0x7f3f880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.380+0000 7f3fa1400700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f880a3c40 0x7f3f880a6100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3fa41111a0 tx=0x7f3f9000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.383+0000 7f3f9aa00700 1 -- 172.21.15.28:0/3745247516 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f9409cf60 con 0x7f3fa4074c70 2024-04-26T01:39:01.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:00 smithi053 ceph-mon[32593]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:01.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:00 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1284093717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:39:01.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:00 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:01.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:00 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.640+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3fa402df80 con 0x7f3fa4074c70 2024-04-26T01:39:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.641+0000 7f3f9aa00700 1 -- 172.21.15.28:0/3745247516 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3f9409c860 con 0x7f3fa4074c70 2024-04-26T01:39:01.638 INFO:teuthology.orchestra.run.smithi028.stdout:163208757327 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f880a3c40 msgr2=0x7f3f880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f880a3c40 0x7f3f880a6100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3fa41111a0 tx=0x7f3f9000b260 comp rx=0 tx=0).stop 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa4074c70 msgr2=0x7f3fa419b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa4074c70 0x7f3fa419b320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400e9a0 tx=0x7f3f9400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 shutdown_connections 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f3f880a3c40 0x7f3f880a6100 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410f4a0 0x7f3fa419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa4074c70 0x7f3fa419b320 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 --2- 172.21.15.28:0/3745247516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fa4075590 0x7f3fa419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.644+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 >> 172.21.15.28:0/3745247516 conn(0x7f3fa4079ba0 msgr2=0x7f3fa40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.645+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 shutdown_connections 2024-04-26T01:39:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.645+0000 7f3fa9000700 1 -- 172.21.15.28:0/3745247516 wait complete. 2024-04-26T01:39:01.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 -- 172.21.15.28:0/2887446868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8108a50 msgr2=0x7fb7a8108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:01.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 --2- 172.21.15.28:0/2887446868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8108a50 0x7fb7a8108e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009990 tx=0x7fb79c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 -- 172.21.15.28:0/2887446868 shutdown_connections 2024-04-26T01:39:01.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 --2- 172.21.15.28:0/2887446868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7a8103450 0x7fb7a810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 --2- 172.21.15.28:0/2887446868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8108a50 0x7fb7a8108e30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 --2- 172.21.15.28:0/2887446868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a8102a90 0x7fb7a8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 -- 172.21.15.28:0/2887446868 >> 172.21.15.28:0/2887446868 conn(0x7fb7a8079ba0 msgr2=0x7fb7a8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 -- 172.21.15.28:0/2887446868 shutdown_connections 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.851+0000 7fb7af600700 1 -- 172.21.15.28:0/2887446868 wait complete. 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.853+0000 7fb7af600700 1 Processor -- start 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.853+0000 7fb7af600700 1 -- start start 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.853+0000 7fb7af600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a8102a90 0x7fb7a819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.853+0000 7fb7af600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8103450 0x7fb7a819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.853+0000 7fb7af600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7a8108a50 0x7fb7a819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.853+0000 7fb7af600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8112c30 con 0x7fb7a8103450 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.853+0000 7fb7af600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8112ab0 con 0x7fb7a8102a90 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.853+0000 7fb7af600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8112db0 con 0x7fb7a8108a50 2024-04-26T01:39:01.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7a7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8103450 0x7fb7a819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7a7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8103450 0x7fb7a819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41932/0 (socket says 172.21.15.28:41932) 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7a7e00700 1 -- 172.21.15.28:0/4233413985 learned_addr learned my addr 172.21.15.28:0/4233413985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7acc00700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a8102a90 0x7fb7a819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7ad600700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7a8108a50 0x7fb7a819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7a7e00700 1 -- 172.21.15.28:0/4233413985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7a8108a50 msgr2=0x7fb7a819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7a7e00700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7a8108a50 0x7fb7a819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7a7e00700 1 -- 172.21.15.28:0/4233413985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a8102a90 msgr2=0x7fb7a819b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7a7e00700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a8102a90 0x7fb7a819b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7a7e00700 1 -- 172.21.15.28:0/4233413985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a81a03c0 con 0x7fb7a8103450 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.854+0000 7fb7acc00700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a8102a90 0x7fb7a819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.855+0000 7fb7a7e00700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8103450 0x7fb7a819b8f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb79400eb30 tx=0x7fb79400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.855+0000 7fb7a5600700 1 -- 172.21.15.28:0/4233413985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79400cbd0 con 0x7fb7a8103450 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.855+0000 7fb7a5600700 1 -- 172.21.15.28:0/4233413985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7940044d0 con 0x7fb7a8103450 2024-04-26T01:39:02.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.855+0000 7fb7a5600700 1 -- 172.21.15.28:0/4233413985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794003e30 con 0x7fb7a8103450 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.855+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a81a06b0 con 0x7fb7a8103450 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.855+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a81a7f30 con 0x7fb7a8103450 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.856+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a8069730 con 0x7fb7a8103450 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.860+0000 7fb7a5600700 1 -- 172.21.15.28:0/4233413985 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb794004000 con 0x7fb7a8103450 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.862+0000 7fb7a5600700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7900a3cc0 0x7fb7900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.862+0000 7fb7a5600700 1 -- 172.21.15.28:0/4233413985 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fb7940d0740 con 0x7fb7a8103450 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.862+0000 7fb7acc00700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7900a3cc0 0x7fb7900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.863+0000 7fb7acc00700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7900a3cc0 0x7fb7900a6180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb79c006010 tx=0x7fb79c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.867+0000 7fb7a5600700 1 -- 172.21.15.28:0/4233413985 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7940042f0 con 0x7fb7a8103450 2024-04-26T01:39:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.107+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fb7a802df80 con 0x7fb7a8103450 2024-04-26T01:39:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.107+0000 7fb7a5600700 1 -- 172.21.15.28:0/4233413985 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb79409d0d0 con 0x7fb7a8103450 2024-04-26T01:39:02.105 INFO:teuthology.orchestra.run.smithi028.stdout:287762808887 2024-04-26T01:39:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7900a3cc0 msgr2=0x7fb7900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7900a3cc0 0x7fb7900a6180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb79c006010 tx=0x7fb79c00b260 comp rx=0 tx=0).stop 2024-04-26T01:39:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8103450 msgr2=0x7fb7a819b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8103450 0x7fb7a819b8f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb79400eb30 tx=0x7fb79400ee40 comp rx=0 tx=0).stop 2024-04-26T01:39:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 shutdown_connections 2024-04-26T01:39:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fb7900a3cc0 0x7fb7900a6180 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7a8108a50 0x7fb7a819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a8102a90 0x7fb7a819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 --2- 172.21.15.28:0/4233413985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7a8103450 0x7fb7a819b8f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.110+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 >> 172.21.15.28:0/4233413985 conn(0x7fb7a8079ba0 msgr2=0x7fb7a810df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.111+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 shutdown_connections 2024-04-26T01:39:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.111+0000 7fb7af600700 1 -- 172.21.15.28:0/4233413985 wait complete. 2024-04-26T01:39:02.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:01 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3745247516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:39:02.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:01 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3745247516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:39:02.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:01 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3745247516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:39:02.649 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182485 got 107374182487 for osd.3 2024-04-26T01:39:02.650 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:02.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 -- 172.21.15.28:0/2375683482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738406b110 msgr2=0x7f738406b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 --2- 172.21.15.28:0/2375683482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738406b110 0x7f738406b570 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7370009990 tx=0x7f7370009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 -- 172.21.15.28:0/2375683482 shutdown_connections 2024-04-26T01:39:02.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 --2- 172.21.15.28:0/2375683482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f738406bab0 0x7f738410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 --2- 172.21.15.28:0/2375683482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7384108a50 0x7f7384108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 --2- 172.21.15.28:0/2375683482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738406b110 0x7f738406b570 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 -- 172.21.15.28:0/2375683482 >> 172.21.15.28:0/2375683482 conn(0x7f73840788f0 msgr2=0x7f7384078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:02.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 -- 172.21.15.28:0/2375683482 shutdown_connections 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.941+0000 7f7388a00700 1 -- 172.21.15.28:0/2375683482 wait complete. 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.943+0000 7f7388a00700 1 Processor -- start 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.943+0000 7f7388a00700 1 -- start start 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.943+0000 7f7388a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f738406b110 0x7f7384101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.943+0000 7f7388a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738406bab0 0x7f7384102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.943+0000 7f7388a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7384108a50 0x7f7384102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.943+0000 7f7388a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7384112c30 con 0x7f738406bab0 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7388a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7384112ab0 con 0x7f7384108a50 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7388a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7384112db0 con 0x7f738406b110 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7382800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7384108a50 0x7f7384102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:02.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7382800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7384108a50 0x7f7384102870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38834/0 (socket says 172.21.15.28:38834) 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7382800700 1 -- 172.21.15.28:0/3303624160 learned_addr learned my addr 172.21.15.28:0/3303624160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7381e00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f738406b110 0x7f7384101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7381400700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738406bab0 0x7f7384102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7382800700 1 -- 172.21.15.28:0/3303624160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f738406b110 msgr2=0x7f7384101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7382800700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f738406b110 0x7f7384101df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7382800700 1 -- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738406bab0 msgr2=0x7f7384102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.944+0000 7f7382800700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738406bab0 0x7f7384102330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.945+0000 7f7382800700 1 -- 172.21.15.28:0/3303624160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73841a76d0 con 0x7f7384108a50 2024-04-26T01:39:02.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.945+0000 7f7382800700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7384108a50 0x7f7384102870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f737c00ef30 tx=0x7f737c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.945+0000 7f737aa00700 1 -- 172.21.15.28:0/3303624160 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c0092e0 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.945+0000 7f737aa00700 1 -- 172.21.15.28:0/3303624160 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f737c004bb0 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.945+0000 7f737aa00700 1 -- 172.21.15.28:0/3303624160 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c004020 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.945+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73841a7960 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.946+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73841a7e50 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.947+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7384104190 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.949+0000 7f737aa00700 1 -- 172.21.15.28:0/3303624160 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f737c01b030 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.950+0000 7f737aa00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f73680a3cc0 0x7f73680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.950+0000 7f737aa00700 1 -- 172.21.15.28:0/3303624160 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f737c0d07e0 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.951+0000 7f7381e00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f73680a3cc0 0x7f73680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.951+0000 7f7381e00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f73680a3cc0 0x7f73680a6180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7374009be0 tx=0x7f7374009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:01.953+0000 7f737aa00700 1 -- 172.21.15.28:0/3303624160 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f737c022020 con 0x7f7384108a50 2024-04-26T01:39:02.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.190+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f738402df80 con 0x7f7384108a50 2024-04-26T01:39:02.708 INFO:teuthology.orchestra.run.smithi028.stdout:77309411425 2024-04-26T01:39:02.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.190+0000 7f737aa00700 1 -- 172.21.15.28:0/3303624160 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f737c010040 con 0x7f7384108a50 2024-04-26T01:39:02.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f73680a3cc0 msgr2=0x7f73680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f73680a3cc0 0x7f73680a6180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7374009be0 tx=0x7f7374009260 comp rx=0 tx=0).stop 2024-04-26T01:39:02.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7384108a50 msgr2=0x7f7384102870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:02.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7384108a50 0x7f7384102870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f737c00ef30 tx=0x7f737c00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 shutdown_connections 2024-04-26T01:39:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f73680a3cc0 0x7f73680a6180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7384108a50 0x7f7384102870 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f738406b110 0x7f7384101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 --2- 172.21.15.28:0/3303624160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738406bab0 0x7f7384102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 >> 172.21.15.28:0/3303624160 conn(0x7f73840788f0 msgr2=0x7f73840ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 shutdown_connections 2024-04-26T01:39:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:02.193+0000 7f7388a00700 1 -- 172.21.15.28:0/3303624160 wait complete. 2024-04-26T01:39:03.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:02 smithi028 ceph-mon[28247]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:03.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:02 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/4233413985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:39:03.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:02 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3303624160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:39:03.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:02 smithi169 ceph-mon[32851]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:03.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:02 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/4233413985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:39:03.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:02 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3303624160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:39:03.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:02 smithi053 ceph-mon[32593]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:03.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:02 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/4233413985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:39:03.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:02 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3303624160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:39:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 -- 172.21.15.28:0/3376370989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8104eb0 msgr2=0x7fdeb8105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 --2- 172.21.15.28:0/3376370989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8104eb0 0x7fdeb8105290 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdea800b0d0 tx=0x7fdea800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 -- 172.21.15.28:0/3376370989 shutdown_connections 2024-04-26T01:39:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 --2- 172.21.15.28:0/3376370989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb8109f90 0x7fdeb8111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 --2- 172.21.15.28:0/3376370989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8104eb0 0x7fdeb8105290 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 --2- 172.21.15.28:0/3376370989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb8105860 0x7fdeb81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 -- 172.21.15.28:0/3376370989 >> 172.21.15.28:0/3376370989 conn(0x7fdeb8100740 msgr2=0x7fdeb8102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 -- 172.21.15.28:0/3376370989 shutdown_connections 2024-04-26T01:39:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.240+0000 7fdebf200700 1 -- 172.21.15.28:0/3376370989 wait complete. 2024-04-26T01:39:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.241+0000 7fdebf200700 1 Processor -- start 2024-04-26T01:39:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.241+0000 7fdebf200700 1 -- start start 2024-04-26T01:39:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdebf200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb8104eb0 0x7fdeb819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdebf200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8105860 0x7fdeb819b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdebf200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb8109f90 0x7fdeb819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdebf200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb8114e70 con 0x7fdeb8104eb0 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdebf200700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb8114cf0 con 0x7fdeb8109f90 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdebf200700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb8114ff0 con 0x7fdeb8105860 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdeb7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb8104eb0 0x7fdeb819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdeb7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb8104eb0 0x7fdeb819b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41974/0 (socket says 172.21.15.28:41974) 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdeb7e00700 1 -- 172.21.15.28:0/1463831036 learned_addr learned my addr 172.21.15.28:0/1463831036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdeaf400700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8105860 0x7fdeb819b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.242+0000 7fdebd200700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb8109f90 0x7fdeb819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb7e00700 1 -- 172.21.15.28:0/1463831036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8105860 msgr2=0x7fdeb819b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb7e00700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8105860 0x7fdeb819b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb7e00700 1 -- 172.21.15.28:0/1463831036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb8109f90 msgr2=0x7fdeb819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb7e00700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb8109f90 0x7fdeb819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb7e00700 1 -- 172.21.15.28:0/1463831036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb81a0390 con 0x7fdeb8104eb0 2024-04-26T01:39:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeaf400700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8105860 0x7fdeb819b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb7e00700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb8104eb0 0x7fdeb819b380 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdea8007a30 tx=0x7fdea8007a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb5600700 1 -- 172.21.15.28:0/1463831036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea801f070 con 0x7fdeb8104eb0 2024-04-26T01:39:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb5600700 1 -- 172.21.15.28:0/1463831036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea8004c60 con 0x7fdeb8104eb0 2024-04-26T01:39:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb81a0620 con 0x7fdeb8104eb0 2024-04-26T01:39:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.243+0000 7fdeb5600700 1 -- 172.21.15.28:0/1463831036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea801b690 con 0x7fdeb8104eb0 2024-04-26T01:39:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.244+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb81a7f00 con 0x7fdeb8104eb0 2024-04-26T01:39:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.245+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb8106390 con 0x7fdeb8104eb0 2024-04-26T01:39:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.247+0000 7fdeb5600700 1 -- 172.21.15.28:0/1463831036 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdea8022070 con 0x7fdeb8104eb0 2024-04-26T01:39:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.248+0000 7fdeb5600700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fde9c0a3c40 0x7fde9c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.248+0000 7fdeb5600700 1 -- 172.21.15.28:0/1463831036 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fdea800f020 con 0x7fdeb8104eb0 2024-04-26T01:39:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.248+0000 7fdeaf400700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fde9c0a3c40 0x7fde9c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.249+0000 7fdeaf400700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fde9c0a3c40 0x7fde9c0a6100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009680 tx=0x7fdea0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.253+0000 7fdeb5600700 1 -- 172.21.15.28:0/1463831036 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdea8029020 con 0x7fdeb8104eb0 2024-04-26T01:39:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.495+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fdeb819c6e0 con 0x7fdeb8104eb0 2024-04-26T01:39:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.496+0000 7fdeb5600700 1 -- 172.21.15.28:0/1463831036 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdea809de60 con 0x7fdeb8104eb0 2024-04-26T01:39:04.493 INFO:teuthology.orchestra.run.smithi028.stdout:219043332163 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fde9c0a3c40 msgr2=0x7fde9c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fde9c0a3c40 0x7fde9c0a6100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009680 tx=0x7fdea0009260 comp rx=0 tx=0).stop 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb8104eb0 msgr2=0x7fdeb819b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb8104eb0 0x7fdeb819b380 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdea8007a30 tx=0x7fdea8007a60 comp rx=0 tx=0).stop 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 shutdown_connections 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fde9c0a3c40 0x7fde9c0a6100 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb8109f90 0x7fdeb819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb8104eb0 0x7fdeb819b380 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 --2- 172.21.15.28:0/1463831036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdeb8105860 0x7fdeb819b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 >> 172.21.15.28:0/1463831036 conn(0x7fdeb8100740 msgr2=0x7fdeb8109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 shutdown_connections 2024-04-26T01:39:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:04.500+0000 7fdebf200700 1 -- 172.21.15.28:0/1463831036 wait complete. 2024-04-26T01:39:05.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.162+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3977065458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34104ec0 msgr2=0x7f7c341052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:05.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.162+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3977065458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34104ec0 0x7f7c341052a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7c240099c0 tx=0x7f7c24009cd0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.163+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3977065458 shutdown_connections 2024-04-26T01:39:05.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.163+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3977065458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c34109fa0 0x7f7c34111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.163+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3977065458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34104ec0 0x7f7c341052a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.163+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3977065458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c34105870 0x7f7c341098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.163+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3977065458 >> 172.21.15.28:0/3977065458 conn(0x7f7c34100750 msgr2=0x7f7c34102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:05.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.163+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3977065458 shutdown_connections 2024-04-26T01:39:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.164+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3977065458 wait complete. 2024-04-26T01:39:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c3ae00700 1 Processor -- start 2024-04-26T01:39:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c3ae00700 1 -- start start 2024-04-26T01:39:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c3ae00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c34104ec0 0x7f7c34076f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c3ae00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c34105870 0x7f7c34077490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c3ae00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34109fa0 0x7f7c340779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c3ae00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c34114e30 con 0x7f7c34109fa0 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c3ae00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7c34114cb0 con 0x7f7c34104ec0 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c3ae00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7c34114fb0 con 0x7f7c34105870 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c38e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34109fa0 0x7f7c340779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c33e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c34104ec0 0x7f7c34076f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c33e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c34104ec0 0x7f7c34076f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38878/0 (socket says 172.21.15.28:38878) 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c33e00700 1 -- 172.21.15.28:0/3780443311 learned_addr learned my addr 172.21.15.28:0/3780443311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.165+0000 7f7c33400700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c34105870 0x7f7c34077490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.166+0000 7f7c33e00700 1 -- 172.21.15.28:0/3780443311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c34105870 msgr2=0x7f7c34077490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.166+0000 7f7c33e00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c34105870 0x7f7c34077490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.166+0000 7f7c33e00700 1 -- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34109fa0 msgr2=0x7f7c340779d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.166+0000 7f7c33e00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34109fa0 0x7f7c340779d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.166+0000 7f7c33e00700 1 -- 172.21.15.28:0/3780443311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c341ab9b0 con 0x7f7c34104ec0 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.166+0000 7f7c38e00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34109fa0 0x7f7c340779d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.166+0000 7f7c33e00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c34104ec0 0x7f7c34076f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7c24006010 tx=0x7f7c24005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.167+0000 7f7c30c00700 1 -- 172.21.15.28:0/3780443311 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24026070 con 0x7f7c34104ec0 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.167+0000 7f7c30c00700 1 -- 172.21.15.28:0/3780443311 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c24005680 con 0x7f7c34104ec0 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.167+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c341abc40 con 0x7f7c34104ec0 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.167+0000 7f7c30c00700 1 -- 172.21.15.28:0/3780443311 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2400fb20 con 0x7f7c34104ec0 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.167+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c341ac090 con 0x7f7c34104ec0 2024-04-26T01:39:05.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.168+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c341063a0 con 0x7f7c34104ec0 2024-04-26T01:39:05.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.170+0000 7f7c30c00700 1 -- 172.21.15.28:0/3780443311 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7c24029070 con 0x7f7c34104ec0 2024-04-26T01:39:05.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.172+0000 7f7c30c00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7c180a3c70 0x7f7c180a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:05.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.172+0000 7f7c30c00700 1 -- 172.21.15.28:0/3780443311 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f7c240d1230 con 0x7f7c34104ec0 2024-04-26T01:39:05.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.172+0000 7f7c33400700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7c180a3c70 0x7f7c180a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:05.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.173+0000 7f7c33400700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7c180a3c70 0x7f7c180a6130 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7c340786c0 tx=0x7f7c20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:05.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.174+0000 7f7c30c00700 1 -- 172.21.15.28:0/3780443311 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c24030020 con 0x7f7c34104ec0 2024-04-26T01:39:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:04 smithi028 ceph-mon[28247]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:05.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:04 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1463831036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:39:05.260 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757323 got 163208757327 for osd.5 2024-04-26T01:39:05.260 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:05.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:04 smithi169 ceph-mon[32851]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:05.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:04 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1463831036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:39:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.415+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f7c3402df80 con 0x7f7c34104ec0 2024-04-26T01:39:05.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.416+0000 7f7c30c00700 1 -- 172.21.15.28:0/3780443311 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7c24013070 con 0x7f7c34104ec0 2024-04-26T01:39:05.412 INFO:teuthology.orchestra.run.smithi028.stdout:197568495688 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7c180a3c70 msgr2=0x7f7c180a6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7c180a3c70 0x7f7c180a6130 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7c340786c0 tx=0x7f7c20009260 comp rx=0 tx=0).stop 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c34104ec0 msgr2=0x7f7c34076f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c34104ec0 0x7f7c34076f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7c24006010 tx=0x7f7c24005280 comp rx=0 tx=0).stop 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 shutdown_connections 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7c180a3c70 0x7f7c180a6130 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c34109fa0 0x7f7c340779d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c34104ec0 0x7f7c34076f50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 --2- 172.21.15.28:0/3780443311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c34105870 0x7f7c34077490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:05.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 >> 172.21.15.28:0/3780443311 conn(0x7f7c34100750 msgr2=0x7f7c34109130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:05.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 shutdown_connections 2024-04-26T01:39:05.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:05.420+0000 7f7c3ae00700 1 -- 172.21.15.28:0/3780443311 wait complete. 2024-04-26T01:39:05.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:04 smithi053 ceph-mon[32593]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:39:05.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:04 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1463831036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:39:05.892 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808882 got 287762808887 for osd.9 2024-04-26T01:39:05.892 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:06.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:06 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3780443311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:39:06.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:06 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3780443311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:39:06.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:06 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3780443311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:39:07.086 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574943 got 55834574949 for osd.1 2024-04-26T01:39:07.087 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:07.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:07 smithi028 ceph-mon[28247]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:39:07.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:07 smithi169 ceph-mon[32851]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:39:07.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:07 smithi053 ceph-mon[32593]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:39:07.754 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411421 got 77309411425 for osd.2 2024-04-26T01:39:07.755 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:08.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:08 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3707118738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:39:08.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:08 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3707118738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:39:08.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:08 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3707118738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:39:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.700+0000 7f1cb4e00700 1 -- 172.21.15.28:0/1060222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb0109c80 msgr2=0x7f1cb010dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.700+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/1060222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb0109c80 0x7f1cb010dcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c009920 tx=0x7f1c9c009c30 comp rx=0 tx=0).stop 2024-04-26T01:39:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.701+0000 7f1cb4e00700 1 -- 172.21.15.28:0/1060222 shutdown_connections 2024-04-26T01:39:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.701+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/1060222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb010e3b0 0x7f1cb0115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.701+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/1060222 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1cb01092d0 0x7f1cb01096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.701+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/1060222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb0109c80 0x7f1cb010dcd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.701+0000 7f1cb4e00700 1 -- 172.21.15.28:0/1060222 >> 172.21.15.28:0/1060222 conn(0x7f1cb0076210 msgr2=0x7f1cb0107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:08.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.701+0000 7f1cb4e00700 1 -- 172.21.15.28:0/1060222 shutdown_connections 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.701+0000 7f1cb4e00700 1 -- 172.21.15.28:0/1060222 wait complete. 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.702+0000 7f1cb4e00700 1 Processor -- start 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.702+0000 7f1cb4e00700 1 -- start start 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.702+0000 7f1cb4e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb01092d0 0x7f1cb01a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.702+0000 7f1cb4e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb0109c80 0x7f1cb01a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.702+0000 7f1cb4e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1cb010e3b0 0x7f1cb019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.702+0000 7f1cb4e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb0119290 con 0x7f1cb01092d0 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.702+0000 7f1cb4e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb0119110 con 0x7f1cb0109c80 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.702+0000 7f1cb4e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb0119410 con 0x7f1cb010e3b0 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cade00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb01092d0 0x7f1cb01a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cade00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb01092d0 0x7f1cb01a5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42048/0 (socket says 172.21.15.28:42048) 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cade00700 1 -- 172.21.15.28:0/3929583784 learned_addr learned my addr 172.21.15.28:0/3929583784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cae800700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1cb010e3b0 0x7f1cb019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cad400700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb0109c80 0x7f1cb01a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cade00700 1 -- 172.21.15.28:0/3929583784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1cb010e3b0 msgr2=0x7f1cb019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cade00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1cb010e3b0 0x7f1cb019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cade00700 1 -- 172.21.15.28:0/3929583784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb0109c80 msgr2=0x7f1cb01a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cade00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb0109c80 0x7f1cb01a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.703+0000 7f1cade00700 1 -- 172.21.15.28:0/3929583784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb019ff70 con 0x7f1cb01092d0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.704+0000 7f1cade00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb01092d0 0x7f1cb01a5570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000e9c0 tx=0x7f1ca000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.704+0000 7f1ca6a00700 1 -- 172.21.15.28:0/3929583784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca000cbd0 con 0x7f1cb01092d0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.704+0000 7f1ca6a00700 1 -- 172.21.15.28:0/3929583784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ca000ee60 con 0x7f1cb01092d0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.704+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb01a0260 con 0x7f1cb01092d0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.704+0000 7f1ca6a00700 1 -- 172.21.15.28:0/3929583784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca00055c0 con 0x7f1cb01092d0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.704+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb01ac4f0 con 0x7f1cb01092d0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.705+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb010a7b0 con 0x7f1cb01092d0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.707+0000 7f1ca6a00700 1 -- 172.21.15.28:0/3929583784 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1ca0004750 con 0x7f1cb01092d0 2024-04-26T01:39:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.708+0000 7f1ca6a00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1c940a3cc0 0x7f1c940a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:08.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.709+0000 7f1ca6a00700 1 -- 172.21.15.28:0/3929583784 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f1ca00d0630 con 0x7f1cb01092d0 2024-04-26T01:39:08.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.709+0000 7f1cad400700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1c940a3cc0 0x7f1c940a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:08.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.710+0000 7f1cad400700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1c940a3cc0 0x7f1c940a6180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c009810 tx=0x7f1c9c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:08.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.713+0000 7f1ca6a00700 1 -- 172.21.15.28:0/3929583784 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ca009cfc0 con 0x7f1cb01092d0 2024-04-26T01:39:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.952+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1cb01a0b70 con 0x7f1cb01092d0 2024-04-26T01:39:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.953+0000 7f1ca6a00700 1 -- 172.21.15.28:0/3929583784 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1ca009c8c0 con 0x7f1cb01092d0 2024-04-26T01:39:08.950 INFO:teuthology.orchestra.run.smithi028.stdout:34359738476 2024-04-26T01:39:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1c940a3cc0 msgr2=0x7f1c940a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1c940a3cc0 0x7f1c940a6180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c009810 tx=0x7f1c9c019040 comp rx=0 tx=0).stop 2024-04-26T01:39:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb01092d0 msgr2=0x7f1cb01a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb01092d0 0x7f1cb01a5570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000e9c0 tx=0x7f1ca000ecd0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 shutdown_connections 2024-04-26T01:39:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f1c940a3cc0 0x7f1c940a6180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1cb010e3b0 0x7f1cb019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb01092d0 0x7f1cb01a5570 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 --2- 172.21.15.28:0/3929583784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb0109c80 0x7f1cb01a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 >> 172.21.15.28:0/3929583784 conn(0x7f1cb0076210 msgr2=0x7f1cb0077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 shutdown_connections 2024-04-26T01:39:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:08.957+0000 7f1cb4e00700 1 -- 172.21.15.28:0/3929583784 wait complete. 2024-04-26T01:39:09.141 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332158 got 219043332163 for osd.7 2024-04-26T01:39:09.142 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.432+0000 7f66cbe00700 1 -- 172.21.15.28:0/3202123077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4108a30 msgr2=0x7f66c4108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.432+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3202123077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4108a30 0x7f66c4108e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f66b80098f0 tx=0x7f66b8009c00 comp rx=0 tx=0).stop 2024-04-26T01:39:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.433+0000 7f66cbe00700 1 -- 172.21.15.28:0/3202123077 shutdown_connections 2024-04-26T01:39:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.433+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3202123077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66c406b0f0 0x7f66c410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.433+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3202123077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4108a30 0x7f66c4108e10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.433+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3202123077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c406a750 0x7f66c406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.433+0000 7f66cbe00700 1 -- 172.21.15.28:0/3202123077 >> 172.21.15.28:0/3202123077 conn(0x7f66c40788f0 msgr2=0x7f66c4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.433+0000 7f66cbe00700 1 -- 172.21.15.28:0/3202123077 shutdown_connections 2024-04-26T01:39:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.433+0000 7f66cbe00700 1 -- 172.21.15.28:0/3202123077 wait complete. 2024-04-26T01:39:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.434+0000 7f66cbe00700 1 Processor -- start 2024-04-26T01:39:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.434+0000 7f66cbe00700 1 -- start start 2024-04-26T01:39:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.434+0000 7f66cbe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c406a750 0x7f66c4199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.434+0000 7f66cbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c406b0f0 0x7f66c4199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.434+0000 7f66cbe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66c4108a30 0x7f66c419da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.434+0000 7f66cbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4112bd0 con 0x7f66c406b0f0 2024-04-26T01:39:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.434+0000 7f66cbe00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4112a50 con 0x7f66c406a750 2024-04-26T01:39:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.434+0000 7f66cbe00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4112d50 con 0x7f66c4108a30 2024-04-26T01:39:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c8a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c406b0f0 0x7f66c4199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c8a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c406b0f0 0x7f66c4199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42028/0 (socket says 172.21.15.28:42028) 2024-04-26T01:39:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c8a00700 1 -- 172.21.15.28:0/3707118738 learned_addr learned my addr 172.21.15.28:0/3707118738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:09.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c9e00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66c4108a30 0x7f66c419da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c9400700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c406a750 0x7f66c4199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c8a00700 1 -- 172.21.15.28:0/3707118738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66c4108a30 msgr2=0x7f66c419da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c8a00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66c4108a30 0x7f66c419da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c8a00700 1 -- 172.21.15.28:0/3707118738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c406a750 msgr2=0x7f66c4199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c8a00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c406a750 0x7f66c4199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.435+0000 7f66c8a00700 1 -- 172.21.15.28:0/3707118738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c419e150 con 0x7f66c406b0f0 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.436+0000 7f66c8a00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c406b0f0 0x7f66c4199680 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f66b000eb30 tx=0x7f66b000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.436+0000 7f66c2000700 1 -- 172.21.15.28:0/3707118738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b000cbd0 con 0x7f66c406b0f0 2024-04-26T01:39:09.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.436+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c419e440 con 0x7f66c406b0f0 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.436+0000 7f66c2000700 1 -- 172.21.15.28:0/3707118738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b00044d0 con 0x7f66c406b0f0 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.436+0000 7f66c2000700 1 -- 172.21.15.28:0/3707118738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0003e30 con 0x7f66c406b0f0 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.436+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c41a5cc0 con 0x7f66c406b0f0 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.437+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c4069730 con 0x7f66c406b0f0 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.440+0000 7f66c2000700 1 -- 172.21.15.28:0/3707118738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f66b0004000 con 0x7f66c406b0f0 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.441+0000 7f66c2000700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f66ac0a3cc0 0x7f66ac0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.441+0000 7f66c2000700 1 -- 172.21.15.28:0/3707118738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f66b00d0880 con 0x7f66c406b0f0 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.441+0000 7f66c9400700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f66ac0a3cc0 0x7f66ac0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.442+0000 7f66c9400700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f66ac0a3cc0 0x7f66ac0a6180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009fd0 tx=0x7f66b8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.447+0000 7f66c2000700 1 -- 172.21.15.28:0/3707118738 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f66b0052ff0 con 0x7f66c406b0f0 2024-04-26T01:39:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.678+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f66c402df80 con 0x7f66c406b0f0 2024-04-26T01:39:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.678+0000 7f66c2000700 1 -- 172.21.15.28:0/3707118738 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f66b00042f0 con 0x7f66c406b0f0 2024-04-26T01:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout:253403070525 2024-04-26T01:39:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f66ac0a3cc0 msgr2=0x7f66ac0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f66ac0a3cc0 0x7f66ac0a6180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009fd0 tx=0x7f66b8019040 comp rx=0 tx=0).stop 2024-04-26T01:39:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c406b0f0 msgr2=0x7f66c4199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c406b0f0 0x7f66c4199680 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f66b000eb30 tx=0x7f66b000ee40 comp rx=0 tx=0).stop 2024-04-26T01:39:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 shutdown_connections 2024-04-26T01:39:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f66ac0a3cc0 0x7f66ac0a6180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66c4108a30 0x7f66c419da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c406a750 0x7f66c4199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 --2- 172.21.15.28:0/3707118738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c406b0f0 0x7f66c4199680 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 >> 172.21.15.28:0/3707118738 conn(0x7f66c40788f0 msgr2=0x7f66c40ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:09.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 shutdown_connections 2024-04-26T01:39:09.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:07.681+0000 7f66cbe00700 1 -- 172.21.15.28:0/3707118738 wait complete. 2024-04-26T01:39:09.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:09 smithi028 ceph-mon[28247]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:39:09.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:09 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3929583784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:39:09.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:09 smithi169 ceph-mon[32851]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:39:09.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:09 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3929583784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:39:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:09 smithi053 ceph-mon[32593]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:39:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:09 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3929583784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:39:10.012 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495681 got 197568495688 for osd.6 2024-04-26T01:39:10.012 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 -- 172.21.15.28:0/3556390185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f7c102a90 msgr2=0x7f9f7c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3556390185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f7c102a90 0x7f9f7c102f10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009990 tx=0x7f9f64009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 -- 172.21.15.28:0/3556390185 shutdown_connections 2024-04-26T01:39:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3556390185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f7c103450 0x7f9f7c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3556390185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f7c108a50 0x7f9f7c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3556390185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f7c102a90 0x7f9f7c102f10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 -- 172.21.15.28:0/3556390185 >> 172.21.15.28:0/3556390185 conn(0x7f9f7c079ba0 msgr2=0x7f9f7c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:10.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 -- 172.21.15.28:0/3556390185 shutdown_connections 2024-04-26T01:39:10.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.519+0000 7f9f80e00700 1 -- 172.21.15.28:0/3556390185 wait complete. 2024-04-26T01:39:10.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.520+0000 7f9f80e00700 1 Processor -- start 2024-04-26T01:39:10.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.520+0000 7f9f80e00700 1 -- start start 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f80e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f7c102a90 0x7f9f7c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f80e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f7c103450 0x7f9f7c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f80e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f7c108a50 0x7f9f7c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f80e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c112c50 con 0x7f9f7c103450 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f80e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c112ad0 con 0x7f9f7c102a90 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f80e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c112dd0 con 0x7f9f7c108a50 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f79e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f7c102a90 0x7f9f7c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f71400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f7c103450 0x7f9f7c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f79e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f7c102a90 0x7f9f7c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53612/0 (socket says 172.21.15.28:53612) 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f79e00700 1 -- 172.21.15.28:0/3266140004 learned_addr learned my addr 172.21.15.28:0/3266140004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f7a800700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f7c108a50 0x7f9f7c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:10.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f79e00700 1 -- 172.21.15.28:0/3266140004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f7c108a50 msgr2=0x7f9f7c0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f79e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f7c108a50 0x7f9f7c0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f79e00700 1 -- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f7c103450 msgr2=0x7f9f7c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f79e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f7c103450 0x7f9f7c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.521+0000 7f9f79e00700 1 -- 172.21.15.28:0/3266140004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7c1a7730 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.522+0000 7f9f79e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f7c102a90 0x7f9f7c074d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00eb30 tx=0x7f9f6c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.522+0000 7f9f73400700 1 -- 172.21.15.28:0/3266140004 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c00cbd0 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.522+0000 7f9f73400700 1 -- 172.21.15.28:0/3266140004 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f6c0044d0 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.522+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7c1a7a20 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.522+0000 7f9f73400700 1 -- 172.21.15.28:0/3266140004 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c003e30 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.522+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f7c1a7fb0 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.523+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f7c069730 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.526+0000 7f9f73400700 1 -- 172.21.15.28:0/3266140004 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9f6c004000 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.527+0000 7f9f73400700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9f5c0a3cc0 0x7f9f5c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.527+0000 7f9f73400700 1 -- 172.21.15.28:0/3266140004 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f9f6c0d0890 con 0x7f9f7c102a90 2024-04-26T01:39:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.527+0000 7f9f71400700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9f5c0a3cc0 0x7f9f5c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:10.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.528+0000 7f9f71400700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9f5c0a3cc0 0x7f9f5c0a6180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c07a3c0 tx=0x7f9f6400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:10.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.529+0000 7f9f73400700 1 -- 172.21.15.28:0/3266140004 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f6c053ff0 con 0x7f9f7c102a90 2024-04-26T01:39:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.763+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9f7c02df80 con 0x7f9f7c102a90 2024-04-26T01:39:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.764+0000 7f9f73400700 1 -- 172.21.15.28:0/3266140004 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9f6c0042f0 con 0x7f9f7c102a90 2024-04-26T01:39:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:141733920854 2024-04-26T01:39:10.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9f5c0a3cc0 msgr2=0x7f9f5c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9f5c0a3cc0 0x7f9f5c0a6180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c07a3c0 tx=0x7f9f6400b260 comp rx=0 tx=0).stop 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f7c102a90 msgr2=0x7f9f7c074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f7c102a90 0x7f9f7c074d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00eb30 tx=0x7f9f6c00ee40 comp rx=0 tx=0).stop 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 shutdown_connections 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f9f5c0a3cc0 0x7f9f5c0a6180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f7c108a50 0x7f9f7c0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f7c102a90 0x7f9f7c074d40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 --2- 172.21.15.28:0/3266140004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f7c103450 0x7f9f7c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.767+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 >> 172.21.15.28:0/3266140004 conn(0x7f9f7c079ba0 msgr2=0x7f9f7c101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.768+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 shutdown_connections 2024-04-26T01:39:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:10.768+0000 7f9f80e00700 1 -- 172.21.15.28:0/3266140004 wait complete. 2024-04-26T01:39:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:11.113 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738470 got 34359738476 for osd.0 2024-04-26T01:39:11.113 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:11.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:11 smithi028 ceph-mon[28247]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:39:11.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:11 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3266140004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:39:11.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:11 smithi169 ceph-mon[32851]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:39:11.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:11 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3266140004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:39:11.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:11 smithi053 ceph-mon[32593]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:39:11.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:11 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3266140004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 -- 172.21.15.28:0/2586970562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06b980 msgr2=0x7ff0ec10f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 --2- 172.21.15.28:0/2586970562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06b980 0x7ff0ec10f8f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009f90 tx=0x7ff0e400b330 comp rx=0 tx=0).stop 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 -- 172.21.15.28:0/2586970562 shutdown_connections 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 --2- 172.21.15.28:0/2586970562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06b980 0x7ff0ec10f8f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 --2- 172.21.15.28:0/2586970562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0ec108a20 0x7ff0ec108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 --2- 172.21.15.28:0/2586970562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ec06afe0 0x7ff0ec06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 -- 172.21.15.28:0/2586970562 >> 172.21.15.28:0/2586970562 conn(0x7ff0ec0788b0 msgr2=0x7ff0ec078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 -- 172.21.15.28:0/2586970562 shutdown_connections 2024-04-26T01:39:11.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.821+0000 7ff0f2400700 1 -- 172.21.15.28:0/2586970562 wait complete. 2024-04-26T01:39:11.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.822+0000 7ff0f2400700 1 Processor -- start 2024-04-26T01:39:11.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0f2400700 1 -- start start 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0f2400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06afe0 0x7ff0ec19b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0f2400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ec06b980 0x7ff0ec19b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0f2400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0ec108a20 0x7ff0ec19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0f2400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ec112c10 con 0x7ff0ec06afe0 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0f2400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ec112a90 con 0x7ff0ec06b980 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0f2400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ec112d90 con 0x7ff0ec108a20 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0ebe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0ec108a20 0x7ff0ec19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0ebe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0ec108a20 0x7ff0ec19fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:41470/0 (socket says 172.21.15.28:41470) 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0ebe00700 1 -- 172.21.15.28:0/3741475328 learned_addr learned my addr 172.21.15.28:0/3741475328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0eb400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06afe0 0x7ff0ec19b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.823+0000 7ff0eaa00700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ec06b980 0x7ff0ec19b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0ebe00700 1 -- 172.21.15.28:0/3741475328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ec06b980 msgr2=0x7ff0ec19b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0ebe00700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ec06b980 0x7ff0ec19b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0ebe00700 1 -- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06afe0 msgr2=0x7ff0ec19b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0ebe00700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06afe0 0x7ff0ec19b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0ebe00700 1 -- 172.21.15.28:0/3741475328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ec1a0330 con 0x7ff0ec108a20 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0eaa00700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ec06b980 0x7ff0ec19b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0eb400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06afe0 0x7ff0ec19b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:39:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0ebe00700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0ec108a20 0x7ff0ec19fbf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009c10 tx=0x7ff0e400cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.824+0000 7ff0e3e00700 1 -- 172.21.15.28:0/3741475328 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4021070 con 0x7ff0ec108a20 2024-04-26T01:39:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.825+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ec1a05c0 con 0x7ff0ec108a20 2024-04-26T01:39:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.825+0000 7ff0e3e00700 1 -- 172.21.15.28:0/3741475328 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e4007ac0 con 0x7ff0ec108a20 2024-04-26T01:39:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.825+0000 7ff0e3e00700 1 -- 172.21.15.28:0/3741475328 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e401de60 con 0x7ff0ec108a20 2024-04-26T01:39:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.825+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ec1a7ea0 con 0x7ff0ec108a20 2024-04-26T01:39:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.826+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0ec069730 con 0x7ff0ec108a20 2024-04-26T01:39:11.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.829+0000 7ff0e3e00700 1 -- 172.21.15.28:0/3741475328 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff0e4024070 con 0x7ff0ec108a20 2024-04-26T01:39:11.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.831+0000 7ff0e3e00700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0d00a3c40 0x7ff0d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:11.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.831+0000 7ff0e3e00700 1 -- 172.21.15.28:0/3741475328 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7ff0e400f020 con 0x7ff0ec108a20 2024-04-26T01:39:11.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.831+0000 7ff0eb400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0d00a3c40 0x7ff0d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:11.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.832+0000 7ff0eb400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0d00a3c40 0x7ff0d00a6100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc005fd0 tx=0x7ff0dc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:11.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:11.839+0000 7ff0e3e00700 1 -- 172.21.15.28:0/3741475328 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0e402b020 con 0x7ff0ec108a20 2024-04-26T01:39:12.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.078+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7ff0ec02df80 con 0x7ff0ec108a20 2024-04-26T01:39:12.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.079+0000 7ff0e3e00700 1 -- 172.21.15.28:0/3741475328 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff0e409e250 con 0x7ff0ec108a20 2024-04-26T01:39:12.076 INFO:teuthology.orchestra.run.smithi028.stdout:343597383726 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.081+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0d00a3c40 msgr2=0x7ff0d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.081+0000 7ff0f2400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0d00a3c40 0x7ff0d00a6100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc005fd0 tx=0x7ff0dc009310 comp rx=0 tx=0).stop 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.081+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0ec108a20 msgr2=0x7ff0ec19fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.081+0000 7ff0f2400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0ec108a20 0x7ff0ec19fbf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009c10 tx=0x7ff0e400cca0 comp rx=0 tx=0).stop 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.082+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 shutdown_connections 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.082+0000 7ff0f2400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff0d00a3c40 0x7ff0d00a6100 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.082+0000 7ff0f2400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0ec108a20 0x7ff0ec19fbf0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.082+0000 7ff0f2400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0ec06afe0 0x7ff0ec19b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.082+0000 7ff0f2400700 1 --2- 172.21.15.28:0/3741475328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ec06b980 0x7ff0ec19b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.082+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 >> 172.21.15.28:0/3741475328 conn(0x7ff0ec0788b0 msgr2=0x7ff0ec10df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.082+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 shutdown_connections 2024-04-26T01:39:12.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:12.082+0000 7ff0f2400700 1 -- 172.21.15.28:0/3741475328 wait complete. 2024-04-26T01:39:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:13 smithi028 ceph-mon[28247]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:13 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3741475328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:39:13.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:13 smithi169 ceph-mon[32851]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:13.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:13 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3741475328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:39:13.417 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383719 got 343597383726 for osd.11 2024-04-26T01:39:13.417 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:13.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:13 smithi053 ceph-mon[32593]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:13.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:13 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3741475328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:39:13.846 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070518 got 253403070525 for osd.8 2024-04-26T01:39:13.846 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:15.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:15 smithi028 ceph-mon[28247]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:15.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:15 smithi169 ceph-mon[32851]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:15.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:15 smithi053 ceph-mon[32593]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:15.621 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920844 got 141733920854 for osd.4 2024-04-26T01:39:15.621 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:16.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 -- 172.21.15.28:0/3794210834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806b0f0 msgr2=0x7f7bc810f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:16.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 --2- 172.21.15.28:0/3794210834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806b0f0 0x7f7bc810f900 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000b0a0 tx=0x7f7bc000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 -- 172.21.15.28:0/3794210834 shutdown_connections 2024-04-26T01:39:16.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 --2- 172.21.15.28:0/3794210834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806b0f0 0x7f7bc810f900 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 --2- 172.21.15.28:0/3794210834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc8108a30 0x7f7bc8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 --2- 172.21.15.28:0/3794210834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bc806a750 0x7f7bc806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 -- 172.21.15.28:0/3794210834 >> 172.21.15.28:0/3794210834 conn(0x7f7bc80788f0 msgr2=0x7f7bc8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:16.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 -- 172.21.15.28:0/3794210834 shutdown_connections 2024-04-26T01:39:16.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.254+0000 7f7bd0400700 1 -- 172.21.15.28:0/3794210834 wait complete. 2024-04-26T01:39:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.255+0000 7f7bd0400700 1 Processor -- start 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.255+0000 7f7bd0400700 1 -- start start 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bd0400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806a750 0x7f7bc8101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bd0400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bc806b0f0 0x7f7bc8101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bd0400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc8108a30 0x7f7bc8102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bd0400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc8112c20 con 0x7f7bc806b0f0 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bd0400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc8112aa0 con 0x7f7bc8108a30 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bd0400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc8112da0 con 0x7f7bc806a750 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcda00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806a750 0x7f7bc8101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcd000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bc806b0f0 0x7f7bc8101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcda00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806a750 0x7f7bc8101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:41492/0 (socket says 172.21.15.28:41492) 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcda00700 1 -- 172.21.15.28:0/2854141002 learned_addr learned my addr 172.21.15.28:0/2854141002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:16.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bce400700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc8108a30 0x7f7bc8102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:16.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcd000700 1 -- 172.21.15.28:0/2854141002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806a750 msgr2=0x7f7bc8101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:16.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcd000700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806a750 0x7f7bc8101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcd000700 1 -- 172.21.15.28:0/2854141002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc8108a30 msgr2=0x7f7bc8102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:16.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcd000700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc8108a30 0x7f7bc8102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcd000700 1 -- 172.21.15.28:0/2854141002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc81abb20 con 0x7f7bc806b0f0 2024-04-26T01:39:16.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.256+0000 7f7bcda00700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806a750 0x7f7bc8101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:16.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.257+0000 7f7bcd000700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bc806b0f0 0x7f7bc8101f50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400e9f0 tx=0x7f7bb400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:16.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.257+0000 7f7bc6000700 1 -- 172.21.15.28:0/2854141002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb400c9c0 con 0x7f7bc806b0f0 2024-04-26T01:39:16.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.257+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc81abdb0 con 0x7f7bc806b0f0 2024-04-26T01:39:16.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.257+0000 7f7bc6000700 1 -- 172.21.15.28:0/2854141002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bb400cb60 con 0x7f7bc806b0f0 2024-04-26T01:39:16.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.257+0000 7f7bc6000700 1 -- 172.21.15.28:0/2854141002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4018680 con 0x7f7bc806b0f0 2024-04-26T01:39:16.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.258+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc81ac260 con 0x7f7bc806b0f0 2024-04-26T01:39:16.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.259+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc8104170 con 0x7f7bc806b0f0 2024-04-26T01:39:16.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.261+0000 7f7bc6000700 1 -- 172.21.15.28:0/2854141002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7bb4010430 con 0x7f7bc806b0f0 2024-04-26T01:39:16.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.263+0000 7f7bc6000700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7bb00a3c40 0x7f7bb00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.263+0000 7f7bc6000700 1 -- 172.21.15.28:0/2854141002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f7bb40d08d0 con 0x7f7bc806b0f0 2024-04-26T01:39:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.263+0000 7f7bcda00700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7bb00a3c40 0x7f7bb00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.264+0000 7f7bcda00700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7bb00a3c40 0x7f7bb00a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc011550 tx=0x7f7bbc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:16.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.270+0000 7f7bc6000700 1 -- 172.21.15.28:0/2854141002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7bb409d260 con 0x7f7bc806b0f0 2024-04-26T01:39:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.512+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f7bc802df80 con 0x7f7bc806b0f0 2024-04-26T01:39:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.513+0000 7f7bc6000700 1 -- 172.21.15.28:0/2854141002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7bb409cb60 con 0x7f7bc806b0f0 2024-04-26T01:39:16.510 INFO:teuthology.orchestra.run.smithi028.stdout:309237645364 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7bb00a3c40 msgr2=0x7f7bb00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7bb00a3c40 0x7f7bb00a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc011550 tx=0x7f7bbc009260 comp rx=0 tx=0).stop 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bc806b0f0 msgr2=0x7f7bc8101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bc806b0f0 0x7f7bc8101f50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400e9f0 tx=0x7f7bb400ed00 comp rx=0 tx=0).stop 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 shutdown_connections 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7bb00a3c40 0x7f7bb00a6100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc8108a30 0x7f7bc8102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc806a750 0x7f7bc8101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 --2- 172.21.15.28:0/2854141002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bc806b0f0 0x7f7bc8101f50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:16.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 >> 172.21.15.28:0/2854141002 conn(0x7f7bc80788f0 msgr2=0x7f7bc80ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:16.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 shutdown_connections 2024-04-26T01:39:16.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:16.516+0000 7f7bd0400700 1 -- 172.21.15.28:0/2854141002 wait complete. 2024-04-26T01:39:17.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:17 smithi028 ceph-mon[28247]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:17.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:17 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2854141002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:39:17.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645357 got 309237645364 for osd.10 2024-04-26T01:39:17.353 DEBUG:teuthology.parallel:result is None 2024-04-26T01:39:17.353 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:39:17.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:39:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:17 smithi169 ceph-mon[32851]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:17 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2854141002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:39:17.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:17 smithi053 ceph-mon[32593]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:17.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:17 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2854141002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:39:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:18.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:18.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:18 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:18.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:18.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:18 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:18.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:18.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:18 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:19 smithi028 ceph-mon[28247]: pgmap v407: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:19.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:19 smithi169 ceph-mon[32851]: pgmap v407: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:19.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:19.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:19.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:19.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:19 smithi053 ceph-mon[32593]: pgmap v407: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:19.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:19.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:19.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 -- 172.21.15.28:0/1457499522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac0076f70 msgr2=0x7feac0077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 --2- 172.21.15.28:0/1457499522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac0076f70 0x7feac0077350 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feab4009990 tx=0x7feab4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 -- 172.21.15.28:0/1457499522 shutdown_connections 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 --2- 172.21.15.28:0/1457499522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac01138e0 0x7feac0115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 --2- 172.21.15.28:0/1457499522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac0076f70 0x7feac0077350 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 --2- 172.21.15.28:0/1457499522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac0077890 0x7feac01133a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 -- 172.21.15.28:0/1457499522 >> 172.21.15.28:0/1457499522 conn(0x7feac0100750 msgr2=0x7feac0102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 -- 172.21.15.28:0/1457499522 shutdown_connections 2024-04-26T01:39:20.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.103+0000 7feac7e00700 1 -- 172.21.15.28:0/1457499522 wait complete. 2024-04-26T01:39:20.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.105+0000 7feac7e00700 1 Processor -- start 2024-04-26T01:39:20.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.105+0000 7feac7e00700 1 -- start start 2024-04-26T01:39:20.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.106+0000 7feac7e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0076f70 0x7feac019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:20.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.106+0000 7feac7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac0077890 0x7feac019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:20.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.106+0000 7feac7e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac01138e0 0x7feac01a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:20.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.106+0000 7feac7e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feac01192c0 con 0x7feac0077890 2024-04-26T01:39:20.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.106+0000 7feac7e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7feac0119140 con 0x7feac0076f70 2024-04-26T01:39:20.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.106+0000 7feac7e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7feac0119440 con 0x7feac01138e0 2024-04-26T01:39:20.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.106+0000 7feac5400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0076f70 0x7feac019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:20.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0076f70 0x7feac019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:34820/0 (socket says 172.21.15.28:34820) 2024-04-26T01:39:20.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac4a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac0077890 0x7feac019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:20.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5400700 1 -- 172.21.15.28:0/2404041423 learned_addr learned my addr 172.21.15.28:0/2404041423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:20.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5e00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac01138e0 0x7feac01a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:20.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5400700 1 -- 172.21.15.28:0/2404041423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac01138e0 msgr2=0x7feac01a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:20.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5400700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac01138e0 0x7feac01a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5400700 1 -- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac0077890 msgr2=0x7feac019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:20.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5400700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac0077890 0x7feac019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5400700 1 -- 172.21.15.28:0/2404041423 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac01a4760 con 0x7feac0076f70 2024-04-26T01:39:20.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5e00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac01138e0 0x7feac01a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:20.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.107+0000 7feac5400700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0076f70 0x7feac019f750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feab4005910 tx=0x7feab400b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:20.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.108+0000 7feabe000700 1 -- 172.21.15.28:0/2404041423 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4027070 con 0x7feac0076f70 2024-04-26T01:39:20.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.108+0000 7feabe000700 1 -- 172.21.15.28:0/2404041423 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab4022470 con 0x7feac0076f70 2024-04-26T01:39:20.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.108+0000 7feabe000700 1 -- 172.21.15.28:0/2404041423 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4017650 con 0x7feac0076f70 2024-04-26T01:39:20.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.108+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac01a49f0 con 0x7feac0076f70 2024-04-26T01:39:20.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.108+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac01ac2d0 con 0x7feac0076f70 2024-04-26T01:39:20.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.109+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feac00786b0 con 0x7feac0076f70 2024-04-26T01:39:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.111+0000 7feabe000700 1 -- 172.21.15.28:0/2404041423 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7feab402a070 con 0x7feac0076f70 2024-04-26T01:39:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.113+0000 7feabe000700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feaa80a3c40 0x7feaa80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.113+0000 7feabe000700 1 -- 172.21.15.28:0/2404041423 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7feab40d12f0 con 0x7feac0076f70 2024-04-26T01:39:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.113+0000 7feac4a00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feaa80a3c40 0x7feaa80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.114+0000 7feac4a00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feaa80a3c40 0x7feaa80a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feac0079a70 tx=0x7feaac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.115+0000 7feabe000700 1 -- 172.21.15.28:0/2404041423 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feab4013070 con 0x7feac0076f70 2024-04-26T01:39:20.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.343+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feac01a0ab0 con 0x7feaa80a3c40 2024-04-26T01:39:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.365+0000 7feabe000700 1 -- 172.21.15.28:0/2404041423 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259061 (secure 0 0 0) 0x7feac01a0ab0 con 0x7feaa80a3c40 2024-04-26T01:39:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:39:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feaa80a3c40 msgr2=0x7feaa80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feaa80a3c40 0x7feaa80a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feac0079a70 tx=0x7feaac009260 comp rx=0 tx=0).stop 2024-04-26T01:39:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0076f70 msgr2=0x7feac019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0076f70 0x7feac019f750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feab4005910 tx=0x7feab400b670 comp rx=0 tx=0).stop 2024-04-26T01:39:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 shutdown_connections 2024-04-26T01:39:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7feaa80a3c40 0x7feaa80a6100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feac01138e0 0x7feac01a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0076f70 0x7feac019f750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 --2- 172.21.15.28:0/2404041423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac0077890 0x7feac019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 >> 172.21.15.28:0/2404041423 conn(0x7feac0100750 msgr2=0x7feac0102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 shutdown_connections 2024-04-26T01:39:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:20.372+0000 7feac7e00700 1 -- 172.21.15.28:0/2404041423 wait complete. 2024-04-26T01:39:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-26T01:39:21.183 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-04-26T01:39:19.820099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660643,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":879,"num_read_kb":565,"num_write":358,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181492,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377792,"kb_used_data":12000,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564032,"statfs":{"total":1151940427776,"available":1151553568768,"internally_reserved":0,"allocated":12288000,"data_stored":7733517,"data_compressed":113448,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":374453564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004861"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777218+0000","last_change":"2024-04-26T01:36:54.288460+0000","last_active":"2024-04-26T01:38:53.777218+0000","last_peered":"2024-04-26T01:38:53.777218+0000","last_clean":"2024-04-26T01:38:53.777218+0000","last_became_active":"2024-04-26T01:36:54.288033+0000","last_became_peered":"2024-04-26T01:36:54.288033+0000","last_unstale":"2024-04-26T01:38:53.777218+0000","last_undegraded":"2024-04-26T01:38:53.777218+0000","last_fullsized":"2024-04-26T01:38:53.777218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:33.737622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.778284+0000","last_change":"2024-04-26T01:36:54.287367+0000","last_active":"2024-04-26T01:38:53.778284+0000","last_peered":"2024-04-26T01:38:53.778284+0000","last_clean":"2024-04-26T01:38:53.778284+0000","last_became_active":"2024-04-26T01:36:54.287024+0000","last_became_peered":"2024-04-26T01:36:54.287024+0000","last_unstale":"2024-04-26T01:38:53.778284+0000","last_undegraded":"2024-04-26T01:38:53.778284+0000","last_fullsized":"2024-04-26T01:38:53.778284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:31.014627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780136+0000","last_change":"2024-04-26T01:36:54.288433+0000","last_active":"2024-04-26T01:38:53.780136+0000","last_peered":"2024-04-26T01:38:53.780136+0000","last_clean":"2024-04-26T01:38:53.780136+0000","last_became_active":"2024-04-26T01:36:54.288002+0000","last_became_peered":"2024-04-26T01:36:54.288002+0000","last_unstale":"2024-04-26T01:38:53.780136+0000","last_undegraded":"2024-04-26T01:38:53.780136+0000","last_fullsized":"2024-04-26T01:38:53.780136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:37.885179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.759001+0000","last_change":"2024-04-26T01:36:54.273857+0000","last_active":"2024-04-26T01:38:53.759001+0000","last_peered":"2024-04-26T01:38:53.759001+0000","last_clean":"2024-04-26T01:38:53.759001+0000","last_became_active":"2024-04-26T01:36:54.272376+0000","last_became_peered":"2024-04-26T01:36:54.272376+0000","last_unstale":"2024-04-26T01:38:53.759001+0000","last_undegraded":"2024-04-26T01:38:53.759001+0000","last_fullsized":"2024-04-26T01:38:53.759001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:30.110044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786123+0000","last_change":"2024-04-26T01:36:54.288606+0000","last_active":"2024-04-26T01:38:53.786123+0000","last_peered":"2024-04-26T01:38:53.786123+0000","last_clean":"2024-04-26T01:38:53.786123+0000","last_became_active":"2024-04-26T01:36:54.288247+0000","last_became_peered":"2024-04-26T01:36:54.288247+0000","last_unstale":"2024-04-26T01:38:53.786123+0000","last_undegraded":"2024-04-26T01:38:53.786123+0000","last_fullsized":"2024-04-26T01:38:53.786123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:47:56.944164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777137+0000","last_change":"2024-04-26T01:36:54.277701+0000","last_active":"2024-04-26T01:38:53.777137+0000","last_peered":"2024-04-26T01:38:53.777137+0000","last_clean":"2024-04-26T01:38:53.777137+0000","last_became_active":"2024-04-26T01:36:54.277306+0000","last_became_peered":"2024-04-26T01:36:54.277306+0000","last_unstale":"2024-04-26T01:38:53.777137+0000","last_undegraded":"2024-04-26T01:38:53.777137+0000","last_fullsized":"2024-04-26T01:38:53.777137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:42:05.058525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.778546+0000","last_change":"2024-04-26T01:36:54.274090+0000","last_active":"2024-04-26T01:38:53.778546+0000","last_peered":"2024-04-26T01:38:53.778546+0000","last_clean":"2024-04-26T01:38:53.778546+0000","last_became_active":"2024-04-26T01:36:54.273700+0000","last_became_peered":"2024-04-26T01:36:54.273700+0000","last_unstale":"2024-04-26T01:38:53.778546+0000","last_undegraded":"2024-04-26T01:38:53.778546+0000","last_fullsized":"2024-04-26T01:38:53.778546+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:32.207534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.779109+0000","last_change":"2024-04-26T01:37:25.499759+0000","last_active":"2024-04-26T01:38:53.779109+0000","last_peered":"2024-04-26T01:38:53.779109+0000","last_clean":"2024-04-26T01:38:53.779109+0000","last_became_active":"2024-04-26T01:37:25.499317+0000","last_became_peered":"2024-04-26T01:37:25.499317+0000","last_unstale":"2024-04-26T01:38:53.779109+0000","last_undegraded":"2024-04-26T01:38:53.779109+0000","last_fullsized":"2024-04-26T01:38:53.779109+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:39.972315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.776288+0000","last_change":"2024-04-26T01:36:54.277457+0000","last_active":"2024-04-26T01:38:53.776288+0000","last_peered":"2024-04-26T01:38:53.776288+0000","last_clean":"2024-04-26T01:38:53.776288+0000","last_became_active":"2024-04-26T01:36:54.277118+0000","last_became_peered":"2024-04-26T01:36:54.277118+0000","last_unstale":"2024-04-26T01:38:53.776288+0000","last_undegraded":"2024-04-26T01:38:53.776288+0000","last_fullsized":"2024-04-26T01:38:53.776288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:05.628027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.762843+0000","last_change":"2024-04-26T01:36:54.278494+0000","last_active":"2024-04-26T01:38:53.762843+0000","last_peered":"2024-04-26T01:38:53.762843+0000","last_clean":"2024-04-26T01:38:53.762843+0000","last_became_active":"2024-04-26T01:36:54.278188+0000","last_became_peered":"2024-04-26T01:36:54.278188+0000","last_unstale":"2024-04-26T01:38:53.762843+0000","last_undegraded":"2024-04-26T01:38:53.762843+0000","last_fullsized":"2024-04-26T01:38:53.762843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:59.150771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.772089+0000","last_change":"2024-04-26T01:36:54.277424+0000","last_active":"2024-04-26T01:38:53.772089+0000","last_peered":"2024-04-26T01:38:53.772089+0000","last_clean":"2024-04-26T01:38:53.772089+0000","last_became_active":"2024-04-26T01:36:54.275752+0000","last_became_peered":"2024-04-26T01:36:54.275752+0000","last_unstale":"2024-04-26T01:38:53.772089+0000","last_undegraded":"2024-04-26T01:38:53.772089+0000","last_fullsized":"2024-04-26T01:38:53.772089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:22.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.776361+0000","last_change":"2024-04-26T01:37:25.502448+0000","last_active":"2024-04-26T01:38:53.776361+0000","last_peered":"2024-04-26T01:38:53.776361+0000","last_clean":"2024-04-26T01:38:53.776361+0000","last_became_active":"2024-04-26T01:37:25.501161+0000","last_became_peered":"2024-04-26T01:37:25.501161+0000","last_unstale":"2024-04-26T01:38:53.776361+0000","last_undegraded":"2024-04-26T01:38:53.776361+0000","last_fullsized":"2024-04-26T01:38:53.776361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:41:40.443780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.791351+0000","last_change":"2024-04-26T01:36:54.268721+0000","last_active":"2024-04-26T01:38:53.791351+0000","last_peered":"2024-04-26T01:38:53.791351+0000","last_clean":"2024-04-26T01:38:53.791351+0000","last_became_active":"2024-04-26T01:36:54.268007+0000","last_became_peered":"2024-04-26T01:36:54.268007+0000","last_unstale":"2024-04-26T01:38:53.791351+0000","last_undegraded":"2024-04-26T01:38:53.791351+0000","last_fullsized":"2024-04-26T01:38:53.791351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:55.120130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777956+0000","last_change":"2024-04-26T01:36:54.286595+0000","last_active":"2024-04-26T01:38:53.777956+0000","last_peered":"2024-04-26T01:38:53.777956+0000","last_clean":"2024-04-26T01:38:53.777956+0000","last_became_active":"2024-04-26T01:36:54.286236+0000","last_became_peered":"2024-04-26T01:36:54.286236+0000","last_unstale":"2024-04-26T01:38:53.777956+0000","last_undegraded":"2024-04-26T01:38:53.777956+0000","last_fullsized":"2024-04-26T01:38:53.777956+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:59.443467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.775994+0000","last_change":"2024-04-26T01:36:54.293254+0000","last_active":"2024-04-26T01:38:53.775994+0000","last_peered":"2024-04-26T01:38:53.775994+0000","last_clean":"2024-04-26T01:38:53.775994+0000","last_became_active":"2024-04-26T01:36:54.292853+0000","last_became_peered":"2024-04-26T01:36:54.292853+0000","last_unstale":"2024-04-26T01:38:53.775994+0000","last_undegraded":"2024-04-26T01:38:53.775994+0000","last_fullsized":"2024-04-26T01:38:53.775994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:19.844006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777829+0000","last_change":"2024-04-26T01:36:54.285346+0000","last_active":"2024-04-26T01:38:53.777829+0000","last_peered":"2024-04-26T01:38:53.777829+0000","last_clean":"2024-04-26T01:38:53.777829+0000","last_became_active":"2024-04-26T01:36:54.284949+0000","last_became_peered":"2024-04-26T01:36:54.284949+0000","last_unstale":"2024-04-26T01:38:53.777829+0000","last_undegraded":"2024-04-26T01:38:53.777829+0000","last_fullsized":"2024-04-26T01:38:53.777829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:33:29.276240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777190+0000","last_change":"2024-04-26T01:36:54.283161+0000","last_active":"2024-04-26T01:38:53.777190+0000","last_peered":"2024-04-26T01:38:53.777190+0000","last_clean":"2024-04-26T01:38:53.777190+0000","last_became_active":"2024-04-26T01:36:54.282908+0000","last_became_peered":"2024-04-26T01:36:54.282908+0000","last_unstale":"2024-04-26T01:38:53.777190+0000","last_undegraded":"2024-04-26T01:38:53.777190+0000","last_fullsized":"2024-04-26T01:38:53.777190+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:26.503964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777767+0000","last_change":"2024-04-26T01:36:54.282516+0000","last_active":"2024-04-26T01:38:53.777767+0000","last_peered":"2024-04-26T01:38:53.777767+0000","last_clean":"2024-04-26T01:38:53.777767+0000","last_became_active":"2024-04-26T01:36:54.278634+0000","last_became_peered":"2024-04-26T01:36:54.278634+0000","last_unstale":"2024-04-26T01:38:53.777767+0000","last_undegraded":"2024-04-26T01:38:53.777767+0000","last_fullsized":"2024-04-26T01:38:53.777767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:10:38.275720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780482+0000","last_change":"2024-04-26T01:36:54.289502+0000","last_active":"2024-04-26T01:38:53.780482+0000","last_peered":"2024-04-26T01:38:53.780482+0000","last_clean":"2024-04-26T01:38:53.780482+0000","last_became_active":"2024-04-26T01:36:54.289079+0000","last_became_peered":"2024-04-26T01:36:54.289079+0000","last_unstale":"2024-04-26T01:38:53.780482+0000","last_undegraded":"2024-04-26T01:38:53.780482+0000","last_fullsized":"2024-04-26T01:38:53.780482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:51.412032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.498085+0000","last_change":"2024-04-26T01:36:50.243164+0000","last_active":"2024-04-26T01:37:25.498085+0000","last_peered":"2024-04-26T01:37:25.498085+0000","last_clean":"2024-04-26T01:37:25.498085+0000","last_became_active":"2024-04-26T01:36:50.242508+0000","last_became_peered":"2024-04-26T01:36:50.242508+0000","last_unstale":"2024-04-26T01:37:25.498085+0000","last_undegraded":"2024-04-26T01:37:25.498085+0000","last_fullsized":"2024-04-26T01:37:25.498085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:06:01.662698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.783619+0000","last_change":"2024-04-26T01:36:54.279809+0000","last_active":"2024-04-26T01:38:53.783619+0000","last_peered":"2024-04-26T01:38:53.783619+0000","last_clean":"2024-04-26T01:38:53.783619+0000","last_became_active":"2024-04-26T01:36:54.279509+0000","last_became_peered":"2024-04-26T01:36:54.279509+0000","last_unstale":"2024-04-26T01:38:53.783619+0000","last_undegraded":"2024-04-26T01:38:53.783619+0000","last_fullsized":"2024-04-26T01:38:53.783619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:54:26.734136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.515168+0000","last_change":"2024-04-26T01:36:50.261942+0000","last_active":"2024-04-26T01:37:25.515168+0000","last_peered":"2024-04-26T01:37:25.515168+0000","last_clean":"2024-04-26T01:37:25.515168+0000","last_became_active":"2024-04-26T01:36:50.261564+0000","last_became_peered":"2024-04-26T01:36:50.261564+0000","last_unstale":"2024-04-26T01:37:25.515168+0000","last_undegraded":"2024-04-26T01:37:25.515168+0000","last_fullsized":"2024-04-26T01:37:25.515168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:25.102557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780874+0000","last_change":"2024-04-26T01:36:54.288011+0000","last_active":"2024-04-26T01:38:53.780874+0000","last_peered":"2024-04-26T01:38:53.780874+0000","last_clean":"2024-04-26T01:38:53.780874+0000","last_became_active":"2024-04-26T01:36:54.286853+0000","last_became_peered":"2024-04-26T01:36:54.286853+0000","last_unstale":"2024-04-26T01:38:53.780874+0000","last_undegraded":"2024-04-26T01:38:53.780874+0000","last_fullsized":"2024-04-26T01:38:53.780874+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:53.422470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495834+0000","last_change":"2024-04-26T01:36:50.235291+0000","last_active":"2024-04-26T01:37:25.495834+0000","last_peered":"2024-04-26T01:37:25.495834+0000","last_clean":"2024-04-26T01:37:25.495834+0000","last_became_active":"2024-04-26T01:36:50.234939+0000","last_became_peered":"2024-04-26T01:36:50.234939+0000","last_unstale":"2024-04-26T01:37:25.495834+0000","last_undegraded":"2024-04-26T01:37:25.495834+0000","last_fullsized":"2024-04-26T01:37:25.495834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:13.048185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785426+0000","last_change":"2024-04-26T01:36:54.282731+0000","last_active":"2024-04-26T01:38:53.785426+0000","last_peered":"2024-04-26T01:38:53.785426+0000","last_clean":"2024-04-26T01:38:53.785426+0000","last_became_active":"2024-04-26T01:36:54.282384+0000","last_became_peered":"2024-04-26T01:36:54.282384+0000","last_unstale":"2024-04-26T01:38:53.785426+0000","last_undegraded":"2024-04-26T01:38:53.785426+0000","last_fullsized":"2024-04-26T01:38:53.785426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:09.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495596+0000","last_change":"2024-04-26T01:36:50.233581+0000","last_active":"2024-04-26T01:37:25.495596+0000","last_peered":"2024-04-26T01:37:25.495596+0000","last_clean":"2024-04-26T01:37:25.495596+0000","last_became_active":"2024-04-26T01:36:50.233210+0000","last_became_peered":"2024-04-26T01:36:50.233210+0000","last_unstale":"2024-04-26T01:37:25.495596+0000","last_undegraded":"2024-04-26T01:37:25.495596+0000","last_fullsized":"2024-04-26T01:37:25.495596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:01.223954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.772477+0000","last_change":"2024-04-26T01:36:54.274657+0000","last_active":"2024-04-26T01:38:53.772477+0000","last_peered":"2024-04-26T01:38:53.772477+0000","last_clean":"2024-04-26T01:38:53.772477+0000","last_became_active":"2024-04-26T01:36:54.274333+0000","last_became_peered":"2024-04-26T01:36:54.274333+0000","last_unstale":"2024-04-26T01:38:53.772477+0000","last_undegraded":"2024-04-26T01:38:53.772477+0000","last_fullsized":"2024-04-26T01:38:53.772477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:37.243307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":169,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:15.737187+0000","last_change":"2024-04-26T01:37:25.502311+0000","last_active":"2024-04-26T01:39:15.737187+0000","last_peered":"2024-04-26T01:39:15.737187+0000","last_clean":"2024-04-26T01:39:15.737187+0000","last_became_active":"2024-04-26T01:37:25.493259+0000","last_became_peered":"2024-04-26T01:37:25.493259+0000","last_unstale":"2024-04-26T01:39:15.737187+0000","last_undegraded":"2024-04-26T01:39:15.737187+0000","last_fullsized":"2024-04-26T01:39:15.737187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:46:49.224905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.773356+0000","last_change":"2024-04-26T01:36:54.275116+0000","last_active":"2024-04-26T01:38:53.773356+0000","last_peered":"2024-04-26T01:38:53.773356+0000","last_clean":"2024-04-26T01:38:53.773356+0000","last_became_active":"2024-04-26T01:36:54.274815+0000","last_became_peered":"2024-04-26T01:36:54.274815+0000","last_unstale":"2024-04-26T01:38:53.773356+0000","last_undegraded":"2024-04-26T01:38:53.773356+0000","last_fullsized":"2024-04-26T01:38:53.773356+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:04.427670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.505327+0000","last_change":"2024-04-26T01:37:25.505327+0000","last_active":"2024-04-26T01:37:25.505327+0000","last_peered":"2024-04-26T01:37:25.505327+0000","last_clean":"2024-04-26T01:37:25.505327+0000","last_became_active":"2024-04-26T01:37:25.497881+0000","last_became_peered":"2024-04-26T01:37:25.497881+0000","last_unstale":"2024-04-26T01:37:25.505327+0000","last_undegraded":"2024-04-26T01:37:25.505327+0000","last_fullsized":"2024-04-26T01:37:25.505327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:09.181372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786730+0000","last_change":"2024-04-26T01:36:54.276415+0000","last_active":"2024-04-26T01:38:53.786730+0000","last_peered":"2024-04-26T01:38:53.786730+0000","last_clean":"2024-04-26T01:38:53.786730+0000","last_became_active":"2024-04-26T01:36:54.276080+0000","last_became_peered":"2024-04-26T01:36:54.276080+0000","last_unstale":"2024-04-26T01:38:53.786730+0000","last_undegraded":"2024-04-26T01:38:53.786730+0000","last_fullsized":"2024-04-26T01:38:53.786730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:08:17.171620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491996+0000","last_change":"2024-04-26T01:36:50.241552+0000","last_active":"2024-04-26T01:37:25.491996+0000","last_peered":"2024-04-26T01:37:25.491996+0000","last_clean":"2024-04-26T01:37:25.491996+0000","last_became_active":"2024-04-26T01:36:50.241022+0000","last_became_peered":"2024-04-26T01:36:50.241022+0000","last_unstale":"2024-04-26T01:37:25.491996+0000","last_undegraded":"2024-04-26T01:37:25.491996+0000","last_fullsized":"2024-04-26T01:37:25.491996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:55.503831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.787730+0000","last_change":"2024-04-26T01:36:54.276641+0000","last_active":"2024-04-26T01:38:53.787730+0000","last_peered":"2024-04-26T01:38:53.787730+0000","last_clean":"2024-04-26T01:38:53.787730+0000","last_became_active":"2024-04-26T01:36:54.276309+0000","last_became_peered":"2024-04-26T01:36:54.276309+0000","last_unstale":"2024-04-26T01:38:53.787730+0000","last_undegraded":"2024-04-26T01:38:53.787730+0000","last_fullsized":"2024-04-26T01:38:53.787730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:53.725584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489011+0000","last_change":"2024-04-26T01:36:50.235504+0000","last_active":"2024-04-26T01:37:25.489011+0000","last_peered":"2024-04-26T01:37:25.489011+0000","last_clean":"2024-04-26T01:37:25.489011+0000","last_became_active":"2024-04-26T01:36:50.235161+0000","last_became_peered":"2024-04-26T01:36:50.235161+0000","last_unstale":"2024-04-26T01:37:25.489011+0000","last_undegraded":"2024-04-26T01:37:25.489011+0000","last_fullsized":"2024-04-26T01:37:25.489011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:05.951521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785198+0000","last_change":"2024-04-26T01:36:54.298236+0000","last_active":"2024-04-26T01:38:53.785198+0000","last_peered":"2024-04-26T01:38:53.785198+0000","last_clean":"2024-04-26T01:38:53.785198+0000","last_became_active":"2024-04-26T01:36:54.297850+0000","last_became_peered":"2024-04-26T01:36:54.297850+0000","last_unstale":"2024-04-26T01:38:53.785198+0000","last_undegraded":"2024-04-26T01:38:53.785198+0000","last_fullsized":"2024-04-26T01:38:53.785198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:35:35.244502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:14.734157+0000","last_change":"2024-04-26T01:36:50.241418+0000","last_active":"2024-04-26T01:39:14.734157+0000","last_peered":"2024-04-26T01:39:14.734157+0000","last_clean":"2024-04-26T01:39:14.734157+0000","last_became_active":"2024-04-26T01:36:50.241044+0000","last_became_peered":"2024-04-26T01:36:50.241044+0000","last_unstale":"2024-04-26T01:39:14.734157+0000","last_undegraded":"2024-04-26T01:39:14.734157+0000","last_fullsized":"2024-04-26T01:39:14.734157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:13.317953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.776030+0000","last_change":"2024-04-26T01:36:54.284410+0000","last_active":"2024-04-26T01:38:53.776030+0000","last_peered":"2024-04-26T01:38:53.776030+0000","last_clean":"2024-04-26T01:38:53.776030+0000","last_became_active":"2024-04-26T01:36:54.283629+0000","last_became_peered":"2024-04-26T01:36:54.283629+0000","last_unstale":"2024-04-26T01:38:53.776030+0000","last_undegraded":"2024-04-26T01:38:53.776030+0000","last_fullsized":"2024-04-26T01:38:53.776030+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:15.380006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.500348+0000","last_change":"2024-04-26T01:37:25.499874+0000","last_active":"2024-04-26T01:37:25.500348+0000","last_peered":"2024-04-26T01:37:25.500348+0000","last_clean":"2024-04-26T01:37:25.500348+0000","last_became_active":"2024-04-26T01:37:25.494878+0000","last_became_peered":"2024-04-26T01:37:25.494878+0000","last_unstale":"2024-04-26T01:37:25.500348+0000","last_undegraded":"2024-04-26T01:37:25.500348+0000","last_fullsized":"2024-04-26T01:37:25.500348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:48:12.678331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777571+0000","last_change":"2024-04-26T01:36:54.285118+0000","last_active":"2024-04-26T01:38:53.777571+0000","last_peered":"2024-04-26T01:38:53.777571+0000","last_clean":"2024-04-26T01:38:53.777571+0000","last_became_active":"2024-04-26T01:36:54.284554+0000","last_became_peered":"2024-04-26T01:36:54.284554+0000","last_unstale":"2024-04-26T01:38:53.777571+0000","last_undegraded":"2024-04-26T01:38:53.777571+0000","last_fullsized":"2024-04-26T01:38:53.777571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:13.721684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489735+0000","last_change":"2024-04-26T01:36:50.262451+0000","last_active":"2024-04-26T01:37:25.489735+0000","last_peered":"2024-04-26T01:37:25.489735+0000","last_clean":"2024-04-26T01:37:25.489735+0000","last_became_active":"2024-04-26T01:36:50.262056+0000","last_became_peered":"2024-04-26T01:36:50.262056+0000","last_unstale":"2024-04-26T01:37:25.489735+0000","last_undegraded":"2024-04-26T01:37:25.489735+0000","last_fullsized":"2024-04-26T01:37:25.489735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:50.813675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.774764+0000","last_change":"2024-04-26T01:36:54.283975+0000","last_active":"2024-04-26T01:38:53.774764+0000","last_peered":"2024-04-26T01:38:53.774764+0000","last_clean":"2024-04-26T01:38:53.774764+0000","last_became_active":"2024-04-26T01:36:54.283564+0000","last_became_peered":"2024-04-26T01:36:54.283564+0000","last_unstale":"2024-04-26T01:38:53.774764+0000","last_undegraded":"2024-04-26T01:38:53.774764+0000","last_fullsized":"2024-04-26T01:38:53.774764+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:53.881763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":308,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:14.627265+0000","last_change":"2024-04-26T01:36:50.238839+0000","last_active":"2024-04-26T01:39:14.627265+0000","last_peered":"2024-04-26T01:39:14.627265+0000","last_clean":"2024-04-26T01:39:14.627265+0000","last_became_active":"2024-04-26T01:36:50.238452+0000","last_became_peered":"2024-04-26T01:36:50.238452+0000","last_unstale":"2024-04-26T01:39:14.627265+0000","last_undegraded":"2024-04-26T01:39:14.627265+0000","last_fullsized":"2024-04-26T01:39:14.627265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:05:40.426825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65899,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":246,"num_read_kb":187,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.768385+0000","last_change":"2024-04-26T01:36:54.271446+0000","last_active":"2024-04-26T01:38:53.768385+0000","last_peered":"2024-04-26T01:38:53.768385+0000","last_clean":"2024-04-26T01:38:53.768385+0000","last_became_active":"2024-04-26T01:36:54.271122+0000","last_became_peered":"2024-04-26T01:36:54.271122+0000","last_unstale":"2024-04-26T01:38:53.768385+0000","last_undegraded":"2024-04-26T01:38:53.768385+0000","last_fullsized":"2024-04-26T01:38:53.768385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:28:24.275481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491729+0000","last_change":"2024-04-26T01:36:50.237726+0000","last_active":"2024-04-26T01:37:25.491729+0000","last_peered":"2024-04-26T01:37:25.491729+0000","last_clean":"2024-04-26T01:37:25.491729+0000","last_became_active":"2024-04-26T01:36:50.237215+0000","last_became_peered":"2024-04-26T01:36:50.237215+0000","last_unstale":"2024-04-26T01:37:25.491729+0000","last_undegraded":"2024-04-26T01:37:25.491729+0000","last_fullsized":"2024-04-26T01:37:25.491729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:41.781876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.779276+0000","last_change":"2024-04-26T01:36:54.290616+0000","last_active":"2024-04-26T01:38:53.779276+0000","last_peered":"2024-04-26T01:38:53.779276+0000","last_clean":"2024-04-26T01:38:53.779276+0000","last_became_active":"2024-04-26T01:36:54.290293+0000","last_became_peered":"2024-04-26T01:36:54.290293+0000","last_unstale":"2024-04-26T01:38:53.779276+0000","last_undegraded":"2024-04-26T01:38:53.779276+0000","last_fullsized":"2024-04-26T01:38:53.779276+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:07.439333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489203+0000","last_change":"2024-04-26T01:36:50.237578+0000","last_active":"2024-04-26T01:37:25.489203+0000","last_peered":"2024-04-26T01:37:25.489203+0000","last_clean":"2024-04-26T01:37:25.489203+0000","last_became_active":"2024-04-26T01:36:50.237224+0000","last_became_peered":"2024-04-26T01:36:50.237224+0000","last_unstale":"2024-04-26T01:37:25.489203+0000","last_undegraded":"2024-04-26T01:37:25.489203+0000","last_fullsized":"2024-04-26T01:37:25.489203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:46:38.078631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.790370+0000","last_change":"2024-04-26T01:36:54.280818+0000","last_active":"2024-04-26T01:38:53.790370+0000","last_peered":"2024-04-26T01:38:53.790370+0000","last_clean":"2024-04-26T01:38:53.790370+0000","last_became_active":"2024-04-26T01:36:54.280440+0000","last_became_peered":"2024-04-26T01:36:54.280440+0000","last_unstale":"2024-04-26T01:38:53.790370+0000","last_undegraded":"2024-04-26T01:38:53.790370+0000","last_fullsized":"2024-04-26T01:38:53.790370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:47.185219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:16.709910+0000","last_change":"2024-04-26T01:36:50.234602+0000","last_active":"2024-04-26T01:39:16.709910+0000","last_peered":"2024-04-26T01:39:16.709910+0000","last_clean":"2024-04-26T01:39:16.709910+0000","last_became_active":"2024-04-26T01:36:50.234175+0000","last_became_peered":"2024-04-26T01:36:50.234175+0000","last_unstale":"2024-04-26T01:39:16.709910+0000","last_undegraded":"2024-04-26T01:39:16.709910+0000","last_fullsized":"2024-04-26T01:39:16.709910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:17:14.236069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.766641+0000","last_change":"2024-04-26T01:36:54.271466+0000","last_active":"2024-04-26T01:38:53.766641+0000","last_peered":"2024-04-26T01:38:53.766641+0000","last_clean":"2024-04-26T01:38:53.766641+0000","last_became_active":"2024-04-26T01:36:54.271086+0000","last_became_peered":"2024-04-26T01:36:54.271086+0000","last_unstale":"2024-04-26T01:38:53.766641+0000","last_undegraded":"2024-04-26T01:38:53.766641+0000","last_fullsized":"2024-04-26T01:38:53.766641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:41.052895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.519343+0000","last_change":"2024-04-26T01:37:25.518730+0000","last_active":"2024-04-26T01:37:25.519343+0000","last_peered":"2024-04-26T01:37:25.519343+0000","last_clean":"2024-04-26T01:37:25.519343+0000","last_became_active":"2024-04-26T01:37:25.504464+0000","last_became_peered":"2024-04-26T01:37:25.504464+0000","last_unstale":"2024-04-26T01:37:25.519343+0000","last_undegraded":"2024-04-26T01:37:25.519343+0000","last_fullsized":"2024-04-26T01:37:25.519343+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:07.578917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.771883+0000","last_change":"2024-04-26T01:36:54.273621+0000","last_active":"2024-04-26T01:38:53.771883+0000","last_peered":"2024-04-26T01:38:53.771883+0000","last_clean":"2024-04-26T01:38:53.771883+0000","last_became_active":"2024-04-26T01:36:54.273218+0000","last_became_peered":"2024-04-26T01:36:54.273218+0000","last_unstale":"2024-04-26T01:38:53.771883+0000","last_undegraded":"2024-04-26T01:38:53.771883+0000","last_fullsized":"2024-04-26T01:38:53.771883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:55.543943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.509030+0000","last_change":"2024-04-26T01:37:25.508575+0000","last_active":"2024-04-26T01:37:25.509030+0000","last_peered":"2024-04-26T01:37:25.509030+0000","last_clean":"2024-04-26T01:37:25.509030+0000","last_became_active":"2024-04-26T01:37:25.496407+0000","last_became_peered":"2024-04-26T01:37:25.496407+0000","last_unstale":"2024-04-26T01:37:25.509030+0000","last_undegraded":"2024-04-26T01:37:25.509030+0000","last_fullsized":"2024-04-26T01:37:25.509030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:05.878252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.758792+0000","last_change":"2024-04-26T01:36:54.272147+0000","last_active":"2024-04-26T01:38:53.758792+0000","last_peered":"2024-04-26T01:38:53.758792+0000","last_clean":"2024-04-26T01:38:53.758792+0000","last_became_active":"2024-04-26T01:36:54.271737+0000","last_became_peered":"2024-04-26T01:36:54.271737+0000","last_unstale":"2024-04-26T01:38:53.758792+0000","last_undegraded":"2024-04-26T01:38:53.758792+0000","last_fullsized":"2024-04-26T01:38:53.758792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:41.445073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.496906+0000","last_change":"2024-04-26T01:36:50.264444+0000","last_active":"2024-04-26T01:37:25.496906+0000","last_peered":"2024-04-26T01:37:25.496906+0000","last_clean":"2024-04-26T01:37:25.496906+0000","last_became_active":"2024-04-26T01:36:50.264058+0000","last_became_peered":"2024-04-26T01:36:50.264058+0000","last_unstale":"2024-04-26T01:37:25.496906+0000","last_undegraded":"2024-04-26T01:37:25.496906+0000","last_fullsized":"2024-04-26T01:37:25.496906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:56:15.428588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780458+0000","last_change":"2024-04-26T01:36:54.289537+0000","last_active":"2024-04-26T01:38:53.780458+0000","last_peered":"2024-04-26T01:38:53.780458+0000","last_clean":"2024-04-26T01:38:53.780458+0000","last_became_active":"2024-04-26T01:36:54.289149+0000","last_became_peered":"2024-04-26T01:36:54.289149+0000","last_unstale":"2024-04-26T01:38:53.780458+0000","last_undegraded":"2024-04-26T01:38:53.780458+0000","last_fullsized":"2024-04-26T01:38:53.780458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:07.801551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491568+0000","last_change":"2024-04-26T01:36:50.242190+0000","last_active":"2024-04-26T01:37:25.491568+0000","last_peered":"2024-04-26T01:37:25.491568+0000","last_clean":"2024-04-26T01:37:25.491568+0000","last_became_active":"2024-04-26T01:36:50.241890+0000","last_became_peered":"2024-04-26T01:36:50.241890+0000","last_unstale":"2024-04-26T01:37:25.491568+0000","last_undegraded":"2024-04-26T01:37:25.491568+0000","last_fullsized":"2024-04-26T01:37:25.491568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:29.472502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.781419+0000","last_change":"2024-04-26T01:36:54.294699+0000","last_active":"2024-04-26T01:38:53.781419+0000","last_peered":"2024-04-26T01:38:53.781419+0000","last_clean":"2024-04-26T01:38:53.781419+0000","last_became_active":"2024-04-26T01:36:54.294313+0000","last_became_peered":"2024-04-26T01:36:54.294313+0000","last_unstale":"2024-04-26T01:38:53.781419+0000","last_undegraded":"2024-04-26T01:38:53.781419+0000","last_fullsized":"2024-04-26T01:38:53.781419+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:15.026357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.771052+0000","last_change":"2024-04-26T01:36:54.276798+0000","last_active":"2024-04-26T01:38:53.771052+0000","last_peered":"2024-04-26T01:38:53.771052+0000","last_clean":"2024-04-26T01:38:53.771052+0000","last_became_active":"2024-04-26T01:36:54.275955+0000","last_became_peered":"2024-04-26T01:36:54.275955+0000","last_unstale":"2024-04-26T01:38:53.771052+0000","last_undegraded":"2024-04-26T01:38:53.771052+0000","last_fullsized":"2024-04-26T01:38:53.771052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:17.020909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:17.711005+0000","last_change":"2024-04-26T01:36:50.263356+0000","last_active":"2024-04-26T01:39:17.711005+0000","last_peered":"2024-04-26T01:39:17.711005+0000","last_clean":"2024-04-26T01:39:17.711005+0000","last_became_active":"2024-04-26T01:36:50.261596+0000","last_became_peered":"2024-04-26T01:36:50.261596+0000","last_unstale":"2024-04-26T01:39:17.711005+0000","last_undegraded":"2024-04-26T01:39:17.711005+0000","last_fullsized":"2024-04-26T01:39:17.711005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:10.773347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785734+0000","last_change":"2024-04-26T01:36:54.288135+0000","last_active":"2024-04-26T01:38:53.785734+0000","last_peered":"2024-04-26T01:38:53.785734+0000","last_clean":"2024-04-26T01:38:53.785734+0000","last_became_active":"2024-04-26T01:36:54.287728+0000","last_became_peered":"2024-04-26T01:36:54.287728+0000","last_unstale":"2024-04-26T01:38:53.785734+0000","last_undegraded":"2024-04-26T01:38:53.785734+0000","last_fullsized":"2024-04-26T01:38:53.785734+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:18.428151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497337+0000","last_change":"2024-04-26T01:36:50.243522+0000","last_active":"2024-04-26T01:37:25.497337+0000","last_peered":"2024-04-26T01:37:25.497337+0000","last_clean":"2024-04-26T01:37:25.497337+0000","last_became_active":"2024-04-26T01:36:50.243084+0000","last_became_peered":"2024-04-26T01:36:50.243084+0000","last_unstale":"2024-04-26T01:37:25.497337+0000","last_undegraded":"2024-04-26T01:37:25.497337+0000","last_fullsized":"2024-04-26T01:37:25.497337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:03.159750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491542+0000","last_change":"2024-04-26T01:35:13.920486+0000","last_active":"2024-04-26T01:37:25.491542+0000","last_peered":"2024-04-26T01:37:25.491542+0000","last_clean":"2024-04-26T01:37:25.491542+0000","last_became_active":"2024-04-26T01:35:13.919834+0000","last_became_peered":"2024-04-26T01:35:13.919834+0000","last_unstale":"2024-04-26T01:37:25.491542+0000","last_undegraded":"2024-04-26T01:37:25.491542+0000","last_fullsized":"2024-04-26T01:37:25.491542+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_clean_scrub_stamp":"2024-04-26T01:31:12.563114+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:53.935491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780533+0000","last_change":"2024-04-26T01:36:54.270553+0000","last_active":"2024-04-26T01:38:53.780533+0000","last_peered":"2024-04-26T01:38:53.780533+0000","last_clean":"2024-04-26T01:38:53.780533+0000","last_became_active":"2024-04-26T01:36:54.269986+0000","last_became_peered":"2024-04-26T01:36:54.269986+0000","last_unstale":"2024-04-26T01:38:53.780533+0000","last_undegraded":"2024-04-26T01:38:53.780533+0000","last_fullsized":"2024-04-26T01:38:53.780533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:19.597337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491607+0000","last_change":"2024-04-26T01:36:50.235583+0000","last_active":"2024-04-26T01:37:25.491607+0000","last_peered":"2024-04-26T01:37:25.491607+0000","last_clean":"2024-04-26T01:37:25.491607+0000","last_became_active":"2024-04-26T01:36:50.235214+0000","last_became_peered":"2024-04-26T01:36:50.235214+0000","last_unstale":"2024-04-26T01:37:25.491607+0000","last_undegraded":"2024-04-26T01:37:25.491607+0000","last_fullsized":"2024-04-26T01:37:25.491607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:05.822872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495731+0000","last_change":"2024-04-26T01:36:50.260988+0000","last_active":"2024-04-26T01:37:25.495731+0000","last_peered":"2024-04-26T01:37:25.495731+0000","last_clean":"2024-04-26T01:37:25.495731+0000","last_became_active":"2024-04-26T01:36:50.260615+0000","last_became_peered":"2024-04-26T01:36:50.260615+0000","last_unstale":"2024-04-26T01:37:25.495731+0000","last_undegraded":"2024-04-26T01:37:25.495731+0000","last_fullsized":"2024-04-26T01:37:25.495731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:15.933939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785032+0000","last_change":"2024-04-26T01:36:54.279650+0000","last_active":"2024-04-26T01:38:53.785032+0000","last_peered":"2024-04-26T01:38:53.785032+0000","last_clean":"2024-04-26T01:38:53.785032+0000","last_became_active":"2024-04-26T01:36:54.275814+0000","last_became_peered":"2024-04-26T01:36:54.275814+0000","last_unstale":"2024-04-26T01:38:53.785032+0000","last_undegraded":"2024-04-26T01:38:53.785032+0000","last_fullsized":"2024-04-26T01:38:53.785032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:58:42.369511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:54.319406+0000","last_change":"2024-04-26T01:36:50.248179+0000","last_active":"2024-04-26T01:38:54.319406+0000","last_peered":"2024-04-26T01:38:54.319406+0000","last_clean":"2024-04-26T01:38:54.319406+0000","last_became_active":"2024-04-26T01:36:50.247805+0000","last_became_peered":"2024-04-26T01:36:50.247805+0000","last_unstale":"2024-04-26T01:38:54.319406+0000","last_undegraded":"2024-04-26T01:38:54.319406+0000","last_fullsized":"2024-04-26T01:38:54.319406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:19:23.740920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.783371+0000","last_change":"2024-04-26T01:36:54.276626+0000","last_active":"2024-04-26T01:38:53.783371+0000","last_peered":"2024-04-26T01:38:53.783371+0000","last_clean":"2024-04-26T01:38:53.783371+0000","last_became_active":"2024-04-26T01:36:54.276238+0000","last_became_peered":"2024-04-26T01:36:54.276238+0000","last_unstale":"2024-04-26T01:38:53.783371+0000","last_undegraded":"2024-04-26T01:38:53.783371+0000","last_fullsized":"2024-04-26T01:38:53.783371+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:57.306950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497813+0000","last_change":"2024-04-26T01:36:50.278502+0000","last_active":"2024-04-26T01:37:25.497813+0000","last_peered":"2024-04-26T01:37:25.497813+0000","last_clean":"2024-04-26T01:37:25.497813+0000","last_became_active":"2024-04-26T01:36:50.278215+0000","last_became_peered":"2024-04-26T01:36:50.278215+0000","last_unstale":"2024-04-26T01:37:25.497813+0000","last_undegraded":"2024-04-26T01:37:25.497813+0000","last_fullsized":"2024-04-26T01:37:25.497813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:05:46.142902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780727+0000","last_change":"2024-04-26T01:36:54.292600+0000","last_active":"2024-04-26T01:38:53.780727+0000","last_peered":"2024-04-26T01:38:53.780727+0000","last_clean":"2024-04-26T01:38:53.780727+0000","last_became_active":"2024-04-26T01:36:54.292260+0000","last_became_peered":"2024-04-26T01:36:54.292260+0000","last_unstale":"2024-04-26T01:38:53.780727+0000","last_undegraded":"2024-04-26T01:38:53.780727+0000","last_fullsized":"2024-04-26T01:38:53.780727+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:10:12.296180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488801+0000","last_change":"2024-04-26T01:36:50.262479+0000","last_active":"2024-04-26T01:37:25.488801+0000","last_peered":"2024-04-26T01:37:25.488801+0000","last_clean":"2024-04-26T01:37:25.488801+0000","last_became_active":"2024-04-26T01:36:50.262109+0000","last_became_peered":"2024-04-26T01:36:50.262109+0000","last_unstale":"2024-04-26T01:37:25.488801+0000","last_undegraded":"2024-04-26T01:37:25.488801+0000","last_fullsized":"2024-04-26T01:37:25.488801+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:19.805321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.759863+0000","last_change":"2024-04-26T01:36:54.270051+0000","last_active":"2024-04-26T01:38:53.759863+0000","last_peered":"2024-04-26T01:38:53.759863+0000","last_clean":"2024-04-26T01:38:53.759863+0000","last_became_active":"2024-04-26T01:36:54.269670+0000","last_became_peered":"2024-04-26T01:36:54.269670+0000","last_unstale":"2024-04-26T01:38:53.759863+0000","last_undegraded":"2024-04-26T01:38:53.759863+0000","last_fullsized":"2024-04-26T01:38:53.759863+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:44.137454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514365+0000","last_change":"2024-04-26T01:36:50.248107+0000","last_active":"2024-04-26T01:37:25.514365+0000","last_peered":"2024-04-26T01:37:25.514365+0000","last_clean":"2024-04-26T01:37:25.514365+0000","last_became_active":"2024-04-26T01:36:50.247695+0000","last_became_peered":"2024-04-26T01:36:50.247695+0000","last_unstale":"2024-04-26T01:37:25.514365+0000","last_undegraded":"2024-04-26T01:37:25.514365+0000","last_fullsized":"2024-04-26T01:37:25.514365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:52.702512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.773517+0000","last_change":"2024-04-26T01:36:54.278686+0000","last_active":"2024-04-26T01:38:53.773517+0000","last_peered":"2024-04-26T01:38:53.773517+0000","last_clean":"2024-04-26T01:38:53.773517+0000","last_became_active":"2024-04-26T01:36:54.278312+0000","last_became_peered":"2024-04-26T01:36:54.278312+0000","last_unstale":"2024-04-26T01:38:53.773517+0000","last_undegraded":"2024-04-26T01:38:53.773517+0000","last_fullsized":"2024-04-26T01:38:53.773517+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:46.193032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491566+0000","last_change":"2024-04-26T01:36:50.236715+0000","last_active":"2024-04-26T01:37:25.491566+0000","last_peered":"2024-04-26T01:37:25.491566+0000","last_clean":"2024-04-26T01:37:25.491566+0000","last_became_active":"2024-04-26T01:36:50.236184+0000","last_became_peered":"2024-04-26T01:36:50.236184+0000","last_unstale":"2024-04-26T01:37:25.491566+0000","last_undegraded":"2024-04-26T01:37:25.491566+0000","last_fullsized":"2024-04-26T01:37:25.491566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:24:16.063401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785920+0000","last_change":"2024-04-26T01:36:54.290601+0000","last_active":"2024-04-26T01:38:53.785920+0000","last_peered":"2024-04-26T01:38:53.785920+0000","last_clean":"2024-04-26T01:38:53.785920+0000","last_became_active":"2024-04-26T01:36:54.290168+0000","last_became_peered":"2024-04-26T01:36:54.290168+0000","last_unstale":"2024-04-26T01:38:53.785920+0000","last_undegraded":"2024-04-26T01:38:53.785920+0000","last_fullsized":"2024-04-26T01:38:53.785920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:48.770431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494696+0000","last_change":"2024-04-26T01:36:50.235394+0000","last_active":"2024-04-26T01:37:25.494696+0000","last_peered":"2024-04-26T01:37:25.494696+0000","last_clean":"2024-04-26T01:37:25.494696+0000","last_became_active":"2024-04-26T01:36:50.235012+0000","last_became_peered":"2024-04-26T01:36:50.235012+0000","last_unstale":"2024-04-26T01:37:25.494696+0000","last_undegraded":"2024-04-26T01:37:25.494696+0000","last_fullsized":"2024-04-26T01:37:25.494696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:24:17.218945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.787197+0000","last_change":"2024-04-26T01:36:54.280490+0000","last_active":"2024-04-26T01:38:53.787197+0000","last_peered":"2024-04-26T01:38:53.787197+0000","last_clean":"2024-04-26T01:38:53.787197+0000","last_became_active":"2024-04-26T01:36:54.280047+0000","last_became_peered":"2024-04-26T01:36:54.280047+0000","last_unstale":"2024-04-26T01:38:53.787197+0000","last_undegraded":"2024-04-26T01:38:53.787197+0000","last_fullsized":"2024-04-26T01:38:53.787197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:32.610972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492756+0000","last_change":"2024-04-26T01:36:50.278325+0000","last_active":"2024-04-26T01:37:25.492756+0000","last_peered":"2024-04-26T01:37:25.492756+0000","last_clean":"2024-04-26T01:37:25.492756+0000","last_became_active":"2024-04-26T01:36:50.277882+0000","last_became_peered":"2024-04-26T01:36:50.277882+0000","last_unstale":"2024-04-26T01:37:25.492756+0000","last_undegraded":"2024-04-26T01:37:25.492756+0000","last_fullsized":"2024-04-26T01:37:25.492756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:06.082223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.772210+0000","last_change":"2024-04-26T01:36:54.278106+0000","last_active":"2024-04-26T01:38:53.772210+0000","last_peered":"2024-04-26T01:38:53.772210+0000","last_clean":"2024-04-26T01:38:53.772210+0000","last_became_active":"2024-04-26T01:36:54.277644+0000","last_became_peered":"2024-04-26T01:36:54.277644+0000","last_unstale":"2024-04-26T01:38:53.772210+0000","last_undegraded":"2024-04-26T01:38:53.772210+0000","last_fullsized":"2024-04-26T01:38:53.772210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:10.619640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489528+0000","last_change":"2024-04-26T01:36:50.237072+0000","last_active":"2024-04-26T01:37:25.489528+0000","last_peered":"2024-04-26T01:37:25.489528+0000","last_clean":"2024-04-26T01:37:25.489528+0000","last_became_active":"2024-04-26T01:36:50.236682+0000","last_became_peered":"2024-04-26T01:36:50.236682+0000","last_unstale":"2024-04-26T01:37:25.489528+0000","last_undegraded":"2024-04-26T01:37:25.489528+0000","last_fullsized":"2024-04-26T01:37:25.489528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:58:17.553492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.779926+0000","last_change":"2024-04-26T01:36:54.289997+0000","last_active":"2024-04-26T01:38:53.779926+0000","last_peered":"2024-04-26T01:38:53.779926+0000","last_clean":"2024-04-26T01:38:53.779926+0000","last_became_active":"2024-04-26T01:36:54.289505+0000","last_became_peered":"2024-04-26T01:36:54.289505+0000","last_unstale":"2024-04-26T01:38:53.779926+0000","last_undegraded":"2024-04-26T01:38:53.779926+0000","last_fullsized":"2024-04-26T01:38:53.779926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:09.340231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491370+0000","last_change":"2024-04-26T01:36:50.245231+0000","last_active":"2024-04-26T01:37:25.491370+0000","last_peered":"2024-04-26T01:37:25.491370+0000","last_clean":"2024-04-26T01:37:25.491370+0000","last_became_active":"2024-04-26T01:36:50.244811+0000","last_became_peered":"2024-04-26T01:36:50.244811+0000","last_unstale":"2024-04-26T01:37:25.491370+0000","last_undegraded":"2024-04-26T01:37:25.491370+0000","last_fullsized":"2024-04-26T01:37:25.491370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:12:08.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.768287+0000","last_change":"2024-04-26T01:36:54.272197+0000","last_active":"2024-04-26T01:38:53.768287+0000","last_peered":"2024-04-26T01:38:53.768287+0000","last_clean":"2024-04-26T01:38:53.768287+0000","last_became_active":"2024-04-26T01:36:54.271776+0000","last_became_peered":"2024-04-26T01:36:54.271776+0000","last_unstale":"2024-04-26T01:38:53.768287+0000","last_undegraded":"2024-04-26T01:38:53.768287+0000","last_fullsized":"2024-04-26T01:38:53.768287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:20.024890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.769047+0000","last_change":"2024-04-26T01:36:54.271120+0000","last_active":"2024-04-26T01:38:53.769047+0000","last_peered":"2024-04-26T01:38:53.769047+0000","last_clean":"2024-04-26T01:38:53.769047+0000","last_became_active":"2024-04-26T01:36:54.270796+0000","last_became_peered":"2024-04-26T01:36:54.270796+0000","last_unstale":"2024-04-26T01:38:53.769047+0000","last_undegraded":"2024-04-26T01:38:53.769047+0000","last_fullsized":"2024-04-26T01:38:53.769047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:04:35.093847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.766562+0000","last_change":"2024-04-26T01:36:54.273013+0000","last_active":"2024-04-26T01:38:53.766562+0000","last_peered":"2024-04-26T01:38:53.766562+0000","last_clean":"2024-04-26T01:38:53.766562+0000","last_became_active":"2024-04-26T01:36:54.272666+0000","last_became_peered":"2024-04-26T01:36:54.272666+0000","last_unstale":"2024-04-26T01:38:53.766562+0000","last_undegraded":"2024-04-26T01:38:53.766562+0000","last_fullsized":"2024-04-26T01:38:53.766562+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:45.203185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.771970+0000","last_change":"2024-04-26T01:36:54.289427+0000","last_active":"2024-04-26T01:38:53.771970+0000","last_peered":"2024-04-26T01:38:53.771970+0000","last_clean":"2024-04-26T01:38:53.771970+0000","last_became_active":"2024-04-26T01:36:54.289090+0000","last_became_peered":"2024-04-26T01:36:54.289090+0000","last_unstale":"2024-04-26T01:38:53.771970+0000","last_undegraded":"2024-04-26T01:38:53.771970+0000","last_fullsized":"2024-04-26T01:38:53.771970+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:18:18.544777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.789535+0000","last_change":"2024-04-26T01:36:54.294308+0000","last_active":"2024-04-26T01:38:53.789535+0000","last_peered":"2024-04-26T01:38:53.789535+0000","last_clean":"2024-04-26T01:38:53.789535+0000","last_became_active":"2024-04-26T01:36:54.291910+0000","last_became_peered":"2024-04-26T01:36:54.291910+0000","last_unstale":"2024-04-26T01:38:53.789535+0000","last_undegraded":"2024-04-26T01:38:53.789535+0000","last_fullsized":"2024-04-26T01:38:53.789535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:35:16.683265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.781583+0000","last_change":"2024-04-26T01:36:54.285479+0000","last_active":"2024-04-26T01:38:53.781583+0000","last_peered":"2024-04-26T01:38:53.781583+0000","last_clean":"2024-04-26T01:38:53.781583+0000","last_became_active":"2024-04-26T01:36:54.284139+0000","last_became_peered":"2024-04-26T01:36:54.284139+0000","last_unstale":"2024-04-26T01:38:53.781583+0000","last_undegraded":"2024-04-26T01:38:53.781583+0000","last_fullsized":"2024-04-26T01:38:53.781583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:55.506496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.762770+0000","last_change":"2024-04-26T01:36:54.270567+0000","last_active":"2024-04-26T01:38:53.762770+0000","last_peered":"2024-04-26T01:38:53.762770+0000","last_clean":"2024-04-26T01:38:53.762770+0000","last_became_active":"2024-04-26T01:36:54.270218+0000","last_became_peered":"2024-04-26T01:36:54.270218+0000","last_unstale":"2024-04-26T01:38:53.762770+0000","last_undegraded":"2024-04-26T01:38:53.762770+0000","last_fullsized":"2024-04-26T01:38:53.762770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:50:43.685086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.760244+0000","last_change":"2024-04-26T01:36:54.269746+0000","last_active":"2024-04-26T01:38:53.760244+0000","last_peered":"2024-04-26T01:38:53.760244+0000","last_clean":"2024-04-26T01:38:53.760244+0000","last_became_active":"2024-04-26T01:36:54.269143+0000","last_became_peered":"2024-04-26T01:36:54.269143+0000","last_unstale":"2024-04-26T01:38:53.760244+0000","last_undegraded":"2024-04-26T01:38:53.760244+0000","last_fullsized":"2024-04-26T01:38:53.760244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:07:02.911645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.781470+0000","last_change":"2024-04-26T01:36:54.279194+0000","last_active":"2024-04-26T01:38:53.781470+0000","last_peered":"2024-04-26T01:38:53.781470+0000","last_clean":"2024-04-26T01:38:53.781470+0000","last_became_active":"2024-04-26T01:36:54.278709+0000","last_became_peered":"2024-04-26T01:36:54.278709+0000","last_unstale":"2024-04-26T01:38:53.781470+0000","last_undegraded":"2024-04-26T01:38:53.781470+0000","last_fullsized":"2024-04-26T01:38:53.781470+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:19.604884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786135+0000","last_change":"2024-04-26T01:36:54.277540+0000","last_active":"2024-04-26T01:38:53.786135+0000","last_peered":"2024-04-26T01:38:53.786135+0000","last_clean":"2024-04-26T01:38:53.786135+0000","last_became_active":"2024-04-26T01:36:54.277193+0000","last_became_peered":"2024-04-26T01:36:54.277193+0000","last_unstale":"2024-04-26T01:38:53.786135+0000","last_undegraded":"2024-04-26T01:38:53.786135+0000","last_fullsized":"2024-04-26T01:38:53.786135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:34.079094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786760+0000","last_change":"2024-04-26T01:36:54.294593+0000","last_active":"2024-04-26T01:38:53.786760+0000","last_peered":"2024-04-26T01:38:53.786760+0000","last_clean":"2024-04-26T01:38:53.786760+0000","last_became_active":"2024-04-26T01:36:54.291894+0000","last_became_peered":"2024-04-26T01:36:54.291894+0000","last_unstale":"2024-04-26T01:38:53.786760+0000","last_undegraded":"2024-04-26T01:38:53.786760+0000","last_fullsized":"2024-04-26T01:38:53.786760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:16.715108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786355+0000","last_change":"2024-04-26T01:36:54.292433+0000","last_active":"2024-04-26T01:38:53.786355+0000","last_peered":"2024-04-26T01:38:53.786355+0000","last_clean":"2024-04-26T01:38:53.786355+0000","last_became_active":"2024-04-26T01:36:54.291445+0000","last_became_peered":"2024-04-26T01:36:54.291445+0000","last_unstale":"2024-04-26T01:38:53.786355+0000","last_undegraded":"2024-04-26T01:38:53.786355+0000","last_fullsized":"2024-04-26T01:38:53.786355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:49.081871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.791176+0000","last_change":"2024-04-26T01:36:54.291892+0000","last_active":"2024-04-26T01:38:53.791176+0000","last_peered":"2024-04-26T01:38:53.791176+0000","last_clean":"2024-04-26T01:38:53.791176+0000","last_became_active":"2024-04-26T01:36:54.291549+0000","last_became_peered":"2024-04-26T01:36:54.291549+0000","last_unstale":"2024-04-26T01:38:53.791176+0000","last_undegraded":"2024-04-26T01:38:53.791176+0000","last_fullsized":"2024-04-26T01:38:53.791176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:09:22.954651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.774026+0000","last_change":"2024-04-26T01:37:25.498138+0000","last_active":"2024-04-26T01:38:53.774026+0000","last_peered":"2024-04-26T01:38:53.774026+0000","last_clean":"2024-04-26T01:38:53.774026+0000","last_became_active":"2024-04-26T01:37:25.497681+0000","last_became_peered":"2024-04-26T01:37:25.497681+0000","last_unstale":"2024-04-26T01:38:53.774026+0000","last_undegraded":"2024-04-26T01:38:53.774026+0000","last_fullsized":"2024-04-26T01:38:53.774026+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:03:25.016623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282972,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70041,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":743,"num_read_kb":446,"num_write":175,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":210123,"data_compressed":5484,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383728,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":479903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645364,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1130496,"data_stored":1137261,"data_compressed":37816,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808891,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":480117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070528,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":480015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332166,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1072108,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495692,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":479506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":546352,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":480777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":545814,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182492,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":480055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757331,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071792,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67053,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66515,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":67477,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:39:21.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:39:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:21.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:21 smithi028 ceph-mon[28247]: pgmap v408: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:21.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:21 smithi028 ceph-mon[28247]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:39:21.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:21 smithi169 ceph-mon[32851]: pgmap v408: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:21.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:21 smithi169 ceph-mon[32851]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:39:21.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:21 smithi053 ceph-mon[32593]: pgmap v408: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:21.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:21 smithi053 ceph-mon[32593]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:39:23.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:23 smithi028 ceph-mon[28247]: pgmap v409: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:23.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:23 smithi169 ceph-mon[32851]: pgmap v409: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 -- 172.21.15.28:0/2682191119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b0006b0f0 msgr2=0x7f2b0010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 --2- 172.21.15.28:0/2682191119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b0006b0f0 0x7f2b0010f900 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2af800b0a0 tx=0x7f2af800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:39:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 -- 172.21.15.28:0/2682191119 shutdown_connections 2024-04-26T01:39:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 --2- 172.21.15.28:0/2682191119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b0006b0f0 0x7f2b0010f900 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 --2- 172.21.15.28:0/2682191119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b00108a30 0x7f2b00108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 --2- 172.21.15.28:0/2682191119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b0006a750 0x7f2b0006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 -- 172.21.15.28:0/2682191119 >> 172.21.15.28:0/2682191119 conn(0x7f2b000788f0 msgr2=0x7f2b00078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 -- 172.21.15.28:0/2682191119 shutdown_connections 2024-04-26T01:39:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.963+0000 7f2b07800700 1 -- 172.21.15.28:0/2682191119 wait complete. 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.964+0000 7f2b07800700 1 Processor -- start 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b07800700 1 -- start start 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b07800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b0006a750 0x7f2b0019b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b07800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b0006b0f0 0x7f2b0019b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b07800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b00108a30 0x7f2b0019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b07800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00112c60 con 0x7f2b00108a30 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b07800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00112ae0 con 0x7f2b0006a750 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b07800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00112de0 con 0x7f2b0006b0f0 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b05800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b00108a30 0x7f2b0019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b04e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b0006a750 0x7f2b0019b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b05800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b00108a30 0x7f2b0019fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58564/0 (socket says 172.21.15.28:58564) 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2b05800700 1 -- 172.21.15.28:0/272772899 learned_addr learned my addr 172.21.15.28:0/272772899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.965+0000 7f2affe00700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b0006b0f0 0x7f2b0019b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2b05800700 1 -- 172.21.15.28:0/272772899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b0006b0f0 msgr2=0x7f2b0019b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2b05800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b0006b0f0 0x7f2b0019b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2b05800700 1 -- 172.21.15.28:0/272772899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b0006a750 msgr2=0x7f2b0019b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2b05800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b0006a750 0x7f2b0019b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2b05800700 1 -- 172.21.15.28:0/272772899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b001a03c0 con 0x7f2b00108a30 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2b04e00700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b0006a750 0x7f2b0019b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2b05800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b00108a30 0x7f2b0019fc80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2af800b070 tx=0x7f2af8009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2afd600700 1 -- 172.21.15.28:0/272772899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8021070 con 0x7f2b00108a30 2024-04-26T01:39:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2afd600700 1 -- 172.21.15.28:0/272772899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2af8004e40 con 0x7f2b00108a30 2024-04-26T01:39:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b001a0650 con 0x7f2b00108a30 2024-04-26T01:39:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.966+0000 7f2afd600700 1 -- 172.21.15.28:0/272772899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8004630 con 0x7f2b00108a30 2024-04-26T01:39:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.967+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b001a7f30 con 0x7f2b00108a30 2024-04-26T01:39:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.968+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b00104170 con 0x7f2b00108a30 2024-04-26T01:39:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.970+0000 7f2afd600700 1 -- 172.21.15.28:0/272772899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2af8007980 con 0x7f2b00108a30 2024-04-26T01:39:23.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:23 smithi053 ceph-mon[32593]: pgmap v409: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.971+0000 7f2afd600700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2ae80a3c40 0x7f2ae80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.971+0000 7f2afd600700 1 -- 172.21.15.28:0/272772899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f2af80d1200 con 0x7f2b00108a30 2024-04-26T01:39:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.972+0000 7f2b04e00700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2ae80a3c40 0x7f2ae80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.972+0000 7f2b04e00700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2ae80a3c40 0x7f2ae80a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009fd0 tx=0x7f2af4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:23.974+0000 7f2afd600700 1 -- 172.21.15.28:0/272772899 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2af809dc20 con 0x7f2b00108a30 2024-04-26T01:39:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.199+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b0019c710 con 0x7f2ae80a3c40 2024-04-26T01:39:24.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.222+0000 7f2afd600700 1 -- 172.21.15.28:0/272772899 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259061 (secure 0 0 0) 0x7f2b0019c710 con 0x7f2ae80a3c40 2024-04-26T01:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:39:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.227+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2ae80a3c40 msgr2=0x7f2ae80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.227+0000 7f2b07800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2ae80a3c40 0x7f2ae80a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009fd0 tx=0x7f2af4009260 comp rx=0 tx=0).stop 2024-04-26T01:39:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.227+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b00108a30 msgr2=0x7f2b0019fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.227+0000 7f2b07800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b00108a30 0x7f2b0019fc80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2af800b070 tx=0x7f2af8009c90 comp rx=0 tx=0).stop 2024-04-26T01:39:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.228+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 shutdown_connections 2024-04-26T01:39:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.228+0000 7f2b07800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f2ae80a3c40 0x7f2ae80a6100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.228+0000 7f2b07800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b00108a30 0x7f2b0019fc80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.228+0000 7f2b07800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b0006a750 0x7f2b0019b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.228+0000 7f2b07800700 1 --2- 172.21.15.28:0/272772899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b0006b0f0 0x7f2b0019b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.228+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 >> 172.21.15.28:0/272772899 conn(0x7f2b000788f0 msgr2=0x7f2b0010dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.228+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 shutdown_connections 2024-04-26T01:39:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:24.228+0000 7f2b07800700 1 -- 172.21.15.28:0/272772899 wait complete. 2024-04-26T01:39:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-26T01:39:25.235 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-04-26T01:39:23.821897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660643,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":904,"num_read_kb":575,"num_write":358,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181492,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377792,"kb_used_data":12000,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564032,"statfs":{"total":1151940427776,"available":1151553568768,"internally_reserved":0,"allocated":12288000,"data_stored":7733517,"data_compressed":113448,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":374453564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004768"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777218+0000","last_change":"2024-04-26T01:36:54.288460+0000","last_active":"2024-04-26T01:38:53.777218+0000","last_peered":"2024-04-26T01:38:53.777218+0000","last_clean":"2024-04-26T01:38:53.777218+0000","last_became_active":"2024-04-26T01:36:54.288033+0000","last_became_peered":"2024-04-26T01:36:54.288033+0000","last_unstale":"2024-04-26T01:38:53.777218+0000","last_undegraded":"2024-04-26T01:38:53.777218+0000","last_fullsized":"2024-04-26T01:38:53.777218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:33.737622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.778284+0000","last_change":"2024-04-26T01:36:54.287367+0000","last_active":"2024-04-26T01:38:53.778284+0000","last_peered":"2024-04-26T01:38:53.778284+0000","last_clean":"2024-04-26T01:38:53.778284+0000","last_became_active":"2024-04-26T01:36:54.287024+0000","last_became_peered":"2024-04-26T01:36:54.287024+0000","last_unstale":"2024-04-26T01:38:53.778284+0000","last_undegraded":"2024-04-26T01:38:53.778284+0000","last_fullsized":"2024-04-26T01:38:53.778284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:31.014627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780136+0000","last_change":"2024-04-26T01:36:54.288433+0000","last_active":"2024-04-26T01:38:53.780136+0000","last_peered":"2024-04-26T01:38:53.780136+0000","last_clean":"2024-04-26T01:38:53.780136+0000","last_became_active":"2024-04-26T01:36:54.288002+0000","last_became_peered":"2024-04-26T01:36:54.288002+0000","last_unstale":"2024-04-26T01:38:53.780136+0000","last_undegraded":"2024-04-26T01:38:53.780136+0000","last_fullsized":"2024-04-26T01:38:53.780136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:37.885179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.759001+0000","last_change":"2024-04-26T01:36:54.273857+0000","last_active":"2024-04-26T01:38:53.759001+0000","last_peered":"2024-04-26T01:38:53.759001+0000","last_clean":"2024-04-26T01:38:53.759001+0000","last_became_active":"2024-04-26T01:36:54.272376+0000","last_became_peered":"2024-04-26T01:36:54.272376+0000","last_unstale":"2024-04-26T01:38:53.759001+0000","last_undegraded":"2024-04-26T01:38:53.759001+0000","last_fullsized":"2024-04-26T01:38:53.759001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:30.110044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786123+0000","last_change":"2024-04-26T01:36:54.288606+0000","last_active":"2024-04-26T01:38:53.786123+0000","last_peered":"2024-04-26T01:38:53.786123+0000","last_clean":"2024-04-26T01:38:53.786123+0000","last_became_active":"2024-04-26T01:36:54.288247+0000","last_became_peered":"2024-04-26T01:36:54.288247+0000","last_unstale":"2024-04-26T01:38:53.786123+0000","last_undegraded":"2024-04-26T01:38:53.786123+0000","last_fullsized":"2024-04-26T01:38:53.786123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:47:56.944164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777137+0000","last_change":"2024-04-26T01:36:54.277701+0000","last_active":"2024-04-26T01:38:53.777137+0000","last_peered":"2024-04-26T01:38:53.777137+0000","last_clean":"2024-04-26T01:38:53.777137+0000","last_became_active":"2024-04-26T01:36:54.277306+0000","last_became_peered":"2024-04-26T01:36:54.277306+0000","last_unstale":"2024-04-26T01:38:53.777137+0000","last_undegraded":"2024-04-26T01:38:53.777137+0000","last_fullsized":"2024-04-26T01:38:53.777137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:42:05.058525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.778546+0000","last_change":"2024-04-26T01:36:54.274090+0000","last_active":"2024-04-26T01:38:53.778546+0000","last_peered":"2024-04-26T01:38:53.778546+0000","last_clean":"2024-04-26T01:38:53.778546+0000","last_became_active":"2024-04-26T01:36:54.273700+0000","last_became_peered":"2024-04-26T01:36:54.273700+0000","last_unstale":"2024-04-26T01:38:53.778546+0000","last_undegraded":"2024-04-26T01:38:53.778546+0000","last_fullsized":"2024-04-26T01:38:53.778546+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:32.207534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.779109+0000","last_change":"2024-04-26T01:37:25.499759+0000","last_active":"2024-04-26T01:38:53.779109+0000","last_peered":"2024-04-26T01:38:53.779109+0000","last_clean":"2024-04-26T01:38:53.779109+0000","last_became_active":"2024-04-26T01:37:25.499317+0000","last_became_peered":"2024-04-26T01:37:25.499317+0000","last_unstale":"2024-04-26T01:38:53.779109+0000","last_undegraded":"2024-04-26T01:38:53.779109+0000","last_fullsized":"2024-04-26T01:38:53.779109+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:39.972315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.776288+0000","last_change":"2024-04-26T01:36:54.277457+0000","last_active":"2024-04-26T01:38:53.776288+0000","last_peered":"2024-04-26T01:38:53.776288+0000","last_clean":"2024-04-26T01:38:53.776288+0000","last_became_active":"2024-04-26T01:36:54.277118+0000","last_became_peered":"2024-04-26T01:36:54.277118+0000","last_unstale":"2024-04-26T01:38:53.776288+0000","last_undegraded":"2024-04-26T01:38:53.776288+0000","last_fullsized":"2024-04-26T01:38:53.776288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:05.628027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.764281+0000","last_change":"2024-04-26T01:36:54.278494+0000","last_active":"2024-04-26T01:38:53.764281+0000","last_peered":"2024-04-26T01:38:53.764281+0000","last_clean":"2024-04-26T01:38:53.764281+0000","last_became_active":"2024-04-26T01:36:54.278188+0000","last_became_peered":"2024-04-26T01:36:54.278188+0000","last_unstale":"2024-04-26T01:38:53.764281+0000","last_undegraded":"2024-04-26T01:38:53.764281+0000","last_fullsized":"2024-04-26T01:38:53.764281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:59.150771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.772253+0000","last_change":"2024-04-26T01:36:54.277424+0000","last_active":"2024-04-26T01:38:53.772253+0000","last_peered":"2024-04-26T01:38:53.772253+0000","last_clean":"2024-04-26T01:38:53.772253+0000","last_became_active":"2024-04-26T01:36:54.275752+0000","last_became_peered":"2024-04-26T01:36:54.275752+0000","last_unstale":"2024-04-26T01:38:53.772253+0000","last_undegraded":"2024-04-26T01:38:53.772253+0000","last_fullsized":"2024-04-26T01:38:53.772253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:22.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.776361+0000","last_change":"2024-04-26T01:37:25.502448+0000","last_active":"2024-04-26T01:38:53.776361+0000","last_peered":"2024-04-26T01:38:53.776361+0000","last_clean":"2024-04-26T01:38:53.776361+0000","last_became_active":"2024-04-26T01:37:25.501161+0000","last_became_peered":"2024-04-26T01:37:25.501161+0000","last_unstale":"2024-04-26T01:38:53.776361+0000","last_undegraded":"2024-04-26T01:38:53.776361+0000","last_fullsized":"2024-04-26T01:38:53.776361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:41:40.443780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.791351+0000","last_change":"2024-04-26T01:36:54.268721+0000","last_active":"2024-04-26T01:38:53.791351+0000","last_peered":"2024-04-26T01:38:53.791351+0000","last_clean":"2024-04-26T01:38:53.791351+0000","last_became_active":"2024-04-26T01:36:54.268007+0000","last_became_peered":"2024-04-26T01:36:54.268007+0000","last_unstale":"2024-04-26T01:38:53.791351+0000","last_undegraded":"2024-04-26T01:38:53.791351+0000","last_fullsized":"2024-04-26T01:38:53.791351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:55.120130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777956+0000","last_change":"2024-04-26T01:36:54.286595+0000","last_active":"2024-04-26T01:38:53.777956+0000","last_peered":"2024-04-26T01:38:53.777956+0000","last_clean":"2024-04-26T01:38:53.777956+0000","last_became_active":"2024-04-26T01:36:54.286236+0000","last_became_peered":"2024-04-26T01:36:54.286236+0000","last_unstale":"2024-04-26T01:38:53.777956+0000","last_undegraded":"2024-04-26T01:38:53.777956+0000","last_fullsized":"2024-04-26T01:38:53.777956+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:59.443467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.775994+0000","last_change":"2024-04-26T01:36:54.293254+0000","last_active":"2024-04-26T01:38:53.775994+0000","last_peered":"2024-04-26T01:38:53.775994+0000","last_clean":"2024-04-26T01:38:53.775994+0000","last_became_active":"2024-04-26T01:36:54.292853+0000","last_became_peered":"2024-04-26T01:36:54.292853+0000","last_unstale":"2024-04-26T01:38:53.775994+0000","last_undegraded":"2024-04-26T01:38:53.775994+0000","last_fullsized":"2024-04-26T01:38:53.775994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:19.844006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777829+0000","last_change":"2024-04-26T01:36:54.285346+0000","last_active":"2024-04-26T01:38:53.777829+0000","last_peered":"2024-04-26T01:38:53.777829+0000","last_clean":"2024-04-26T01:38:53.777829+0000","last_became_active":"2024-04-26T01:36:54.284949+0000","last_became_peered":"2024-04-26T01:36:54.284949+0000","last_unstale":"2024-04-26T01:38:53.777829+0000","last_undegraded":"2024-04-26T01:38:53.777829+0000","last_fullsized":"2024-04-26T01:38:53.777829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:33:29.276240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777190+0000","last_change":"2024-04-26T01:36:54.283161+0000","last_active":"2024-04-26T01:38:53.777190+0000","last_peered":"2024-04-26T01:38:53.777190+0000","last_clean":"2024-04-26T01:38:53.777190+0000","last_became_active":"2024-04-26T01:36:54.282908+0000","last_became_peered":"2024-04-26T01:36:54.282908+0000","last_unstale":"2024-04-26T01:38:53.777190+0000","last_undegraded":"2024-04-26T01:38:53.777190+0000","last_fullsized":"2024-04-26T01:38:53.777190+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:26.503964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777767+0000","last_change":"2024-04-26T01:36:54.282516+0000","last_active":"2024-04-26T01:38:53.777767+0000","last_peered":"2024-04-26T01:38:53.777767+0000","last_clean":"2024-04-26T01:38:53.777767+0000","last_became_active":"2024-04-26T01:36:54.278634+0000","last_became_peered":"2024-04-26T01:36:54.278634+0000","last_unstale":"2024-04-26T01:38:53.777767+0000","last_undegraded":"2024-04-26T01:38:53.777767+0000","last_fullsized":"2024-04-26T01:38:53.777767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:10:38.275720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780482+0000","last_change":"2024-04-26T01:36:54.289502+0000","last_active":"2024-04-26T01:38:53.780482+0000","last_peered":"2024-04-26T01:38:53.780482+0000","last_clean":"2024-04-26T01:38:53.780482+0000","last_became_active":"2024-04-26T01:36:54.289079+0000","last_became_peered":"2024-04-26T01:36:54.289079+0000","last_unstale":"2024-04-26T01:38:53.780482+0000","last_undegraded":"2024-04-26T01:38:53.780482+0000","last_fullsized":"2024-04-26T01:38:53.780482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:51.412032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.498085+0000","last_change":"2024-04-26T01:36:50.243164+0000","last_active":"2024-04-26T01:37:25.498085+0000","last_peered":"2024-04-26T01:37:25.498085+0000","last_clean":"2024-04-26T01:37:25.498085+0000","last_became_active":"2024-04-26T01:36:50.242508+0000","last_became_peered":"2024-04-26T01:36:50.242508+0000","last_unstale":"2024-04-26T01:37:25.498085+0000","last_undegraded":"2024-04-26T01:37:25.498085+0000","last_fullsized":"2024-04-26T01:37:25.498085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:06:01.662698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.783619+0000","last_change":"2024-04-26T01:36:54.279809+0000","last_active":"2024-04-26T01:38:53.783619+0000","last_peered":"2024-04-26T01:38:53.783619+0000","last_clean":"2024-04-26T01:38:53.783619+0000","last_became_active":"2024-04-26T01:36:54.279509+0000","last_became_peered":"2024-04-26T01:36:54.279509+0000","last_unstale":"2024-04-26T01:38:53.783619+0000","last_undegraded":"2024-04-26T01:38:53.783619+0000","last_fullsized":"2024-04-26T01:38:53.783619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:54:26.734136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.515168+0000","last_change":"2024-04-26T01:36:50.261942+0000","last_active":"2024-04-26T01:37:25.515168+0000","last_peered":"2024-04-26T01:37:25.515168+0000","last_clean":"2024-04-26T01:37:25.515168+0000","last_became_active":"2024-04-26T01:36:50.261564+0000","last_became_peered":"2024-04-26T01:36:50.261564+0000","last_unstale":"2024-04-26T01:37:25.515168+0000","last_undegraded":"2024-04-26T01:37:25.515168+0000","last_fullsized":"2024-04-26T01:37:25.515168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:25.102557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780874+0000","last_change":"2024-04-26T01:36:54.288011+0000","last_active":"2024-04-26T01:38:53.780874+0000","last_peered":"2024-04-26T01:38:53.780874+0000","last_clean":"2024-04-26T01:38:53.780874+0000","last_became_active":"2024-04-26T01:36:54.286853+0000","last_became_peered":"2024-04-26T01:36:54.286853+0000","last_unstale":"2024-04-26T01:38:53.780874+0000","last_undegraded":"2024-04-26T01:38:53.780874+0000","last_fullsized":"2024-04-26T01:38:53.780874+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:53.422470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495891+0000","last_change":"2024-04-26T01:36:50.235291+0000","last_active":"2024-04-26T01:37:25.495891+0000","last_peered":"2024-04-26T01:37:25.495891+0000","last_clean":"2024-04-26T01:37:25.495891+0000","last_became_active":"2024-04-26T01:36:50.234939+0000","last_became_peered":"2024-04-26T01:36:50.234939+0000","last_unstale":"2024-04-26T01:37:25.495891+0000","last_undegraded":"2024-04-26T01:37:25.495891+0000","last_fullsized":"2024-04-26T01:37:25.495891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:13.048185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785426+0000","last_change":"2024-04-26T01:36:54.282731+0000","last_active":"2024-04-26T01:38:53.785426+0000","last_peered":"2024-04-26T01:38:53.785426+0000","last_clean":"2024-04-26T01:38:53.785426+0000","last_became_active":"2024-04-26T01:36:54.282384+0000","last_became_peered":"2024-04-26T01:36:54.282384+0000","last_unstale":"2024-04-26T01:38:53.785426+0000","last_undegraded":"2024-04-26T01:38:53.785426+0000","last_fullsized":"2024-04-26T01:38:53.785426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:09.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495653+0000","last_change":"2024-04-26T01:36:50.233581+0000","last_active":"2024-04-26T01:37:25.495653+0000","last_peered":"2024-04-26T01:37:25.495653+0000","last_clean":"2024-04-26T01:37:25.495653+0000","last_became_active":"2024-04-26T01:36:50.233210+0000","last_became_peered":"2024-04-26T01:36:50.233210+0000","last_unstale":"2024-04-26T01:37:25.495653+0000","last_undegraded":"2024-04-26T01:37:25.495653+0000","last_fullsized":"2024-04-26T01:37:25.495653+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:01.223954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.772477+0000","last_change":"2024-04-26T01:36:54.274657+0000","last_active":"2024-04-26T01:38:53.772477+0000","last_peered":"2024-04-26T01:38:53.772477+0000","last_clean":"2024-04-26T01:38:53.772477+0000","last_became_active":"2024-04-26T01:36:54.274333+0000","last_became_peered":"2024-04-26T01:36:54.274333+0000","last_unstale":"2024-04-26T01:38:53.772477+0000","last_undegraded":"2024-04-26T01:38:53.772477+0000","last_fullsized":"2024-04-26T01:38:53.772477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:37.243307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:21.750316+0000","last_change":"2024-04-26T01:37:25.502311+0000","last_active":"2024-04-26T01:39:21.750316+0000","last_peered":"2024-04-26T01:39:21.750316+0000","last_clean":"2024-04-26T01:39:21.750316+0000","last_became_active":"2024-04-26T01:37:25.493259+0000","last_became_peered":"2024-04-26T01:37:25.493259+0000","last_unstale":"2024-04-26T01:39:21.750316+0000","last_undegraded":"2024-04-26T01:39:21.750316+0000","last_fullsized":"2024-04-26T01:39:21.750316+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:46:49.224905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.773356+0000","last_change":"2024-04-26T01:36:54.275116+0000","last_active":"2024-04-26T01:38:53.773356+0000","last_peered":"2024-04-26T01:38:53.773356+0000","last_clean":"2024-04-26T01:38:53.773356+0000","last_became_active":"2024-04-26T01:36:54.274815+0000","last_became_peered":"2024-04-26T01:36:54.274815+0000","last_unstale":"2024-04-26T01:38:53.773356+0000","last_undegraded":"2024-04-26T01:38:53.773356+0000","last_fullsized":"2024-04-26T01:38:53.773356+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:04.427670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.505832+0000","last_change":"2024-04-26T01:37:25.505327+0000","last_active":"2024-04-26T01:37:25.505832+0000","last_peered":"2024-04-26T01:37:25.505832+0000","last_clean":"2024-04-26T01:37:25.505832+0000","last_became_active":"2024-04-26T01:37:25.497881+0000","last_became_peered":"2024-04-26T01:37:25.497881+0000","last_unstale":"2024-04-26T01:37:25.505832+0000","last_undegraded":"2024-04-26T01:37:25.505832+0000","last_fullsized":"2024-04-26T01:37:25.505832+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:09.181372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786730+0000","last_change":"2024-04-26T01:36:54.276415+0000","last_active":"2024-04-26T01:38:53.786730+0000","last_peered":"2024-04-26T01:38:53.786730+0000","last_clean":"2024-04-26T01:38:53.786730+0000","last_became_active":"2024-04-26T01:36:54.276080+0000","last_became_peered":"2024-04-26T01:36:54.276080+0000","last_unstale":"2024-04-26T01:38:53.786730+0000","last_undegraded":"2024-04-26T01:38:53.786730+0000","last_fullsized":"2024-04-26T01:38:53.786730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:08:17.171620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491996+0000","last_change":"2024-04-26T01:36:50.241552+0000","last_active":"2024-04-26T01:37:25.491996+0000","last_peered":"2024-04-26T01:37:25.491996+0000","last_clean":"2024-04-26T01:37:25.491996+0000","last_became_active":"2024-04-26T01:36:50.241022+0000","last_became_peered":"2024-04-26T01:36:50.241022+0000","last_unstale":"2024-04-26T01:37:25.491996+0000","last_undegraded":"2024-04-26T01:37:25.491996+0000","last_fullsized":"2024-04-26T01:37:25.491996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:55.503831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.788030+0000","last_change":"2024-04-26T01:36:54.276641+0000","last_active":"2024-04-26T01:38:53.788030+0000","last_peered":"2024-04-26T01:38:53.788030+0000","last_clean":"2024-04-26T01:38:53.788030+0000","last_became_active":"2024-04-26T01:36:54.276309+0000","last_became_peered":"2024-04-26T01:36:54.276309+0000","last_unstale":"2024-04-26T01:38:53.788030+0000","last_undegraded":"2024-04-26T01:38:53.788030+0000","last_fullsized":"2024-04-26T01:38:53.788030+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:53.725584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489011+0000","last_change":"2024-04-26T01:36:50.235504+0000","last_active":"2024-04-26T01:37:25.489011+0000","last_peered":"2024-04-26T01:37:25.489011+0000","last_clean":"2024-04-26T01:37:25.489011+0000","last_became_active":"2024-04-26T01:36:50.235161+0000","last_became_peered":"2024-04-26T01:36:50.235161+0000","last_unstale":"2024-04-26T01:37:25.489011+0000","last_undegraded":"2024-04-26T01:37:25.489011+0000","last_fullsized":"2024-04-26T01:37:25.489011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:05.951521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786346+0000","last_change":"2024-04-26T01:36:54.298236+0000","last_active":"2024-04-26T01:38:53.786346+0000","last_peered":"2024-04-26T01:38:53.786346+0000","last_clean":"2024-04-26T01:38:53.786346+0000","last_became_active":"2024-04-26T01:36:54.297850+0000","last_became_peered":"2024-04-26T01:36:54.297850+0000","last_unstale":"2024-04-26T01:38:53.786346+0000","last_undegraded":"2024-04-26T01:38:53.786346+0000","last_fullsized":"2024-04-26T01:38:53.786346+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:35:35.244502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:18.742600+0000","last_change":"2024-04-26T01:36:50.241418+0000","last_active":"2024-04-26T01:39:18.742600+0000","last_peered":"2024-04-26T01:39:18.742600+0000","last_clean":"2024-04-26T01:39:18.742600+0000","last_became_active":"2024-04-26T01:36:50.241044+0000","last_became_peered":"2024-04-26T01:36:50.241044+0000","last_unstale":"2024-04-26T01:39:18.742600+0000","last_undegraded":"2024-04-26T01:39:18.742600+0000","last_fullsized":"2024-04-26T01:39:18.742600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:13.317953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.776030+0000","last_change":"2024-04-26T01:36:54.284410+0000","last_active":"2024-04-26T01:38:53.776030+0000","last_peered":"2024-04-26T01:38:53.776030+0000","last_clean":"2024-04-26T01:38:53.776030+0000","last_became_active":"2024-04-26T01:36:54.283629+0000","last_became_peered":"2024-04-26T01:36:54.283629+0000","last_unstale":"2024-04-26T01:38:53.776030+0000","last_undegraded":"2024-04-26T01:38:53.776030+0000","last_fullsized":"2024-04-26T01:38:53.776030+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:15.380006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.500348+0000","last_change":"2024-04-26T01:37:25.499874+0000","last_active":"2024-04-26T01:37:25.500348+0000","last_peered":"2024-04-26T01:37:25.500348+0000","last_clean":"2024-04-26T01:37:25.500348+0000","last_became_active":"2024-04-26T01:37:25.494878+0000","last_became_peered":"2024-04-26T01:37:25.494878+0000","last_unstale":"2024-04-26T01:37:25.500348+0000","last_undegraded":"2024-04-26T01:37:25.500348+0000","last_fullsized":"2024-04-26T01:37:25.500348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:48:12.678331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.777571+0000","last_change":"2024-04-26T01:36:54.285118+0000","last_active":"2024-04-26T01:38:53.777571+0000","last_peered":"2024-04-26T01:38:53.777571+0000","last_clean":"2024-04-26T01:38:53.777571+0000","last_became_active":"2024-04-26T01:36:54.284554+0000","last_became_peered":"2024-04-26T01:36:54.284554+0000","last_unstale":"2024-04-26T01:38:53.777571+0000","last_undegraded":"2024-04-26T01:38:53.777571+0000","last_fullsized":"2024-04-26T01:38:53.777571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:13.721684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489735+0000","last_change":"2024-04-26T01:36:50.262451+0000","last_active":"2024-04-26T01:37:25.489735+0000","last_peered":"2024-04-26T01:37:25.489735+0000","last_clean":"2024-04-26T01:37:25.489735+0000","last_became_active":"2024-04-26T01:36:50.262056+0000","last_became_peered":"2024-04-26T01:36:50.262056+0000","last_unstale":"2024-04-26T01:37:25.489735+0000","last_undegraded":"2024-04-26T01:37:25.489735+0000","last_fullsized":"2024-04-26T01:37:25.489735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:50.813675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.774764+0000","last_change":"2024-04-26T01:36:54.283975+0000","last_active":"2024-04-26T01:38:53.774764+0000","last_peered":"2024-04-26T01:38:53.774764+0000","last_clean":"2024-04-26T01:38:53.774764+0000","last_became_active":"2024-04-26T01:36:54.283564+0000","last_became_peered":"2024-04-26T01:36:54.283564+0000","last_unstale":"2024-04-26T01:38:53.774764+0000","last_undegraded":"2024-04-26T01:38:53.774764+0000","last_fullsized":"2024-04-26T01:38:53.774764+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:53.881763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":318,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:19.635280+0000","last_change":"2024-04-26T01:36:50.238839+0000","last_active":"2024-04-26T01:39:19.635280+0000","last_peered":"2024-04-26T01:39:19.635280+0000","last_clean":"2024-04-26T01:39:19.635280+0000","last_became_active":"2024-04-26T01:36:50.238452+0000","last_became_peered":"2024-04-26T01:36:50.238452+0000","last_unstale":"2024-04-26T01:39:19.635280+0000","last_undegraded":"2024-04-26T01:39:19.635280+0000","last_fullsized":"2024-04-26T01:39:19.635280+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:05:40.426825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65899,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":256,"num_read_kb":192,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.768491+0000","last_change":"2024-04-26T01:36:54.271446+0000","last_active":"2024-04-26T01:38:53.768491+0000","last_peered":"2024-04-26T01:38:53.768491+0000","last_clean":"2024-04-26T01:38:53.768491+0000","last_became_active":"2024-04-26T01:36:54.271122+0000","last_became_peered":"2024-04-26T01:36:54.271122+0000","last_unstale":"2024-04-26T01:38:53.768491+0000","last_undegraded":"2024-04-26T01:38:53.768491+0000","last_fullsized":"2024-04-26T01:38:53.768491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:28:24.275481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491729+0000","last_change":"2024-04-26T01:36:50.237726+0000","last_active":"2024-04-26T01:37:25.491729+0000","last_peered":"2024-04-26T01:37:25.491729+0000","last_clean":"2024-04-26T01:37:25.491729+0000","last_became_active":"2024-04-26T01:36:50.237215+0000","last_became_peered":"2024-04-26T01:36:50.237215+0000","last_unstale":"2024-04-26T01:37:25.491729+0000","last_undegraded":"2024-04-26T01:37:25.491729+0000","last_fullsized":"2024-04-26T01:37:25.491729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:41.781876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.779276+0000","last_change":"2024-04-26T01:36:54.290616+0000","last_active":"2024-04-26T01:38:53.779276+0000","last_peered":"2024-04-26T01:38:53.779276+0000","last_clean":"2024-04-26T01:38:53.779276+0000","last_became_active":"2024-04-26T01:36:54.290293+0000","last_became_peered":"2024-04-26T01:36:54.290293+0000","last_unstale":"2024-04-26T01:38:53.779276+0000","last_undegraded":"2024-04-26T01:38:53.779276+0000","last_fullsized":"2024-04-26T01:38:53.779276+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:07.439333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489203+0000","last_change":"2024-04-26T01:36:50.237578+0000","last_active":"2024-04-26T01:37:25.489203+0000","last_peered":"2024-04-26T01:37:25.489203+0000","last_clean":"2024-04-26T01:37:25.489203+0000","last_became_active":"2024-04-26T01:36:50.237224+0000","last_became_peered":"2024-04-26T01:36:50.237224+0000","last_unstale":"2024-04-26T01:37:25.489203+0000","last_undegraded":"2024-04-26T01:37:25.489203+0000","last_fullsized":"2024-04-26T01:37:25.489203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:46:38.078631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.790370+0000","last_change":"2024-04-26T01:36:54.280818+0000","last_active":"2024-04-26T01:38:53.790370+0000","last_peered":"2024-04-26T01:38:53.790370+0000","last_clean":"2024-04-26T01:38:53.790370+0000","last_became_active":"2024-04-26T01:36:54.280440+0000","last_became_peered":"2024-04-26T01:36:54.280440+0000","last_unstale":"2024-04-26T01:38:53.790370+0000","last_undegraded":"2024-04-26T01:38:53.790370+0000","last_fullsized":"2024-04-26T01:38:53.790370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:47.185219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":158,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:21.722031+0000","last_change":"2024-04-26T01:36:50.234602+0000","last_active":"2024-04-26T01:39:21.722031+0000","last_peered":"2024-04-26T01:39:21.722031+0000","last_clean":"2024-04-26T01:39:21.722031+0000","last_became_active":"2024-04-26T01:36:50.234175+0000","last_became_peered":"2024-04-26T01:36:50.234175+0000","last_unstale":"2024-04-26T01:39:21.722031+0000","last_undegraded":"2024-04-26T01:39:21.722031+0000","last_fullsized":"2024-04-26T01:39:21.722031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:17:14.236069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.766641+0000","last_change":"2024-04-26T01:36:54.271466+0000","last_active":"2024-04-26T01:38:53.766641+0000","last_peered":"2024-04-26T01:38:53.766641+0000","last_clean":"2024-04-26T01:38:53.766641+0000","last_became_active":"2024-04-26T01:36:54.271086+0000","last_became_peered":"2024-04-26T01:36:54.271086+0000","last_unstale":"2024-04-26T01:38:53.766641+0000","last_undegraded":"2024-04-26T01:38:53.766641+0000","last_fullsized":"2024-04-26T01:38:53.766641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:41.052895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.519343+0000","last_change":"2024-04-26T01:37:25.518730+0000","last_active":"2024-04-26T01:37:25.519343+0000","last_peered":"2024-04-26T01:37:25.519343+0000","last_clean":"2024-04-26T01:37:25.519343+0000","last_became_active":"2024-04-26T01:37:25.504464+0000","last_became_peered":"2024-04-26T01:37:25.504464+0000","last_unstale":"2024-04-26T01:37:25.519343+0000","last_undegraded":"2024-04-26T01:37:25.519343+0000","last_fullsized":"2024-04-26T01:37:25.519343+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:07.578917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.771883+0000","last_change":"2024-04-26T01:36:54.273621+0000","last_active":"2024-04-26T01:38:53.771883+0000","last_peered":"2024-04-26T01:38:53.771883+0000","last_clean":"2024-04-26T01:38:53.771883+0000","last_became_active":"2024-04-26T01:36:54.273218+0000","last_became_peered":"2024-04-26T01:36:54.273218+0000","last_unstale":"2024-04-26T01:38:53.771883+0000","last_undegraded":"2024-04-26T01:38:53.771883+0000","last_fullsized":"2024-04-26T01:38:53.771883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:55.543943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.509030+0000","last_change":"2024-04-26T01:37:25.508575+0000","last_active":"2024-04-26T01:37:25.509030+0000","last_peered":"2024-04-26T01:37:25.509030+0000","last_clean":"2024-04-26T01:37:25.509030+0000","last_became_active":"2024-04-26T01:37:25.496407+0000","last_became_peered":"2024-04-26T01:37:25.496407+0000","last_unstale":"2024-04-26T01:37:25.509030+0000","last_undegraded":"2024-04-26T01:37:25.509030+0000","last_fullsized":"2024-04-26T01:37:25.509030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:05.878252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.758792+0000","last_change":"2024-04-26T01:36:54.272147+0000","last_active":"2024-04-26T01:38:53.758792+0000","last_peered":"2024-04-26T01:38:53.758792+0000","last_clean":"2024-04-26T01:38:53.758792+0000","last_became_active":"2024-04-26T01:36:54.271737+0000","last_became_peered":"2024-04-26T01:36:54.271737+0000","last_unstale":"2024-04-26T01:38:53.758792+0000","last_undegraded":"2024-04-26T01:38:53.758792+0000","last_fullsized":"2024-04-26T01:38:53.758792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:41.445073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.496906+0000","last_change":"2024-04-26T01:36:50.264444+0000","last_active":"2024-04-26T01:37:25.496906+0000","last_peered":"2024-04-26T01:37:25.496906+0000","last_clean":"2024-04-26T01:37:25.496906+0000","last_became_active":"2024-04-26T01:36:50.264058+0000","last_became_peered":"2024-04-26T01:36:50.264058+0000","last_unstale":"2024-04-26T01:37:25.496906+0000","last_undegraded":"2024-04-26T01:37:25.496906+0000","last_fullsized":"2024-04-26T01:37:25.496906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:56:15.428588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780458+0000","last_change":"2024-04-26T01:36:54.289537+0000","last_active":"2024-04-26T01:38:53.780458+0000","last_peered":"2024-04-26T01:38:53.780458+0000","last_clean":"2024-04-26T01:38:53.780458+0000","last_became_active":"2024-04-26T01:36:54.289149+0000","last_became_peered":"2024-04-26T01:36:54.289149+0000","last_unstale":"2024-04-26T01:38:53.780458+0000","last_undegraded":"2024-04-26T01:38:53.780458+0000","last_fullsized":"2024-04-26T01:38:53.780458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:07.801551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491568+0000","last_change":"2024-04-26T01:36:50.242190+0000","last_active":"2024-04-26T01:37:25.491568+0000","last_peered":"2024-04-26T01:37:25.491568+0000","last_clean":"2024-04-26T01:37:25.491568+0000","last_became_active":"2024-04-26T01:36:50.241890+0000","last_became_peered":"2024-04-26T01:36:50.241890+0000","last_unstale":"2024-04-26T01:37:25.491568+0000","last_undegraded":"2024-04-26T01:37:25.491568+0000","last_fullsized":"2024-04-26T01:37:25.491568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:29.472502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.781419+0000","last_change":"2024-04-26T01:36:54.294699+0000","last_active":"2024-04-26T01:38:53.781419+0000","last_peered":"2024-04-26T01:38:53.781419+0000","last_clean":"2024-04-26T01:38:53.781419+0000","last_became_active":"2024-04-26T01:36:54.294313+0000","last_became_peered":"2024-04-26T01:36:54.294313+0000","last_unstale":"2024-04-26T01:38:53.781419+0000","last_undegraded":"2024-04-26T01:38:53.781419+0000","last_fullsized":"2024-04-26T01:38:53.781419+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:15.026357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.771052+0000","last_change":"2024-04-26T01:36:54.276798+0000","last_active":"2024-04-26T01:38:53.771052+0000","last_peered":"2024-04-26T01:38:53.771052+0000","last_clean":"2024-04-26T01:38:53.771052+0000","last_became_active":"2024-04-26T01:36:54.275955+0000","last_became_peered":"2024-04-26T01:36:54.275955+0000","last_unstale":"2024-04-26T01:38:53.771052+0000","last_undegraded":"2024-04-26T01:38:53.771052+0000","last_fullsized":"2024-04-26T01:38:53.771052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:17.020909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:39:17.711005+0000","last_change":"2024-04-26T01:36:50.263356+0000","last_active":"2024-04-26T01:39:17.711005+0000","last_peered":"2024-04-26T01:39:17.711005+0000","last_clean":"2024-04-26T01:39:17.711005+0000","last_became_active":"2024-04-26T01:36:50.261596+0000","last_became_peered":"2024-04-26T01:36:50.261596+0000","last_unstale":"2024-04-26T01:39:17.711005+0000","last_undegraded":"2024-04-26T01:39:17.711005+0000","last_fullsized":"2024-04-26T01:39:17.711005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:10.773347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785734+0000","last_change":"2024-04-26T01:36:54.288135+0000","last_active":"2024-04-26T01:38:53.785734+0000","last_peered":"2024-04-26T01:38:53.785734+0000","last_clean":"2024-04-26T01:38:53.785734+0000","last_became_active":"2024-04-26T01:36:54.287728+0000","last_became_peered":"2024-04-26T01:36:54.287728+0000","last_unstale":"2024-04-26T01:38:53.785734+0000","last_undegraded":"2024-04-26T01:38:53.785734+0000","last_fullsized":"2024-04-26T01:38:53.785734+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:18.428151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497337+0000","last_change":"2024-04-26T01:36:50.243522+0000","last_active":"2024-04-26T01:37:25.497337+0000","last_peered":"2024-04-26T01:37:25.497337+0000","last_clean":"2024-04-26T01:37:25.497337+0000","last_became_active":"2024-04-26T01:36:50.243084+0000","last_became_peered":"2024-04-26T01:36:50.243084+0000","last_unstale":"2024-04-26T01:37:25.497337+0000","last_undegraded":"2024-04-26T01:37:25.497337+0000","last_fullsized":"2024-04-26T01:37:25.497337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:03.159750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491542+0000","last_change":"2024-04-26T01:35:13.920486+0000","last_active":"2024-04-26T01:37:25.491542+0000","last_peered":"2024-04-26T01:37:25.491542+0000","last_clean":"2024-04-26T01:37:25.491542+0000","last_became_active":"2024-04-26T01:35:13.919834+0000","last_became_peered":"2024-04-26T01:35:13.919834+0000","last_unstale":"2024-04-26T01:37:25.491542+0000","last_undegraded":"2024-04-26T01:37:25.491542+0000","last_fullsized":"2024-04-26T01:37:25.491542+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_clean_scrub_stamp":"2024-04-26T01:31:12.563114+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:53.935491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780533+0000","last_change":"2024-04-26T01:36:54.270553+0000","last_active":"2024-04-26T01:38:53.780533+0000","last_peered":"2024-04-26T01:38:53.780533+0000","last_clean":"2024-04-26T01:38:53.780533+0000","last_became_active":"2024-04-26T01:36:54.269986+0000","last_became_peered":"2024-04-26T01:36:54.269986+0000","last_unstale":"2024-04-26T01:38:53.780533+0000","last_undegraded":"2024-04-26T01:38:53.780533+0000","last_fullsized":"2024-04-26T01:38:53.780533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:19.597337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491607+0000","last_change":"2024-04-26T01:36:50.235583+0000","last_active":"2024-04-26T01:37:25.491607+0000","last_peered":"2024-04-26T01:37:25.491607+0000","last_clean":"2024-04-26T01:37:25.491607+0000","last_became_active":"2024-04-26T01:36:50.235214+0000","last_became_peered":"2024-04-26T01:36:50.235214+0000","last_unstale":"2024-04-26T01:37:25.491607+0000","last_undegraded":"2024-04-26T01:37:25.491607+0000","last_fullsized":"2024-04-26T01:37:25.491607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:05.822872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.495731+0000","last_change":"2024-04-26T01:36:50.260988+0000","last_active":"2024-04-26T01:37:25.495731+0000","last_peered":"2024-04-26T01:37:25.495731+0000","last_clean":"2024-04-26T01:37:25.495731+0000","last_became_active":"2024-04-26T01:36:50.260615+0000","last_became_peered":"2024-04-26T01:36:50.260615+0000","last_unstale":"2024-04-26T01:37:25.495731+0000","last_undegraded":"2024-04-26T01:37:25.495731+0000","last_fullsized":"2024-04-26T01:37:25.495731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:15.933939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785032+0000","last_change":"2024-04-26T01:36:54.279650+0000","last_active":"2024-04-26T01:38:53.785032+0000","last_peered":"2024-04-26T01:38:53.785032+0000","last_clean":"2024-04-26T01:38:53.785032+0000","last_became_active":"2024-04-26T01:36:54.275814+0000","last_became_peered":"2024-04-26T01:36:54.275814+0000","last_unstale":"2024-04-26T01:38:53.785032+0000","last_undegraded":"2024-04-26T01:38:53.785032+0000","last_fullsized":"2024-04-26T01:38:53.785032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:58:42.369511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'13","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:54.319406+0000","last_change":"2024-04-26T01:36:50.248179+0000","last_active":"2024-04-26T01:38:54.319406+0000","last_peered":"2024-04-26T01:38:54.319406+0000","last_clean":"2024-04-26T01:38:54.319406+0000","last_became_active":"2024-04-26T01:36:50.247805+0000","last_became_peered":"2024-04-26T01:36:50.247805+0000","last_unstale":"2024-04-26T01:38:54.319406+0000","last_undegraded":"2024-04-26T01:38:54.319406+0000","last_fullsized":"2024-04-26T01:38:54.319406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:19:23.740920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.783371+0000","last_change":"2024-04-26T01:36:54.276626+0000","last_active":"2024-04-26T01:38:53.783371+0000","last_peered":"2024-04-26T01:38:53.783371+0000","last_clean":"2024-04-26T01:38:53.783371+0000","last_became_active":"2024-04-26T01:36:54.276238+0000","last_became_peered":"2024-04-26T01:36:54.276238+0000","last_unstale":"2024-04-26T01:38:53.783371+0000","last_undegraded":"2024-04-26T01:38:53.783371+0000","last_fullsized":"2024-04-26T01:38:53.783371+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:57.306950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.497813+0000","last_change":"2024-04-26T01:36:50.278502+0000","last_active":"2024-04-26T01:37:25.497813+0000","last_peered":"2024-04-26T01:37:25.497813+0000","last_clean":"2024-04-26T01:37:25.497813+0000","last_became_active":"2024-04-26T01:36:50.278215+0000","last_became_peered":"2024-04-26T01:36:50.278215+0000","last_unstale":"2024-04-26T01:37:25.497813+0000","last_undegraded":"2024-04-26T01:37:25.497813+0000","last_fullsized":"2024-04-26T01:37:25.497813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:05:46.142902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.780727+0000","last_change":"2024-04-26T01:36:54.292600+0000","last_active":"2024-04-26T01:38:53.780727+0000","last_peered":"2024-04-26T01:38:53.780727+0000","last_clean":"2024-04-26T01:38:53.780727+0000","last_became_active":"2024-04-26T01:36:54.292260+0000","last_became_peered":"2024-04-26T01:36:54.292260+0000","last_unstale":"2024-04-26T01:38:53.780727+0000","last_undegraded":"2024-04-26T01:38:53.780727+0000","last_fullsized":"2024-04-26T01:38:53.780727+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:10:12.296180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.488801+0000","last_change":"2024-04-26T01:36:50.262479+0000","last_active":"2024-04-26T01:37:25.488801+0000","last_peered":"2024-04-26T01:37:25.488801+0000","last_clean":"2024-04-26T01:37:25.488801+0000","last_became_active":"2024-04-26T01:36:50.262109+0000","last_became_peered":"2024-04-26T01:36:50.262109+0000","last_unstale":"2024-04-26T01:37:25.488801+0000","last_undegraded":"2024-04-26T01:37:25.488801+0000","last_fullsized":"2024-04-26T01:37:25.488801+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:19.805321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.759863+0000","last_change":"2024-04-26T01:36:54.270051+0000","last_active":"2024-04-26T01:38:53.759863+0000","last_peered":"2024-04-26T01:38:53.759863+0000","last_clean":"2024-04-26T01:38:53.759863+0000","last_became_active":"2024-04-26T01:36:54.269670+0000","last_became_peered":"2024-04-26T01:36:54.269670+0000","last_unstale":"2024-04-26T01:38:53.759863+0000","last_undegraded":"2024-04-26T01:38:53.759863+0000","last_fullsized":"2024-04-26T01:38:53.759863+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:44.137454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.514365+0000","last_change":"2024-04-26T01:36:50.248107+0000","last_active":"2024-04-26T01:37:25.514365+0000","last_peered":"2024-04-26T01:37:25.514365+0000","last_clean":"2024-04-26T01:37:25.514365+0000","last_became_active":"2024-04-26T01:36:50.247695+0000","last_became_peered":"2024-04-26T01:36:50.247695+0000","last_unstale":"2024-04-26T01:37:25.514365+0000","last_undegraded":"2024-04-26T01:37:25.514365+0000","last_fullsized":"2024-04-26T01:37:25.514365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:52.702512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.773517+0000","last_change":"2024-04-26T01:36:54.278686+0000","last_active":"2024-04-26T01:38:53.773517+0000","last_peered":"2024-04-26T01:38:53.773517+0000","last_clean":"2024-04-26T01:38:53.773517+0000","last_became_active":"2024-04-26T01:36:54.278312+0000","last_became_peered":"2024-04-26T01:36:54.278312+0000","last_unstale":"2024-04-26T01:38:53.773517+0000","last_undegraded":"2024-04-26T01:38:53.773517+0000","last_fullsized":"2024-04-26T01:38:53.773517+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:46.193032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491566+0000","last_change":"2024-04-26T01:36:50.236715+0000","last_active":"2024-04-26T01:37:25.491566+0000","last_peered":"2024-04-26T01:37:25.491566+0000","last_clean":"2024-04-26T01:37:25.491566+0000","last_became_active":"2024-04-26T01:36:50.236184+0000","last_became_peered":"2024-04-26T01:36:50.236184+0000","last_unstale":"2024-04-26T01:37:25.491566+0000","last_undegraded":"2024-04-26T01:37:25.491566+0000","last_fullsized":"2024-04-26T01:37:25.491566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:24:16.063401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.785920+0000","last_change":"2024-04-26T01:36:54.290601+0000","last_active":"2024-04-26T01:38:53.785920+0000","last_peered":"2024-04-26T01:38:53.785920+0000","last_clean":"2024-04-26T01:38:53.785920+0000","last_became_active":"2024-04-26T01:36:54.290168+0000","last_became_peered":"2024-04-26T01:36:54.290168+0000","last_unstale":"2024-04-26T01:38:53.785920+0000","last_undegraded":"2024-04-26T01:38:53.785920+0000","last_fullsized":"2024-04-26T01:38:53.785920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:48.770431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.494696+0000","last_change":"2024-04-26T01:36:50.235394+0000","last_active":"2024-04-26T01:37:25.494696+0000","last_peered":"2024-04-26T01:37:25.494696+0000","last_clean":"2024-04-26T01:37:25.494696+0000","last_became_active":"2024-04-26T01:36:50.235012+0000","last_became_peered":"2024-04-26T01:36:50.235012+0000","last_unstale":"2024-04-26T01:37:25.494696+0000","last_undegraded":"2024-04-26T01:37:25.494696+0000","last_fullsized":"2024-04-26T01:37:25.494696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:24:17.218945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.787197+0000","last_change":"2024-04-26T01:36:54.280490+0000","last_active":"2024-04-26T01:38:53.787197+0000","last_peered":"2024-04-26T01:38:53.787197+0000","last_clean":"2024-04-26T01:38:53.787197+0000","last_became_active":"2024-04-26T01:36:54.280047+0000","last_became_peered":"2024-04-26T01:36:54.280047+0000","last_unstale":"2024-04-26T01:38:53.787197+0000","last_undegraded":"2024-04-26T01:38:53.787197+0000","last_fullsized":"2024-04-26T01:38:53.787197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:32.610972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.492756+0000","last_change":"2024-04-26T01:36:50.278325+0000","last_active":"2024-04-26T01:37:25.492756+0000","last_peered":"2024-04-26T01:37:25.492756+0000","last_clean":"2024-04-26T01:37:25.492756+0000","last_became_active":"2024-04-26T01:36:50.277882+0000","last_became_peered":"2024-04-26T01:36:50.277882+0000","last_unstale":"2024-04-26T01:37:25.492756+0000","last_undegraded":"2024-04-26T01:37:25.492756+0000","last_fullsized":"2024-04-26T01:37:25.492756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:06.082223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.772210+0000","last_change":"2024-04-26T01:36:54.278106+0000","last_active":"2024-04-26T01:38:53.772210+0000","last_peered":"2024-04-26T01:38:53.772210+0000","last_clean":"2024-04-26T01:38:53.772210+0000","last_became_active":"2024-04-26T01:36:54.277644+0000","last_became_peered":"2024-04-26T01:36:54.277644+0000","last_unstale":"2024-04-26T01:38:53.772210+0000","last_undegraded":"2024-04-26T01:38:53.772210+0000","last_fullsized":"2024-04-26T01:38:53.772210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:10.619640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.489528+0000","last_change":"2024-04-26T01:36:50.237072+0000","last_active":"2024-04-26T01:37:25.489528+0000","last_peered":"2024-04-26T01:37:25.489528+0000","last_clean":"2024-04-26T01:37:25.489528+0000","last_became_active":"2024-04-26T01:36:50.236682+0000","last_became_peered":"2024-04-26T01:36:50.236682+0000","last_unstale":"2024-04-26T01:37:25.489528+0000","last_undegraded":"2024-04-26T01:37:25.489528+0000","last_fullsized":"2024-04-26T01:37:25.489528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:58:17.553492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.779926+0000","last_change":"2024-04-26T01:36:54.289997+0000","last_active":"2024-04-26T01:38:53.779926+0000","last_peered":"2024-04-26T01:38:53.779926+0000","last_clean":"2024-04-26T01:38:53.779926+0000","last_became_active":"2024-04-26T01:36:54.289505+0000","last_became_peered":"2024-04-26T01:36:54.289505+0000","last_unstale":"2024-04-26T01:38:53.779926+0000","last_undegraded":"2024-04-26T01:38:53.779926+0000","last_fullsized":"2024-04-26T01:38:53.779926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:09.340231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:37:25.491370+0000","last_change":"2024-04-26T01:36:50.245231+0000","last_active":"2024-04-26T01:37:25.491370+0000","last_peered":"2024-04-26T01:37:25.491370+0000","last_clean":"2024-04-26T01:37:25.491370+0000","last_became_active":"2024-04-26T01:36:50.244811+0000","last_became_peered":"2024-04-26T01:36:50.244811+0000","last_unstale":"2024-04-26T01:37:25.491370+0000","last_undegraded":"2024-04-26T01:37:25.491370+0000","last_fullsized":"2024-04-26T01:37:25.491370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:12:08.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.768287+0000","last_change":"2024-04-26T01:36:54.272197+0000","last_active":"2024-04-26T01:38:53.768287+0000","last_peered":"2024-04-26T01:38:53.768287+0000","last_clean":"2024-04-26T01:38:53.768287+0000","last_became_active":"2024-04-26T01:36:54.271776+0000","last_became_peered":"2024-04-26T01:36:54.271776+0000","last_unstale":"2024-04-26T01:38:53.768287+0000","last_undegraded":"2024-04-26T01:38:53.768287+0000","last_fullsized":"2024-04-26T01:38:53.768287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:20.024890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.769047+0000","last_change":"2024-04-26T01:36:54.271120+0000","last_active":"2024-04-26T01:38:53.769047+0000","last_peered":"2024-04-26T01:38:53.769047+0000","last_clean":"2024-04-26T01:38:53.769047+0000","last_became_active":"2024-04-26T01:36:54.270796+0000","last_became_peered":"2024-04-26T01:36:54.270796+0000","last_unstale":"2024-04-26T01:38:53.769047+0000","last_undegraded":"2024-04-26T01:38:53.769047+0000","last_fullsized":"2024-04-26T01:38:53.769047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:04:35.093847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.766562+0000","last_change":"2024-04-26T01:36:54.273013+0000","last_active":"2024-04-26T01:38:53.766562+0000","last_peered":"2024-04-26T01:38:53.766562+0000","last_clean":"2024-04-26T01:38:53.766562+0000","last_became_active":"2024-04-26T01:36:54.272666+0000","last_became_peered":"2024-04-26T01:36:54.272666+0000","last_unstale":"2024-04-26T01:38:53.766562+0000","last_undegraded":"2024-04-26T01:38:53.766562+0000","last_fullsized":"2024-04-26T01:38:53.766562+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:45.203185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.771970+0000","last_change":"2024-04-26T01:36:54.289427+0000","last_active":"2024-04-26T01:38:53.771970+0000","last_peered":"2024-04-26T01:38:53.771970+0000","last_clean":"2024-04-26T01:38:53.771970+0000","last_became_active":"2024-04-26T01:36:54.289090+0000","last_became_peered":"2024-04-26T01:36:54.289090+0000","last_unstale":"2024-04-26T01:38:53.771970+0000","last_undegraded":"2024-04-26T01:38:53.771970+0000","last_fullsized":"2024-04-26T01:38:53.771970+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:18:18.544777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.789535+0000","last_change":"2024-04-26T01:36:54.294308+0000","last_active":"2024-04-26T01:38:53.789535+0000","last_peered":"2024-04-26T01:38:53.789535+0000","last_clean":"2024-04-26T01:38:53.789535+0000","last_became_active":"2024-04-26T01:36:54.291910+0000","last_became_peered":"2024-04-26T01:36:54.291910+0000","last_unstale":"2024-04-26T01:38:53.789535+0000","last_undegraded":"2024-04-26T01:38:53.789535+0000","last_fullsized":"2024-04-26T01:38:53.789535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:35:16.683265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.781583+0000","last_change":"2024-04-26T01:36:54.285479+0000","last_active":"2024-04-26T01:38:53.781583+0000","last_peered":"2024-04-26T01:38:53.781583+0000","last_clean":"2024-04-26T01:38:53.781583+0000","last_became_active":"2024-04-26T01:36:54.284139+0000","last_became_peered":"2024-04-26T01:36:54.284139+0000","last_unstale":"2024-04-26T01:38:53.781583+0000","last_undegraded":"2024-04-26T01:38:53.781583+0000","last_fullsized":"2024-04-26T01:38:53.781583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:55.506496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.762770+0000","last_change":"2024-04-26T01:36:54.270567+0000","last_active":"2024-04-26T01:38:53.762770+0000","last_peered":"2024-04-26T01:38:53.762770+0000","last_clean":"2024-04-26T01:38:53.762770+0000","last_became_active":"2024-04-26T01:36:54.270218+0000","last_became_peered":"2024-04-26T01:36:54.270218+0000","last_unstale":"2024-04-26T01:38:53.762770+0000","last_undegraded":"2024-04-26T01:38:53.762770+0000","last_fullsized":"2024-04-26T01:38:53.762770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:50:43.685086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.760244+0000","last_change":"2024-04-26T01:36:54.269746+0000","last_active":"2024-04-26T01:38:53.760244+0000","last_peered":"2024-04-26T01:38:53.760244+0000","last_clean":"2024-04-26T01:38:53.760244+0000","last_became_active":"2024-04-26T01:36:54.269143+0000","last_became_peered":"2024-04-26T01:36:54.269143+0000","last_unstale":"2024-04-26T01:38:53.760244+0000","last_undegraded":"2024-04-26T01:38:53.760244+0000","last_fullsized":"2024-04-26T01:38:53.760244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:07:02.911645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.781470+0000","last_change":"2024-04-26T01:36:54.279194+0000","last_active":"2024-04-26T01:38:53.781470+0000","last_peered":"2024-04-26T01:38:53.781470+0000","last_clean":"2024-04-26T01:38:53.781470+0000","last_became_active":"2024-04-26T01:36:54.278709+0000","last_became_peered":"2024-04-26T01:36:54.278709+0000","last_unstale":"2024-04-26T01:38:53.781470+0000","last_undegraded":"2024-04-26T01:38:53.781470+0000","last_fullsized":"2024-04-26T01:38:53.781470+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:19.604884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786135+0000","last_change":"2024-04-26T01:36:54.277540+0000","last_active":"2024-04-26T01:38:53.786135+0000","last_peered":"2024-04-26T01:38:53.786135+0000","last_clean":"2024-04-26T01:38:53.786135+0000","last_became_active":"2024-04-26T01:36:54.277193+0000","last_became_peered":"2024-04-26T01:36:54.277193+0000","last_unstale":"2024-04-26T01:38:53.786135+0000","last_undegraded":"2024-04-26T01:38:53.786135+0000","last_fullsized":"2024-04-26T01:38:53.786135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:34.079094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786760+0000","last_change":"2024-04-26T01:36:54.294593+0000","last_active":"2024-04-26T01:38:53.786760+0000","last_peered":"2024-04-26T01:38:53.786760+0000","last_clean":"2024-04-26T01:38:53.786760+0000","last_became_active":"2024-04-26T01:36:54.291894+0000","last_became_peered":"2024-04-26T01:36:54.291894+0000","last_unstale":"2024-04-26T01:38:53.786760+0000","last_undegraded":"2024-04-26T01:38:53.786760+0000","last_fullsized":"2024-04-26T01:38:53.786760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:16.715108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.786355+0000","last_change":"2024-04-26T01:36:54.292433+0000","last_active":"2024-04-26T01:38:53.786355+0000","last_peered":"2024-04-26T01:38:53.786355+0000","last_clean":"2024-04-26T01:38:53.786355+0000","last_became_active":"2024-04-26T01:36:54.291445+0000","last_became_peered":"2024-04-26T01:36:54.291445+0000","last_unstale":"2024-04-26T01:38:53.786355+0000","last_undegraded":"2024-04-26T01:38:53.786355+0000","last_fullsized":"2024-04-26T01:38:53.786355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:49.081871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.791176+0000","last_change":"2024-04-26T01:36:54.291892+0000","last_active":"2024-04-26T01:38:53.791176+0000","last_peered":"2024-04-26T01:38:53.791176+0000","last_clean":"2024-04-26T01:38:53.791176+0000","last_became_active":"2024-04-26T01:36:54.291549+0000","last_became_peered":"2024-04-26T01:36:54.291549+0000","last_unstale":"2024-04-26T01:38:53.791176+0000","last_undegraded":"2024-04-26T01:38:53.791176+0000","last_fullsized":"2024-04-26T01:38:53.791176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:09:22.954651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:38:53.774026+0000","last_change":"2024-04-26T01:37:25.498138+0000","last_active":"2024-04-26T01:38:53.774026+0000","last_peered":"2024-04-26T01:38:53.774026+0000","last_clean":"2024-04-26T01:38:53.774026+0000","last_became_active":"2024-04-26T01:37:25.497681+0000","last_became_peered":"2024-04-26T01:37:25.497681+0000","last_unstale":"2024-04-26T01:38:53.774026+0000","last_undegraded":"2024-04-26T01:38:53.774026+0000","last_fullsized":"2024-04-26T01:38:53.774026+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:03:25.016623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282972,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70041,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":768,"num_read_kb":456,"num_write":175,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":210123,"data_compressed":5484,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383729,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":479903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645365,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1130496,"data_stored":1137261,"data_compressed":37816,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":480117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070529,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":480015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332167,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1072108,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495692,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":479506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":546352,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":480777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411430,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":545814,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182493,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":480055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757332,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071792,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34344,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67053,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66515,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":67477,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:39:25.239 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:39:25.239 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T01:39:25.239 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T01:39:25.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T01:39:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:25 smithi169 ceph-mon[32851]: pgmap v410: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:25 smithi169 ceph-mon[32851]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:39:25.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:25 smithi028 ceph-mon[28247]: pgmap v410: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:25.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:25 smithi028 ceph-mon[28247]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:39:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:25 smithi053 ceph-mon[32593]: pgmap v410: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:25 smithi053 ceph-mon[32593]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:39:27.733 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:27 smithi028 ceph-mon[28247]: pgmap v411: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:27.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:27 smithi169 ceph-mon[32851]: pgmap v411: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:27 smithi053 ceph-mon[32593]: pgmap v411: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:28.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.024+0000 7f95ec400700 1 -- 172.21.15.28:0/2312612707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e4109c80 msgr2=0x7f95e410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:28.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.024+0000 7f95ec400700 1 --2- 172.21.15.28:0/2312612707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e4109c80 0x7f95e410dcd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f95d00098f0 tx=0x7f95d0009c00 comp rx=0 tx=0).stop 2024-04-26T01:39:28.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.025+0000 7f95ec400700 1 -- 172.21.15.28:0/2312612707 shutdown_connections 2024-04-26T01:39:28.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.025+0000 7f95ec400700 1 --2- 172.21.15.28:0/2312612707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95e410e3b0 0x7f95e4115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.025+0000 7f95ec400700 1 --2- 172.21.15.28:0/2312612707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95e41092d0 0x7f95e41096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.025+0000 7f95ec400700 1 --2- 172.21.15.28:0/2312612707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e4109c80 0x7f95e410dcd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.025+0000 7f95ec400700 1 -- 172.21.15.28:0/2312612707 >> 172.21.15.28:0/2312612707 conn(0x7f95e4076210 msgr2=0x7f95e4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.025+0000 7f95ec400700 1 -- 172.21.15.28:0/2312612707 shutdown_connections 2024-04-26T01:39:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.025+0000 7f95ec400700 1 -- 172.21.15.28:0/2312612707 wait complete. 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.026+0000 7f95ec400700 1 Processor -- start 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.026+0000 7f95ec400700 1 -- start start 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.026+0000 7f95ec400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e41092d0 0x7f95e41a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.026+0000 7f95ec400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95e4109c80 0x7f95e41a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95ec400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95e410e3b0 0x7f95e419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95ec400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95e4119270 con 0x7f95e410e3b0 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95ec400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f95e41190f0 con 0x7f95e41092d0 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95ec400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f95e41193f0 con 0x7f95e4109c80 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95e4109c80 0x7f95e41a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e41092d0 0x7f95e41a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95ea400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95e410e3b0 0x7f95e419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95e4109c80 0x7f95e41a5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:57374/0 (socket says 172.21.15.28:57374) 2024-04-26T01:39:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e41092d0 0x7f95e41a5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:34860/0 (socket says 172.21.15.28:34860) 2024-04-26T01:39:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9000700 1 -- 172.21.15.28:0/2782764542 learned_addr learned my addr 172.21.15.28:0/2782764542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9000700 1 -- 172.21.15.28:0/2782764542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e41092d0 msgr2=0x7f95e41a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9000700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e41092d0 0x7f95e41a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9000700 1 -- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95e410e3b0 msgr2=0x7f95e419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9000700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95e410e3b0 0x7f95e419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9000700 1 -- 172.21.15.28:0/2782764542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e419ff80 con 0x7f95e4109c80 2024-04-26T01:39:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.027+0000 7f95e9a00700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e41092d0 0x7f95e41a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.028+0000 7f95e9000700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95e4109c80 0x7f95e41a5ac0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f95d000c010 tx=0x7f95d000f9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.028+0000 7f95e2000700 1 -- 172.21.15.28:0/2782764542 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0027070 con 0x7f95e4109c80 2024-04-26T01:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.028+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e41a0210 con 0x7f95e4109c80 2024-04-26T01:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.028+0000 7f95e2000700 1 -- 172.21.15.28:0/2782764542 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95d00054f0 con 0x7f95e4109c80 2024-04-26T01:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.028+0000 7f95e2000700 1 -- 172.21.15.28:0/2782764542 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0004930 con 0x7f95e4109c80 2024-04-26T01:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.028+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e41ac500 con 0x7f95e4109c80 2024-04-26T01:39:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.030+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e410a7b0 con 0x7f95e4109c80 2024-04-26T01:39:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.032+0000 7f95e2000700 1 -- 172.21.15.28:0/2782764542 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f95d002a070 con 0x7f95e4109c80 2024-04-26T01:39:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.033+0000 7f95e2000700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f95cc0a3c40 0x7f95cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.033+0000 7f95e2000700 1 -- 172.21.15.28:0/2782764542 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f95d00d0d60 con 0x7f95e4109c80 2024-04-26T01:39:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.033+0000 7f95e9a00700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f95cc0a3c40 0x7f95cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.034+0000 7f95e9a00700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f95cc0a3c40 0x7f95cc0a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f95d8011490 tx=0x7f95d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.038+0000 7f95e2000700 1 -- 172.21.15.28:0/2782764542 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95d005eff0 con 0x7f95e4109c80 2024-04-26T01:39:28.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.321+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f95e4069730 con 0x7f95e4109c80 2024-04-26T01:39:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.322+0000 7f95e2000700 1 -- 172.21.15.28:0/2782764542 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f95d00bd020 con 0x7f95e4109c80 2024-04-26T01:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f95cc0a3c40 msgr2=0x7f95cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f95cc0a3c40 0x7f95cc0a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f95d8011490 tx=0x7f95d8009260 comp rx=0 tx=0).stop 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95e4109c80 msgr2=0x7f95e41a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95e4109c80 0x7f95e41a5ac0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f95d000c010 tx=0x7f95d000f9b0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 shutdown_connections 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f95cc0a3c40 0x7f95cc0a6100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95e410e3b0 0x7f95e419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95e41092d0 0x7f95e41a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 --2- 172.21.15.28:0/2782764542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95e4109c80 0x7f95e41a5ac0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:28.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 >> 172.21.15.28:0/2782764542 conn(0x7f95e4076210 msgr2=0x7f95e4077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:28.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 shutdown_connections 2024-04-26T01:39:28.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:28.325+0000 7f95ec400700 1 -- 172.21.15.28:0/2782764542 wait complete. 2024-04-26T01:39:28.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:28 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2782764542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:39:28.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:28 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2782764542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:39:28.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:28 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2782764542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:39:29.336 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T01:39:29.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:39:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:29.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:29 smithi169 ceph-mon[32851]: pgmap v412: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:29 smithi053 ceph-mon[32593]: pgmap v412: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:29.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:29 smithi028 ceph-mon[28247]: pgmap v412: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:31.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:31 smithi028 ceph-mon[28247]: pgmap v413: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:31.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:31 smithi169 ceph-mon[32851]: pgmap v413: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:31.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:31 smithi053 ceph-mon[32593]: pgmap v413: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 -- 172.21.15.28:0/3770752045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 msgr2=0x7f24b810dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 --2- 172.21.15.28:0/3770752045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 0x7f24b810dcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f24a00099c0 tx=0x7f24a0009cd0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 -- 172.21.15.28:0/3770752045 shutdown_connections 2024-04-26T01:39:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 --2- 172.21.15.28:0/3770752045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24b810e3b0 0x7f24b8115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 --2- 172.21.15.28:0/3770752045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b81092d0 0x7f24b81096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 --2- 172.21.15.28:0/3770752045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 0x7f24b810dcd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 -- 172.21.15.28:0/3770752045 >> 172.21.15.28:0/3770752045 conn(0x7f24b8076210 msgr2=0x7f24b8107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 -- 172.21.15.28:0/3770752045 shutdown_connections 2024-04-26T01:39:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.126+0000 7f24bca00700 1 -- 172.21.15.28:0/3770752045 wait complete. 2024-04-26T01:39:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.127+0000 7f24bca00700 1 Processor -- start 2024-04-26T01:39:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.127+0000 7f24bca00700 1 -- start start 2024-04-26T01:39:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24bca00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24b81092d0 0x7f24b819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24bca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 0x7f24b819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24bca00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b810e3b0 0x7f24b81a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24bca00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8119290 con 0x7f24b8109c80 2024-04-26T01:39:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24bca00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8119110 con 0x7f24b810e3b0 2024-04-26T01:39:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24bca00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8119410 con 0x7f24b81092d0 2024-04-26T01:39:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24afe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 0x7f24b819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24b5e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24b81092d0 0x7f24b819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24afe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 0x7f24b819fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42250/0 (socket says 172.21.15.28:42250) 2024-04-26T01:39:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24afe00700 1 -- 172.21.15.28:0/258767458 learned_addr learned my addr 172.21.15.28:0/258767458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.128+0000 7f24b6800700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b810e3b0 0x7f24b81a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24afe00700 1 -- 172.21.15.28:0/258767458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24b81092d0 msgr2=0x7f24b819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24afe00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24b81092d0 0x7f24b819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24afe00700 1 -- 172.21.15.28:0/258767458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b810e3b0 msgr2=0x7f24b81a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24afe00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b810e3b0 0x7f24b81a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24afe00700 1 -- 172.21.15.28:0/258767458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b81a4730 con 0x7f24b8109c80 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24b5e00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24b81092d0 0x7f24b819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24afe00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 0x7f24b819fc60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24a0005f50 tx=0x7f24a0005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24aea00700 1 -- 172.21.15.28:0/258767458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a0026070 con 0x7f24b8109c80 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24aea00700 1 -- 172.21.15.28:0/258767458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24a0003c70 con 0x7f24b8109c80 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b81a49c0 con 0x7f24b8109c80 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24aea00700 1 -- 172.21.15.28:0/258767458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a000f960 con 0x7f24b8109c80 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.129+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b81ac2a0 con 0x7f24b8109c80 2024-04-26T01:39:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.131+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b8069730 con 0x7f24b8109c80 2024-04-26T01:39:32.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.134+0000 7f24aea00700 1 -- 172.21.15.28:0/258767458 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f24a0029070 con 0x7f24b8109c80 2024-04-26T01:39:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.135+0000 7f24aea00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f249c0a3c40 0x7f249c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.136+0000 7f24aea00700 1 -- 172.21.15.28:0/258767458 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f24a00d0ff0 con 0x7f24b8109c80 2024-04-26T01:39:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.136+0000 7f24b5e00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f249c0a3c40 0x7f249c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.137+0000 7f24b5e00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f249c0a3c40 0x7f249c0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f24a8011490 tx=0x7f24a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.139+0000 7f24aea00700 1 -- 172.21.15.28:0/258767458 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24a0030020 con 0x7f24b8109c80 2024-04-26T01:39:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.420+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24b802df80 con 0x7f24b8109c80 2024-04-26T01:39:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.422+0000 7f24aea00700 1 -- 172.21.15.28:0/258767458 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11861 (secure 0 0 0) 0x7f24a0013070 con 0x7f24b8109c80 2024-04-26T01:39:32.419 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:39:32.419 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:37:21.237658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:39:32.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f249c0a3c40 msgr2=0x7f249c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:32.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f249c0a3c40 0x7f249c0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f24a8011490 tx=0x7f24a8009260 comp rx=0 tx=0).stop 2024-04-26T01:39:32.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 msgr2=0x7f24b819fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:32.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 0x7f24b819fc60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24a0005f50 tx=0x7f24a0005280 comp rx=0 tx=0).stop 2024-04-26T01:39:32.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 shutdown_connections 2024-04-26T01:39:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f249c0a3c40 0x7f249c0a6100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b810e3b0 0x7f24b81a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24b81092d0 0x7f24b819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 --2- 172.21.15.28:0/258767458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8109c80 0x7f24b819fc60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 >> 172.21.15.28:0/258767458 conn(0x7f24b8076210 msgr2=0x7f24b8076830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 shutdown_connections 2024-04-26T01:39:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:32.425+0000 7f24bca00700 1 -- 172.21.15.28:0/258767458 wait complete. 2024-04-26T01:39:32.424 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-26T01:39:32.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:32 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/258767458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:32.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:32 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/258767458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:32.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:32 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/258767458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:33.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:39:33.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:33 smithi028 ceph-mon[28247]: pgmap v414: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:33.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:33 smithi169 ceph-mon[32851]: pgmap v414: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:33.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:33 smithi053 ceph-mon[32593]: pgmap v414: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:35.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:35 smithi028 ceph-mon[28247]: pgmap v415: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:35.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:35 smithi169 ceph-mon[32851]: pgmap v415: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:35.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:35 smithi053 ceph-mon[32593]: pgmap v415: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 -- 172.21.15.28:0/2477235338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c108f80 msgr2=0x7ff41c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 --2- 172.21.15.28:0/2477235338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c108f80 0x7ff41c109400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff4040098f0 tx=0x7ff404009c00 comp rx=0 tx=0).stop 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 -- 172.21.15.28:0/2477235338 shutdown_connections 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 --2- 172.21.15.28:0/2477235338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff41c109940 0x7ff41c115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 --2- 172.21.15.28:0/2477235338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff41c10f030 0x7ff41c10f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 --2- 172.21.15.28:0/2477235338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c108f80 0x7ff41c109400 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 -- 172.21.15.28:0/2477235338 >> 172.21.15.28:0/2477235338 conn(0x7ff41c0787d0 msgr2=0x7ff41c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 -- 172.21.15.28:0/2477235338 shutdown_connections 2024-04-26T01:39:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.099+0000 7ff420a00700 1 -- 172.21.15.28:0/2477235338 wait complete. 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.101+0000 7ff420a00700 1 Processor -- start 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.101+0000 7ff420a00700 1 -- start start 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.101+0000 7ff420a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff41c108f80 0x7ff41c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.101+0000 7ff420a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c109940 0x7ff41c19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.101+0000 7ff420a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff41c10f030 0x7ff41c1a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.101+0000 7ff420a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c1190f0 con 0x7ff41c10f030 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.101+0000 7ff420a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c118f70 con 0x7ff41c109940 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.101+0000 7ff420a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c119270 con 0x7ff41c108f80 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff413e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c109940 0x7ff41c19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff41be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff41c10f030 0x7ff41c1a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff41b400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff41c108f80 0x7ff41c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff413e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c109940 0x7ff41c19fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:60502/0 (socket says 172.21.15.28:60502) 2024-04-26T01:39:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff413e00700 1 -- 172.21.15.28:0/2601035194 learned_addr learned my addr 172.21.15.28:0/2601035194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff41be00700 1 -- 172.21.15.28:0/2601035194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff41c108f80 msgr2=0x7ff41c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff41be00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff41c108f80 0x7ff41c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff41be00700 1 -- 172.21.15.28:0/2601035194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c109940 msgr2=0x7ff41c19fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff41be00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c109940 0x7ff41c19fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.102+0000 7ff41be00700 1 -- 172.21.15.28:0/2601035194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41c1a45f0 con 0x7ff41c10f030 2024-04-26T01:39:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.103+0000 7ff41be00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff41c10f030 0x7ff41c1a3eb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff414009cd0 tx=0x7ff41400c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.103+0000 7ff418c00700 1 -- 172.21.15.28:0/2601035194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41400cd10 con 0x7ff41c10f030 2024-04-26T01:39:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.103+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff41c1a4880 con 0x7ff41c10f030 2024-04-26T01:39:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.103+0000 7ff418c00700 1 -- 172.21.15.28:0/2601035194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff414003a00 con 0x7ff41c10f030 2024-04-26T01:39:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.103+0000 7ff418c00700 1 -- 172.21.15.28:0/2601035194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414004520 con 0x7ff41c10f030 2024-04-26T01:39:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.103+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff41c1ac160 con 0x7ff41c10f030 2024-04-26T01:39:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.104+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff41c111380 con 0x7ff41c10f030 2024-04-26T01:39:36.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.106+0000 7ff418c00700 1 -- 172.21.15.28:0/2601035194 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff41401a020 con 0x7ff41c10f030 2024-04-26T01:39:36.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.108+0000 7ff418c00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff4000a3c40 0x7ff4000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:36.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.108+0000 7ff418c00700 1 -- 172.21.15.28:0/2601035194 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7ff4140cfd00 con 0x7ff41c10f030 2024-04-26T01:39:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.108+0000 7ff41b400700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff4000a3c40 0x7ff4000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.109+0000 7ff41b400700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff4000a3c40 0x7ff4000a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009c60 tx=0x7ff40c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.113+0000 7ff418c00700 1 -- 172.21.15.28:0/2601035194 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff414010040 con 0x7ff41c10f030 2024-04-26T01:39:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.390+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff41c1a0940 con 0x7ff41c10f030 2024-04-26T01:39:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.392+0000 7ff418c00700 1 -- 172.21.15.28:0/2601035194 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11861 (secure 0 0 0) 0x7ff41409c690 con 0x7ff41c10f030 2024-04-26T01:39:36.389 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:39:36.389 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:37:21.237658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:39:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff4000a3c40 msgr2=0x7ff4000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff4000a3c40 0x7ff4000a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009c60 tx=0x7ff40c009260 comp rx=0 tx=0).stop 2024-04-26T01:39:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff41c10f030 msgr2=0x7ff41c1a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff41c10f030 0x7ff41c1a3eb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff414009cd0 tx=0x7ff41400c620 comp rx=0 tx=0).stop 2024-04-26T01:39:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 shutdown_connections 2024-04-26T01:39:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff4000a3c40 0x7ff4000a6100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff41c10f030 0x7ff41c1a3eb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff41c108f80 0x7ff41c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 --2- 172.21.15.28:0/2601035194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff41c109940 0x7ff41c19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 >> 172.21.15.28:0/2601035194 conn(0x7ff41c0787d0 msgr2=0x7ff41c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 shutdown_connections 2024-04-26T01:39:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:36.395+0000 7ff420a00700 1 -- 172.21.15.28:0/2601035194 wait complete. 2024-04-26T01:39:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-26T01:39:36.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:36 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2601035194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:36.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:36 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2601035194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:36.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:36 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2601035194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:37.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:39:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:37.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:37 smithi028 ceph-mon[28247]: pgmap v416: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:37.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:37 smithi169 ceph-mon[32851]: pgmap v416: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:37 smithi053 ceph-mon[32593]: pgmap v416: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:39.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:39 smithi028 ceph-mon[28247]: pgmap v417: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:39.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:39 smithi169 ceph-mon[32851]: pgmap v417: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:39.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:39 smithi053 ceph-mon[32593]: pgmap v417: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 -- 172.21.15.28:0/902318265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c4108b50 msgr2=0x7f23c4108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 --2- 172.21.15.28:0/902318265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c4108b50 0x7f23c4108f30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f23b40098f0 tx=0x7f23b4009c00 comp rx=0 tx=0).stop 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 -- 172.21.15.28:0/902318265 shutdown_connections 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 --2- 172.21.15.28:0/902318265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23c406bb40 0x7f23c410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 --2- 172.21.15.28:0/902318265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c4108b50 0x7f23c4108f30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 --2- 172.21.15.28:0/902318265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23c406b180 0x7f23c406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 -- 172.21.15.28:0/902318265 >> 172.21.15.28:0/902318265 conn(0x7f23c4078740 msgr2=0x7f23c4078b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 -- 172.21.15.28:0/902318265 shutdown_connections 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 -- 172.21.15.28:0/902318265 wait complete. 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 Processor -- start 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.077+0000 7f23c8c00700 1 -- start start 2024-04-26T01:39:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c8c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23c406b180 0x7f23c4199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c8c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c406bb40 0x7f23c41996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c8c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23c4108b50 0x7f23c419da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c8c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23c4112c70 con 0x7f23c4108b50 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c8c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f23c4112af0 con 0x7f23c406b180 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c8c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f23c4112df0 con 0x7f23c406bb40 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c1400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c406bb40 0x7f23c41996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c1400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c406bb40 0x7f23c41996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:49102/0 (socket says 172.21.15.28:49102) 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c1e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23c406b180 0x7f23c4199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c1400700 1 -- 172.21.15.28:0/21044973 learned_addr learned my addr 172.21.15.28:0/21044973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.078+0000 7f23c2800700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23c4108b50 0x7f23c419da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:40.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23c1e00700 1 -- 172.21.15.28:0/21044973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c406bb40 msgr2=0x7f23c41996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:40.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23c1e00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c406bb40 0x7f23c41996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23c1e00700 1 -- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23c4108b50 msgr2=0x7f23c419da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23c1e00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23c4108b50 0x7f23c419da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23c1e00700 1 -- 172.21.15.28:0/21044973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c419e170 con 0x7f23c406b180 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23c1e00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23c406b180 0x7f23c4199160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f23b4000c00 tx=0x7f23b400fa20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23baa00700 1 -- 172.21.15.28:0/21044973 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b4005040 con 0x7f23c406b180 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23baa00700 1 -- 172.21.15.28:0/21044973 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23b4005690 con 0x7f23c406b180 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23baa00700 1 -- 172.21.15.28:0/21044973 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b40038e0 con 0x7f23c406b180 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.079+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c419e400 con 0x7f23c406b180 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.080+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c4101c40 con 0x7f23c406b180 2024-04-26T01:39:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.081+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23c41041f0 con 0x7f23c406b180 2024-04-26T01:39:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.083+0000 7f23baa00700 1 -- 172.21.15.28:0/21044973 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f23b400b760 con 0x7f23c406b180 2024-04-26T01:39:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.084+0000 7f23baa00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f23a80a3c40 0x7f23a80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.084+0000 7f23baa00700 1 -- 172.21.15.28:0/21044973 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f23b40d0cc0 con 0x7f23c406b180 2024-04-26T01:39:40.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.085+0000 7f23c1400700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f23a80a3c40 0x7f23a80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:40.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.085+0000 7f23c1400700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f23a80a3c40 0x7f23a80a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009680 tx=0x7f23b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:40.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.089+0000 7f23baa00700 1 -- 172.21.15.28:0/21044973 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23b40d7080 con 0x7f23c406b180 2024-04-26T01:39:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.326+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23c419a4c0 con 0x7f23c406b180 2024-04-26T01:39:40.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.328+0000 7f23baa00700 1 -- 172.21.15.28:0/21044973 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22630 (secure 0 0 0) 0x7f23b4013030 con 0x7f23c406b180 2024-04-26T01:39:40.326 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:39:40.326 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:37:25.472266+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":7,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.332+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f23a80a3c40 msgr2=0x7f23a80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.332+0000 7f23c8c00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f23a80a3c40 0x7f23a80a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009680 tx=0x7f23b0009260 comp rx=0 tx=0).stop 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23c406b180 msgr2=0x7f23c4199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23c406b180 0x7f23c4199160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f23b4000c00 tx=0x7f23b400fa20 comp rx=0 tx=0).stop 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 shutdown_connections 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f23a80a3c40 0x7f23a80a6100 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23c4108b50 0x7f23c419da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23c406b180 0x7f23c4199160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 --2- 172.21.15.28:0/21044973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c406bb40 0x7f23c41996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 >> 172.21.15.28:0/21044973 conn(0x7f23c4078740 msgr2=0x7f23c4100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 shutdown_connections 2024-04-26T01:39:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:40.333+0000 7f23c8c00700 1 -- 172.21.15.28:0/21044973 wait complete. 2024-04-26T01:39:40.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:40 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/21044973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:39:40.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:40 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/21044973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:39:41.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:40 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/21044973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:39:41.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:39:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:41.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:41 smithi028 ceph-mon[28247]: pgmap v418: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:41.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:41 smithi169 ceph-mon[32851]: pgmap v418: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:41.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:41 smithi053 ceph-mon[32593]: pgmap v418: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:43.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:43 smithi028 ceph-mon[28247]: pgmap v419: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:43.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:43 smithi169 ceph-mon[32851]: pgmap v419: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:43.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:43 smithi053 ceph-mon[32593]: pgmap v419: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:44.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.197+0000 7fd84fe00700 1 -- 172.21.15.28:0/1546294871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 msgr2=0x7fd848111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:44.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.197+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1546294871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 0x7fd848111b70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd84000b0d0 tx=0x7fd84000b3e0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.198+0000 7fd84fe00700 1 -- 172.21.15.28:0/1546294871 shutdown_connections 2024-04-26T01:39:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.198+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1546294871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 0x7fd848111b70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.198+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1546294871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd848104ec0 0x7fd8481052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.198+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1546294871 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd848105870 0x7fd8481098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.198+0000 7fd84fe00700 1 -- 172.21.15.28:0/1546294871 >> 172.21.15.28:0/1546294871 conn(0x7fd848100750 msgr2=0x7fd848102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.198+0000 7fd84fe00700 1 -- 172.21.15.28:0/1546294871 shutdown_connections 2024-04-26T01:39:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.198+0000 7fd84fe00700 1 -- 172.21.15.28:0/1546294871 wait complete. 2024-04-26T01:39:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.199+0000 7fd84fe00700 1 Processor -- start 2024-04-26T01:39:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.199+0000 7fd84fe00700 1 -- start start 2024-04-26T01:39:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84fe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd848104ec0 0x7fd84819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84fe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd848105870 0x7fd84819b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 0x7fd84819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84fe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd848114e40 con 0x7fd848109fa0 2024-04-26T01:39:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84fe00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd848114cc0 con 0x7fd848104ec0 2024-04-26T01:39:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84fe00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd848114fc0 con 0x7fd848105870 2024-04-26T01:39:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84d400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd848104ec0 0x7fd84819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 0x7fd84819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.200+0000 7fd84ca00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd848105870 0x7fd84819b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84ca00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd848105870 0x7fd84819b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:49112/0 (socket says 172.21.15.28:49112) 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84ca00700 1 -- 172.21.15.28:0/1405375842 learned_addr learned my addr 172.21.15.28:0/1405375842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84d400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd848104ec0 0x7fd84819b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:49086/0 (socket says 172.21.15.28:49086) 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84ca00700 1 -- 172.21.15.28:0/1405375842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd848104ec0 msgr2=0x7fd84819b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84ca00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd848104ec0 0x7fd84819b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84ca00700 1 -- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 msgr2=0x7fd84819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84ca00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 0x7fd84819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84ca00700 1 -- 172.21.15.28:0/1405375842 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8481a0320 con 0x7fd848105870 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84de00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 0x7fd84819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.201+0000 7fd84ca00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd848105870 0x7fd84819b850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd83400e970 tx=0x7fd83400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.202+0000 7fd846000700 1 -- 172.21.15.28:0/1405375842 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd834004a30 con 0x7fd848105870 2024-04-26T01:39:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.202+0000 7fd846000700 1 -- 172.21.15.28:0/1405375842 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd83400cc90 con 0x7fd848105870 2024-04-26T01:39:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.202+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8481a0610 con 0x7fd848105870 2024-04-26T01:39:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.202+0000 7fd846000700 1 -- 172.21.15.28:0/1405375842 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd834005610 con 0x7fd848105870 2024-04-26T01:39:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.202+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8481a7e90 con 0x7fd848105870 2024-04-26T01:39:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.203+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd848069730 con 0x7fd848105870 2024-04-26T01:39:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.205+0000 7fd846000700 1 -- 172.21.15.28:0/1405375842 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd83400c7e0 con 0x7fd848105870 2024-04-26T01:39:44.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.207+0000 7fd846000700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd8300a3cc0 0x7fd8300a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.207+0000 7fd846000700 1 -- 172.21.15.28:0/1405375842 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fd8340d04c0 con 0x7fd848105870 2024-04-26T01:39:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.207+0000 7fd84d400700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd8300a3cc0 0x7fd8300a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.208+0000 7fd84d400700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd8300a3cc0 0x7fd8300a6180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd83c009c60 tx=0x7fd83c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:44.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.209+0000 7fd846000700 1 -- 172.21.15.28:0/1405375842 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd83409ce50 con 0x7fd848105870 2024-04-26T01:39:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.492+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd84819c670 con 0x7fd848105870 2024-04-26T01:39:44.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.493+0000 7fd846000700 1 -- 172.21.15.28:0/1405375842 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11861 (secure 0 0 0) 0x7fd83409c750 con 0x7fd848105870 2024-04-26T01:39:44.490 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:39:44.490 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:37:21.237658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.497+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd8300a3cc0 msgr2=0x7fd8300a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.497+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd8300a3cc0 0x7fd8300a6180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd83c009c60 tx=0x7fd83c009260 comp rx=0 tx=0).stop 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.497+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd848105870 msgr2=0x7fd84819b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.497+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd848105870 0x7fd84819b850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd83400e970 tx=0x7fd83400ec80 comp rx=0 tx=0).stop 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.498+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 shutdown_connections 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.498+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fd8300a3cc0 0x7fd8300a6180 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.498+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd848109fa0 0x7fd84819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.498+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd848104ec0 0x7fd84819b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.498+0000 7fd84fe00700 1 --2- 172.21.15.28:0/1405375842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd848105870 0x7fd84819b850 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.498+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 >> 172.21.15.28:0/1405375842 conn(0x7fd848100750 msgr2=0x7fd8481012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:44.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.498+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 shutdown_connections 2024-04-26T01:39:44.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:44.498+0000 7fd84fe00700 1 -- 172.21.15.28:0/1405375842 wait complete. 2024-04-26T01:39:44.495 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-26T01:39:44.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:44 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1405375842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:44.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:44 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1405375842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:44.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:44 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1405375842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:39:45.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:36:57.152667+0000', 'modified': '2024-04-26T01:37:21.237658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24523, 'mds_1': 24509, 'mds_2': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24491': {'gid': 24491, 'name': 'g', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/2114874192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2114874192}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2114874192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24497': {'gid': 24497, 'name': 'a', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/2299937376', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 2299937376}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 2299937376}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.169:6835/1802608598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6834', 'nonce': 1802608598}, {'type': 'v1', 'addr': '172.21.15.169:6835', 'nonce': 1802608598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.169:6837/3777955238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6836', 'nonce': 3777955238}, {'type': 'v1', 'addr': '172.21.15.169:6837', 'nonce': 3777955238}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.53:6839/955096275', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6838', 'nonce': 955096275}, {'type': 'v1', 'addr': '172.21.15.53:6839', 'nonce': 955096275}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.169:6839/2157805186', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6838', 'nonce': 2157805186}, {'type': 'v1', 'addr': '172.21.15.169:6839', 'nonce': 2157805186}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24523, 'qdb_cluster': [24505, 24523, 24509]} 2024-04-26T01:39:45.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:39:45.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-26T01:39:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:45.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:45 smithi169 ceph-mon[32851]: pgmap v420: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:45.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:45 smithi028 ceph-mon[28247]: pgmap v420: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:45.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:45 smithi053 ceph-mon[32593]: pgmap v420: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:47.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:47 smithi028 ceph-mon[28247]: pgmap v421: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:47.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:47 smithi169 ceph-mon[32851]: pgmap v421: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:47.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:47 smithi053 ceph-mon[32593]: pgmap v421: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.180+0000 7f7959a00700 1 -- 172.21.15.28:0/522256709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 msgr2=0x7f7954075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.180+0000 7f7959a00700 1 --2- 172.21.15.28:0/522256709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 0x7f7954075050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7944009990 tx=0x7f7944009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.180+0000 7f7959a00700 1 -- 172.21.15.28:0/522256709 shutdown_connections 2024-04-26T01:39:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.180+0000 7f7959a00700 1 --2- 172.21.15.28:0/522256709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795410f440 0x7f7954111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.180+0000 7f7959a00700 1 --2- 172.21.15.28:0/522256709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 0x7f7954075050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.180+0000 7f7959a00700 1 --2- 172.21.15.28:0/522256709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7954075590 0x7f795410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.180+0000 7f7959a00700 1 -- 172.21.15.28:0/522256709 >> 172.21.15.28:0/522256709 conn(0x7f7954079ba0 msgr2=0x7f7954079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.181+0000 7f7959a00700 1 -- 172.21.15.28:0/522256709 shutdown_connections 2024-04-26T01:39:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.181+0000 7f7959a00700 1 -- 172.21.15.28:0/522256709 wait complete. 2024-04-26T01:39:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.182+0000 7f7959a00700 1 Processor -- start 2024-04-26T01:39:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.182+0000 7f7959a00700 1 -- start start 2024-04-26T01:39:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7959a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 0x7f795419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7959a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954075590 0x7f795419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7959a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f795410f440 0x7f795419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7959a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7954114e70 con 0x7f7954074c70 2024-04-26T01:39:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7959a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7954114cf0 con 0x7f7954075590 2024-04-26T01:39:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7959a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7954114ff0 con 0x7f795410f440 2024-04-26T01:39:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7952a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 0x7f795419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7953400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f795410f440 0x7f795419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7952000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954075590 0x7f795419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7952a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 0x7f795419b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44970/0 (socket says 172.21.15.28:44970) 2024-04-26T01:39:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.183+0000 7f7952a00700 1 -- 172.21.15.28:0/3399950788 learned_addr learned my addr 172.21.15.28:0/3399950788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f7952a00700 1 -- 172.21.15.28:0/3399950788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f795410f440 msgr2=0x7f795419fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f7952a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f795410f440 0x7f795419fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f7952a00700 1 -- 172.21.15.28:0/3399950788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954075590 msgr2=0x7f795419b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f7952a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954075590 0x7f795419b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f7952a00700 1 -- 172.21.15.28:0/3399950788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79541a02f0 con 0x7f7954074c70 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f7953400700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f795410f440 0x7f795419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f7952000700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954075590 0x7f795419b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f7952a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 0x7f795419b2e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7944000c00 tx=0x7f79440057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f794b400700 1 -- 172.21.15.28:0/3399950788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7944027070 con 0x7f7954074c70 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.184+0000 7f794b400700 1 -- 172.21.15.28:0/3399950788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7944022470 con 0x7f7954074c70 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.185+0000 7f7959a00700 1 -- 172.21.15.28:0/3399950788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79541a0580 con 0x7f7954074c70 2024-04-26T01:39:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.185+0000 7f794b400700 1 -- 172.21.15.28:0/3399950788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7944017650 con 0x7f7954074c70 2024-04-26T01:39:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.185+0000 7f7959a00700 1 -- 172.21.15.28:0/3399950788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79541a7e60 con 0x7f7954074c70 2024-04-26T01:39:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.188+0000 7f794b400700 1 -- 172.21.15.28:0/3399950788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f794402a070 con 0x7f7954074c70 2024-04-26T01:39:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.189+0000 7f794b400700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f79380a3cc0 0x7f79380a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.189+0000 7f7964422b80 1 Processor -- start 2024-04-26T01:39:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f794b400700 1 -- 172.21.15.28:0/3399950788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f79440d0fe0 con 0x7f7954074c70 2024-04-26T01:39:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f7952000700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f79380a3cc0 0x7f79380a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f7964422b80 1 -- start start 2024-04-26T01:39:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f7964422b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e1f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f7964422b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e23af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f7964422b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e24450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f7964422b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a93d68e40 con 0x558a93e24030 2024-04-26T01:39:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f7964422b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558a93d0f6e0 con 0x558a93e1fcc0 2024-04-26T01:39:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.190+0000 7f7964422b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558a93d64e00 con 0x558a93e1f3a0 2024-04-26T01:39:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952000700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f79380a3cc0 0x7f79380a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7954111100 tx=0x7f7940009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e23af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e1f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e23af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:49118/0 (socket says 172.21.15.28:49118) 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952000700 1 -- 172.21.15.28:0/514669070 learned_addr learned my addr 172.21.15.28:0/514669070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7953400700 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e24450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952a00700 1 -- 172.21.15.28:0/514669070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 msgr2=0x558a93e23af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952a00700 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e23af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952a00700 1 -- 172.21.15.28:0/514669070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 msgr2=0x558a93e24450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952a00700 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e24450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952a00700 1 -- 172.21.15.28:0/514669070 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a93d05630 con 0x558a93e1f3a0 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7953400700 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e24450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.191+0000 7f7952a00700 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e1f780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f794409d2d0 tx=0x7f7944009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b99e8593c630245f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f793fe00700 1 -- 172.21.15.28:0/514669070 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79440a16f0 con 0x558a93e1f3a0 2024-04-26T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f793fe00700 1 -- 172.21.15.28:0/514669070 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79440a5f20 con 0x558a93e1f3a0 2024-04-26T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f793fe00700 1 -- 172.21.15.28:0/514669070 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794409d800 con 0x558a93e1f3a0 2024-04-26T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f7964422b80 1 -- 172.21.15.28:0/514669070 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 msgr2=0x558a93e1f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f7964422b80 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e1f780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f794409d2d0 tx=0x7f7944009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f7964422b80 1 -- 172.21.15.28:0/514669070 shutdown_connections 2024-04-26T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f7964422b80 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e24450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f7964422b80 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e1f780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f7964422b80 1 --2- 172.21.15.28:0/514669070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e23af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.192+0000 7f7964422b80 1 -- 172.21.15.28:0/514669070 >> 172.21.15.28:0/514669070 conn(0x558a93d44f00 msgr2=0x558a93e118a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.193+0000 7f7964422b80 1 -- 172.21.15.28:0/514669070 shutdown_connections 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.193+0000 7f7964422b80 1 -- 172.21.15.28:0/514669070 wait complete. 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.193+0000 7f7964422b80 1 -- start start 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.193+0000 7f7964422b80 1 Processor -- start 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7964422b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e2b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7964422b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e28860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7964422b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e28da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7964422b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a93e2bee0 con 0x558a93e24030 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7964422b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558a93d22810 con 0x558a93e1fcc0 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7964422b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558a93e2c060 con 0x558a93e1f3a0 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7964422b80 10 client.0 _ll_register_callbacks cb 0x558a93d016d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7952000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e28860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7952a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e2b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7953400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e28da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7952000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e28860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:49126/0 (socket says 172.21.15.28:49126) 2024-04-26T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.194+0000 7f7952000700 1 -- 172.21.15.28:0/4187501776 learned_addr learned my addr 172.21.15.28:0/4187501776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f7952a00700 1 -- 172.21.15.28:0/4187501776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 msgr2=0x558a93e28860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f7952a00700 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e28860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f7952a00700 1 -- 172.21.15.28:0/4187501776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 msgr2=0x558a93e28da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f7952a00700 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e28da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f7952a00700 1 -- 172.21.15.28:0/4187501776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a93cb6880 con 0x558a93e1f3a0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f7952a00700 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e2b560 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7944099d10 tx=0x7f7944005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebd06ca451db5913 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f793cc00700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f793cc00700 1 -- 172.21.15.28:0/4187501776 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79440b5070 con 0x558a93e1f3a0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f793cc00700 1 -- 172.21.15.28:0/4187501776 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f79440bc1e0 con 0x558a93e1f3a0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.195+0000 7f7964422b80 1 -- 172.21.15.28:0/4187501776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79440a5f20 con 0x558a93e1f3a0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f793cc00700 1 -- 172.21.15.28:0/4187501776 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f794409d5a0 con 0x558a93e1f3a0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f7964422b80 10 client.24898 fetch_fsmap learned FSMap version 24 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f7964422b80 1 -- 172.21.15.28:0/4187501776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558a93cffaa0 con 0x558a93e1f3a0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f793cc00700 1 -- 172.21.15.28:0/4187501776 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79440a5f20 con 0x558a93e1f3a0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f793cc00700 1 -- 172.21.15.28:0/4187501776 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f7944005210 con 0x558a93e1f3a0 2024-04-26T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f7964422b80 10 client.24898 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f7964422b80 10 client.24898 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f7964422b80 20 client.24898 populate_metadata read hostname 'smithi028' 2024-04-26T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f7964422b80 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x558a93e38880 0x558a93e39cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f7964422b80 4 client.24898 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.196+0000 7f7964422b80 1 -- 172.21.15.28:0/4187501776 --> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a93d76630 con 0x558a93e38880 2024-04-26T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.197+0000 7f7952000700 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x558a93e38880 0x558a93e39cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.197+0000 7f7952000700 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x558a93e38880 0x558a93e39cf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed90b42e380b651d server_cookie=474a10f682185a7a in_seq=0 out_seq=0 2024-04-26T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.197+0000 7f793cc00700 10 client.24898 ms_handle_connect on v2:172.21.15.169:6836/3777955238 2024-04-26T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.198+0000 7f793cc00700 1 -- 172.21.15.28:0/4187501776 <== mds.2 v2:172.21.15.169:6836/3777955238 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a93d76630 con 0x558a93e38880 2024-04-26T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.198+0000 7f793cc00700 10 client.24898 handle_command_reply: tid=0 2024-04-26T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.198+0000 7f7964422b80 1 client.24898 shutdown 2024-04-26T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 -- 172.21.15.28:0/4187501776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 msgr2=0x558a93e2b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e2b560 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7944099d10 tx=0x7f7944005fb0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 -- 172.21.15.28:0/4187501776 shutdown_connections 2024-04-26T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x558a93e38880 0x558a93e39cf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e24030 0x558a93e28da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f793cc00700 0 client.24898 ms_handle_reset on v2:172.21.15.169:6836/3777955238 2024-04-26T01:39:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1f3a0 0x558a93e2b560 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 --2- 172.21.15.28:0/4187501776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e1fcc0 0x558a93e28860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 -- 172.21.15.28:0/4187501776 >> 172.21.15.28:0/4187501776 conn(0x558a93d44f00 msgr2=0x558a93e10e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 -- 172.21.15.28:0/4187501776 shutdown_connections 2024-04-26T01:39:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 1 -- 172.21.15.28:0/4187501776 wait complete. 2024-04-26T01:39:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.199+0000 7f7964422b80 20 client.24898 trim_cache size 0 max 16384 2024-04-26T01:39:48.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7964422b80 1 Processor -- start 2024-04-26T01:39:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7964422b80 1 -- start start 2024-04-26T01:39:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7964422b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e28cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7964422b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e34cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7964422b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 0x558a93e115d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7964422b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a93e38030 con 0x558a93e1f3a0 2024-04-26T01:39:48.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7964422b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558a93e38700 con 0x558a93e24030 2024-04-26T01:39:48.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7964422b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558a93e38dd0 con 0x558a93e1fcc0 2024-04-26T01:39:48.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7952a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e28cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7953400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 0x558a93e115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7952000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e34cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7952a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e28cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44992/0 (socket says 172.21.15.28:44992) 2024-04-26T01:39:48.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.222+0000 7f7952a00700 1 -- 172.21.15.28:0/2915244709 learned_addr learned my addr 172.21.15.28:0/2915244709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:48.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.223+0000 7f7952000700 1 -- 172.21.15.28:0/2915244709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 msgr2=0x558a93e115d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.223+0000 7f7952000700 1 --2- 172.21.15.28:0/2915244709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 0x558a93e115d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.223+0000 7f7952000700 1 -- 172.21.15.28:0/2915244709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 msgr2=0x558a93e28cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.223+0000 7f7952000700 1 --2- 172.21.15.28:0/2915244709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e28cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.223+0000 7f7952000700 1 -- 172.21.15.28:0/2915244709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a93e130a0 con 0x558a93e1fcc0 2024-04-26T01:39:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.223+0000 7f7952000700 1 --2- 172.21.15.28:0/2915244709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e34cb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x558a93e3a520 tx=0x7f7940014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d69f1826735c1b44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f793f400700 1 -- 172.21.15.28:0/2915244709 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794002f070 con 0x558a93e1fcc0 2024-04-26T01:39:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f793f400700 1 -- 172.21.15.28:0/2915244709 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f794002a6e0 con 0x558a93e1fcc0 2024-04-26T01:39:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f793f400700 1 -- 172.21.15.28:0/2915244709 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940029ca0 con 0x558a93e1fcc0 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 -- 172.21.15.28:0/2915244709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 msgr2=0x558a93e34cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 --2- 172.21.15.28:0/2915244709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e34cb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x558a93e3a520 tx=0x7f7940014000 comp rx=0 tx=0).stop 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 -- 172.21.15.28:0/2915244709 shutdown_connections 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 --2- 172.21.15.28:0/2915244709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 0x558a93e115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 --2- 172.21.15.28:0/2915244709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e28cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 --2- 172.21.15.28:0/2915244709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e34cb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 -- 172.21.15.28:0/2915244709 >> 172.21.15.28:0/2915244709 conn(0x558a93d44f00 msgr2=0x558a93e16230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 -- 172.21.15.28:0/2915244709 shutdown_connections 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.224+0000 7f7964422b80 1 -- 172.21.15.28:0/2915244709 wait complete. 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.225+0000 7f7964422b80 1 -- start start 2024-04-26T01:39:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.225+0000 7f7964422b80 1 Processor -- start 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.225+0000 7f7964422b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e43350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7964422b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e3e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7964422b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 0x558a93e3e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7964422b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a93e39b70 con 0x558a93e1f3a0 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7964422b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558a93e394a0 con 0x558a93e24030 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7964422b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558a93e3a240 con 0x558a93e1fcc0 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7964422b80 10 client.0 _ll_register_callbacks cb 0x558a93d016d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e43350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e43350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45006/0 (socket says 172.21.15.28:45006) 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952a00700 1 -- 172.21.15.28:0/2628409585 learned_addr learned my addr 172.21.15.28:0/2628409585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952000700 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e3e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7953400700 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 0x558a93e3e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952a00700 1 -- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 msgr2=0x558a93e3e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952a00700 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e3e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952a00700 1 -- 172.21.15.28:0/2628409585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 msgr2=0x558a93e3e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952a00700 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 0x558a93e3e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952a00700 1 -- 172.21.15.28:0/2628409585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a93e3ebf0 con 0x558a93e1f3a0 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.226+0000 7f7952000700 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e3e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f7952a00700 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e43350 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f794400ffd0 tx=0x7f7944022c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff50df742fbd0d34 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f793cc00700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f793cc00700 1 -- 172.21.15.28:0/2628409585 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79440b5040 con 0x558a93e1f3a0 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f793cc00700 1 -- 172.21.15.28:0/2628409585 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f79440a5ca0 con 0x558a93e1f3a0 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f7964422b80 1 -- 172.21.15.28:0/2628409585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f794002a6e0 con 0x558a93e1f3a0 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f793cc00700 1 -- 172.21.15.28:0/2628409585 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f79440a7070 con 0x558a93e1f3a0 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f7964422b80 10 client.14760 fetch_fsmap learned FSMap version 24 2024-04-26T01:39:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f7964422b80 1 -- 172.21.15.28:0/2628409585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558a93e3ebf0 con 0x558a93e1f3a0 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.227+0000 7f793cc00700 1 -- 172.21.15.28:0/2628409585 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f794002a6e0 con 0x558a93e1f3a0 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.228+0000 7f793cc00700 1 -- 172.21.15.28:0/2628409585 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f79440050f0 con 0x558a93e1f3a0 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.228+0000 7f7964422b80 10 client.14760 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.228+0000 7f7964422b80 10 client.14760 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.228+0000 7f7964422b80 20 client.14760 populate_metadata read hostname 'smithi028' 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.228+0000 7f7964422b80 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x558a93ea4e20 0x558a93ea3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.228+0000 7f7964422b80 4 client.14760 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.228+0000 7f7964422b80 1 -- 172.21.15.28:0/2628409585 --> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] -- command(tid 0: {"prefix": "status"}) -- 0x558a93d706d0 con 0x558a93ea4e20 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.228+0000 7f7952000700 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x558a93ea4e20 0x558a93ea3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7952000700 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x558a93ea4e20 0x558a93ea3b30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68e3388e17a0b6e1 server_cookie=b8b0b1824debe1dc in_seq=0 out_seq=0 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f793cc00700 10 client.14760 ms_handle_connect on v2:172.21.15.169:6836/3777955238 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f793cc00700 1 -- 172.21.15.28:0/2628409585 <== mds.2 v2:172.21.15.169:6836/3777955238 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558a93d706d0 con 0x558a93ea4e20 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f793cc00700 10 client.14760 handle_command_reply: tid=0 2024-04-26T01:39:48.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 client.14760 shutdown 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 -- 172.21.15.28:0/2628409585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 msgr2=0x558a93e43350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e43350 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f794400ffd0 tx=0x7f7944022c30 comp rx=0 tx=0).stop 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 -- 172.21.15.28:0/2628409585 shutdown_connections 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x558a93ea4e20 0x558a93ea3b30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558a93e24030 0x558a93e3e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a93e1f3a0 0x558a93e43350 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 --2- 172.21.15.28:0/2628409585 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558a93e1fcc0 0x558a93e3e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f793cc00700 0 client.14760 ms_handle_reset on v2:172.21.15.169:6836/3777955238 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.229+0000 7f7964422b80 1 -- 172.21.15.28:0/2628409585 >> 172.21.15.28:0/2628409585 conn(0x558a93d44f00 msgr2=0x558a93e24410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.230+0000 7f7964422b80 1 -- 172.21.15.28:0/2628409585 shutdown_connections 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.230+0000 7f7964422b80 1 -- 172.21.15.28:0/2628409585 wait complete. 2024-04-26T01:39:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.230+0000 7f7964422b80 20 client.14760 trim_cache size 0 max 16384 2024-04-26T01:39:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:39:48.230 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:39:48.230 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-26T01:39:48.230 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24505, 2024-04-26T01:39:48.230 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:39:48.230 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:39:48.230 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:39:48.230 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 153.83240831800001, 2024-04-26T01:39:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-26T01:39:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-26T01:39:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T01:39:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 212.03847644800001 2024-04-26T01:39:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:39:48.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.230+0000 7f7959a00700 1 -- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f79380a3cc0 msgr2=0x7f79380a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.230+0000 7f7959a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f79380a3cc0 0x7f79380a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7954111100 tx=0x7f7940009260 comp rx=0 tx=0).stop 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.230+0000 7f7959a00700 1 -- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 msgr2=0x7f795419b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.230+0000 7f7959a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 0x7f795419b2e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7944000c00 tx=0x7f79440057c0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.231+0000 7f7959a00700 1 -- 172.21.15.28:0/3399950788 shutdown_connections 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.231+0000 7f7959a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f79380a3cc0 0x7f79380a6180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.231+0000 7f7959a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f795410f440 0x7f795419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.231+0000 7f7959a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7954074c70 0x7f795419b2e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.231+0000 7f7959a00700 1 --2- 172.21.15.28:0/3399950788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954075590 0x7f795419b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.231+0000 7f7959a00700 1 -- 172.21.15.28:0/3399950788 >> 172.21.15.28:0/3399950788 conn(0x7f7954079ba0 msgr2=0x7f79540ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.231+0000 7f7959a00700 1 -- 172.21.15.28:0/3399950788 shutdown_connections 2024-04-26T01:39:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:48.231+0000 7f7959a00700 1 -- 172.21.15.28:0/3399950788 wait complete. 2024-04-26T01:39:49.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:39:49.602 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:49.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:49 smithi169 ceph-mon[32851]: pgmap v422: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:49 smithi028 ceph-mon[28247]: pgmap v422: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:49.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:49 smithi053 ceph-mon[32593]: pgmap v422: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:51.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:51 smithi169 ceph-mon[32851]: pgmap v423: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:51.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.935+0000 7f3115e00700 1 -- 172.21.15.28:0/1807705415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110109fc0 msgr2=0x7f3110111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.935+0000 7f3115e00700 1 --2- 172.21.15.28:0/1807705415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110109fc0 0x7f3110111b90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f310800b0a0 tx=0x7f310800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:51 smithi028 ceph-mon[28247]: pgmap v423: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.936+0000 7f3115e00700 1 -- 172.21.15.28:0/1807705415 shutdown_connections 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.936+0000 7f3115e00700 1 --2- 172.21.15.28:0/1807705415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110109fc0 0x7f3110111b90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.936+0000 7f3115e00700 1 --2- 172.21.15.28:0/1807705415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3110104ee0 0x7f31101052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.936+0000 7f3115e00700 1 --2- 172.21.15.28:0/1807705415 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3110105890 0x7f31101098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.936+0000 7f3115e00700 1 -- 172.21.15.28:0/1807705415 >> 172.21.15.28:0/1807705415 conn(0x7f3110100770 msgr2=0x7f3110102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.936+0000 7f3115e00700 1 -- 172.21.15.28:0/1807705415 shutdown_connections 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.936+0000 7f3115e00700 1 -- 172.21.15.28:0/1807705415 wait complete. 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.937+0000 7f3115e00700 1 Processor -- start 2024-04-26T01:39:51.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.937+0000 7f3115e00700 1 -- start start 2024-04-26T01:39:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.937+0000 7f3115e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3110104ee0 0x7f311019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.937+0000 7f3115e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110105890 0x7f311019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.937+0000 7f3115e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3110109fc0 0x7f311019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f3115e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3110114ec0 con 0x7f3110109fc0 2024-04-26T01:39:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f3115e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3110114d40 con 0x7f3110105890 2024-04-26T01:39:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f3115e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3110115040 con 0x7f3110104ee0 2024-04-26T01:39:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310ea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3110104ee0 0x7f311019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310ea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3110104ee0 0x7f311019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54040/0 (socket says 172.21.15.28:54040) 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310ea00700 1 -- 172.21.15.28:0/28046845 learned_addr learned my addr 172.21.15.28:0/28046845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f3107e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110105890 0x7f311019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310f400700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3110109fc0 0x7f311019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310ea00700 1 -- 172.21.15.28:0/28046845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110105890 msgr2=0x7f311019b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310ea00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110105890 0x7f311019b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310ea00700 1 -- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3110109fc0 msgr2=0x7f311019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310ea00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3110109fc0 0x7f311019fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.938+0000 7f310ea00700 1 -- 172.21.15.28:0/28046845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31101a0330 con 0x7f3110104ee0 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.939+0000 7f310ea00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3110104ee0 0x7f311019b370 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f310000e9a0 tx=0x7f310000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.939+0000 7f3114a00700 1 -- 172.21.15.28:0/28046845 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310000ca10 con 0x7f3110104ee0 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.939+0000 7f3114a00700 1 -- 172.21.15.28:0/28046845 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f310000cbb0 con 0x7f3110104ee0 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.939+0000 7f3115e00700 1 -- 172.21.15.28:0/28046845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31101a0620 con 0x7f3110104ee0 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.939+0000 7f3114a00700 1 -- 172.21.15.28:0/28046845 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100018680 con 0x7f3110104ee0 2024-04-26T01:39:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.939+0000 7f3115e00700 1 -- 172.21.15.28:0/28046845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31101a7ea0 con 0x7f3110104ee0 2024-04-26T01:39:51.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.942+0000 7f3114a00700 1 -- 172.21.15.28:0/28046845 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3100010430 con 0x7f3110104ee0 2024-04-26T01:39:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f3114a00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f30f00a3c40 0x7f30f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f31206f3b80 1 Processor -- start 2024-04-26T01:39:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f3114a00700 1 -- 172.21.15.28:0/28046845 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f31000d05d0 con 0x7f3110104ee0 2024-04-26T01:39:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f31206f3b80 1 -- start start 2024-04-26T01:39:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f3107e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f30f00a3c40 0x7f30f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27225d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c272268570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27226aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f31206f3b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2721b0e40 con 0x55c27225d2f0 2024-04-26T01:39:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f31206f3b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c2721576e0 con 0x55c272268ab0 2024-04-26T01:39:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.944+0000 7f31206f3b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c2721ace00 con 0x55c27225dc10 2024-04-26T01:39:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27225d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27225d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37036/0 (socket says 172.21.15.28:37036) 2024-04-26T01:39:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310ea00700 1 -- 172.21.15.28:0/1933500707 learned_addr learned my addr 172.21.15.28:0/1933500707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310f400700 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27226aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f3107e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f30f00a3c40 0x7f30f00a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f311010b580 tx=0x7f30f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f3107e00700 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c272268570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310ea00700 1 -- 172.21.15.28:0/1933500707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 msgr2=0x55c272268570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310ea00700 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c272268570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310ea00700 1 -- 172.21.15.28:0/1933500707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 msgr2=0x55c27226aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310ea00700 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27226aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.945+0000 7f310ea00700 1 -- 172.21.15.28:0/1933500707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c27214d630 con 0x55c27225d2f0 2024-04-26T01:39:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.946+0000 7f310ea00700 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27225d6d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f310009c7f0 tx=0x7f310009cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560045707f9e02d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.946+0000 7f3104c00700 1 -- 172.21.15.28:0/1933500707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000a0c10 con 0x55c27225d2f0 2024-04-26T01:39:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.946+0000 7f3104c00700 1 -- 172.21.15.28:0/1933500707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31000a54d0 con 0x55c27225d2f0 2024-04-26T01:39:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.946+0000 7f3104c00700 1 -- 172.21.15.28:0/1933500707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000a5040 con 0x55c27225d2f0 2024-04-26T01:39:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 -- 172.21.15.28:0/1933500707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 msgr2=0x55c27225d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27225d6d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f310009c7f0 tx=0x7f310009cf60 comp rx=0 tx=0).stop 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 -- 172.21.15.28:0/1933500707 shutdown_connections 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27226aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27225d6d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 --2- 172.21.15.28:0/1933500707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c272268570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 -- 172.21.15.28:0/1933500707 >> 172.21.15.28:0/1933500707 conn(0x55c27218c550 msgr2=0x55c272259790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 -- 172.21.15.28:0/1933500707 shutdown_connections 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 -- 172.21.15.28:0/1933500707 wait complete. 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.947+0000 7f31206f3b80 1 -- start start 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.948+0000 7f31206f3b80 1 Processor -- start 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.948+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c272268260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.948+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225dc10 0x55c272265560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.948+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c272265aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.948+0000 7f31206f3b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c27226b5a0 con 0x55c27225dc10 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.948+0000 7f31206f3b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c27216a810 con 0x55c272268ab0 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.948+0000 7f31206f3b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c27226b720 con 0x55c27225d2f0 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.948+0000 7f31206f3b80 10 client.0 _ll_register_callbacks cb 0x55c2721496d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f3107e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225dc10 0x55c272265560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310ea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c272268260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310ea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c272268260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54066/0 (socket says 172.21.15.28:54066) 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310f400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c272265aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310ea00700 1 -- 172.21.15.28:0/3745479318 learned_addr learned my addr 172.21.15.28:0/3745479318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f3107e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225dc10 0x55c272265560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37046/0 (socket says 172.21.15.28:37046) 2024-04-26T01:39:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310f400700 1 -- 172.21.15.28:0/3745479318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 msgr2=0x55c272268260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310f400700 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c272268260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310f400700 1 -- 172.21.15.28:0/3745479318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225dc10 msgr2=0x55c272265560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310f400700 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225dc10 0x55c272265560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310f400700 1 -- 172.21.15.28:0/3745479318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2720fe880 con 0x55c272268ab0 2024-04-26T01:39:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f3107e00700 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225dc10 0x55c272265560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.949+0000 7f310ea00700 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c272268260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f310f400700 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c272265aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3108009fd0 tx=0x7f310800be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81c14461e7ddf952 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f30fd600700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f30fd600700 1 -- 172.21.15.28:0/3745479318 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108003c00 con 0x55c272268ab0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f31206f3b80 1 -- 172.21.15.28:0/3745479318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31000a54d0 con 0x55c272268ab0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f30fd600700 1 -- 172.21.15.28:0/3745479318 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f3108003da0 con 0x55c272268ab0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f30fd600700 1 -- 172.21.15.28:0/3745479318 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f31080074e0 con 0x55c272268ab0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f31206f3b80 10 client.24875 fetch_fsmap learned FSMap version 24 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f31206f3b80 1 -- 172.21.15.28:0/3745479318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c272147aa0 con 0x55c272268ab0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.950+0000 7f30fd600700 1 -- 172.21.15.28:0/3745479318 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31000a54d0 con 0x55c272268ab0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.951+0000 7f30fd600700 1 -- 172.21.15.28:0/3745479318 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3108011070 con 0x55c272268ab0 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.951+0000 7f31206f3b80 10 client.24875 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.951+0000 7f31206f3b80 10 client.24875 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.951+0000 7f31206f3b80 20 client.24875 populate_metadata read hostname 'smithi028' 2024-04-26T01:39:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.951+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x55c2722782c0 0x55c272279730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.951+0000 7f31206f3b80 4 client.24875 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:39:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.951+0000 7f31206f3b80 1 -- 172.21.15.28:0/3745479318 --> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2721be630 con 0x55c2722782c0 2024-04-26T01:39:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.951+0000 7f310ea00700 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x55c2722782c0 0x55c272279730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.952+0000 7f310ea00700 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x55c2722782c0 0x55c272279730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24dd6ca5f69b0319 server_cookie=4c85ae2a86c56ce0 in_seq=0 out_seq=0 2024-04-26T01:39:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.952+0000 7f30fd600700 10 client.24875 ms_handle_connect on v2:172.21.15.53:6838/955096275 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f30fd600700 1 -- 172.21.15.28:0/3745479318 <== mds.1 v2:172.21.15.53:6838/955096275 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c2721be630 con 0x55c2722782c0 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f30fd600700 10 client.24875 handle_command_reply: tid=0 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 client.24875 shutdown 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 -- 172.21.15.28:0/3745479318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 msgr2=0x55c272265aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c272265aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3108009fd0 tx=0x7f310800be20 comp rx=0 tx=0).stop 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 -- 172.21.15.28:0/3745479318 shutdown_connections 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c272265aa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c272268260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x55c2722782c0 0x55c272279730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3745479318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225dc10 0x55c272265560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f30fd600700 0 client.24875 ms_handle_reset on v2:172.21.15.53:6838/955096275 2024-04-26T01:39:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 -- 172.21.15.28:0/3745479318 >> 172.21.15.28:0/3745479318 conn(0x55c27218c550 msgr2=0x55c27225fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.953+0000 7f31206f3b80 1 -- 172.21.15.28:0/3745479318 shutdown_connections 2024-04-26T01:39:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.954+0000 7f31206f3b80 1 -- 172.21.15.28:0/3745479318 wait complete. 2024-04-26T01:39:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.954+0000 7f31206f3b80 20 client.24875 trim_cache size 0 max 16384 2024-04-26T01:39:51.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:51 smithi053 ceph-mon[32593]: pgmap v423: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:39:51.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f31206f3b80 1 Processor -- start 2024-04-26T01:39:51.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f31206f3b80 1 -- start start 2024-04-26T01:39:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c2722644c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c27225dc10 0x55c2722598c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c272268ab0 0x55c2722627e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f31206f3b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c272277a70 con 0x55c272268ab0 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f31206f3b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c272278140 con 0x55c27225dc10 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f31206f3b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c272278810 con 0x55c27225d2f0 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f310ea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c2722644c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f3107e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c27225dc10 0x55c2722598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f310f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c272268ab0 0x55c2722627e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f3107e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c27225dc10 0x55c2722598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:49702/0 (socket says 172.21.15.28:49702) 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f3107e00700 1 -- 172.21.15.28:0/3260188303 learned_addr learned my addr 172.21.15.28:0/3260188303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.977+0000 7f310ea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c2722644c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54082/0 (socket says 172.21.15.28:54082) 2024-04-26T01:39:51.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f310ea00700 1 -- 172.21.15.28:0/3260188303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c27225dc10 msgr2=0x55c2722598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f310ea00700 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c27225dc10 0x55c2722598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f310ea00700 1 -- 172.21.15.28:0/3260188303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c272268ab0 msgr2=0x55c2722627e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f310ea00700 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c272268ab0 0x55c2722627e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f310ea00700 1 -- 172.21.15.28:0/3260188303 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c27227a690 con 0x55c27225d2f0 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f310f400700 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c272268ab0 0x55c2722627e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f3107e00700 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c27225dc10 0x55c2722598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f310ea00700 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c2722644c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f310009d110 tx=0x7f3100098440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aabd6229c3fc58d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f30ffe00700 1 -- 172.21.15.28:0/3260188303 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000a0c10 con 0x55c27225d2f0 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.978+0000 7f30ffe00700 1 -- 172.21.15.28:0/3260188303 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f310000cd60 con 0x55c27225d2f0 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f30ffe00700 1 -- 172.21.15.28:0/3260188303 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310009db90 con 0x55c27225d2f0 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 -- 172.21.15.28:0/3260188303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 msgr2=0x55c2722644c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c2722644c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f310009d110 tx=0x7f3100098440 comp rx=0 tx=0).stop 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 -- 172.21.15.28:0/3260188303 shutdown_connections 2024-04-26T01:39:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c272268ab0 0x55c2722627e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225d2f0 0x55c2722644c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3260188303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c27225dc10 0x55c2722598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 -- 172.21.15.28:0/3260188303 >> 172.21.15.28:0/3260188303 conn(0x55c27218c550 msgr2=0x55c2722591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 -- 172.21.15.28:0/3260188303 shutdown_connections 2024-04-26T01:39:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.979+0000 7f31206f3b80 1 -- 172.21.15.28:0/3260188303 wait complete. 2024-04-26T01:39:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.980+0000 7f31206f3b80 1 -- start start 2024-04-26T01:39:51.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.980+0000 7f31206f3b80 1 Processor -- start 2024-04-26T01:39:51.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27227e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c27227ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f31206f3b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27227f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f31206f3b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2722795b0 con 0x55c27225d2f0 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f31206f3b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c272278ee0 con 0x55c272268ab0 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f31206f3b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c272279c80 con 0x55c27225dc10 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f31206f3b80 10 client.0 _ll_register_callbacks cb 0x55c2721496d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f310ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27227e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f310ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27227e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37062/0 (socket says 172.21.15.28:37062) 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f310ea00700 1 -- 172.21.15.28:0/3422193370 learned_addr learned my addr 172.21.15.28:0/3422193370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f3107e00700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c27227ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f310f400700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27227f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f3107e00700 1 -- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 msgr2=0x55c27227f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f3107e00700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27227f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f3107e00700 1 -- 172.21.15.28:0/3422193370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 msgr2=0x55c27227e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.981+0000 7f3107e00700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27227e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.982+0000 7f3107e00700 1 -- 172.21.15.28:0/3422193370 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2722de650 con 0x55c27225dc10 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.982+0000 7f310ea00700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27227e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.982+0000 7f310f400700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27227f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.982+0000 7f3107e00700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c27227ecc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55c27227a570 tx=0x7f30f80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4941782c2d6c9275 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.982+0000 7f30fd600700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.982+0000 7f30fd600700 1 -- 172.21.15.28:0/3422193370 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f802e060 con 0x55c27225dc10 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.982+0000 7f31206f3b80 1 -- 172.21.15.28:0/3422193370 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f310000cd60 con 0x55c27225dc10 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.982+0000 7f30fd600700 1 -- 172.21.15.28:0/3422193370 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f30f8004320 con 0x55c27225dc10 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f30fd600700 1 -- 172.21.15.28:0/3422193370 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f30f8004520 con 0x55c27225dc10 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f31206f3b80 10 client.24922 fetch_fsmap learned FSMap version 24 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f31206f3b80 1 -- 172.21.15.28:0/3422193370 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c2722df9d0 con 0x55c27225dc10 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f30fd600700 1 -- 172.21.15.28:0/3422193370 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f310000cd60 con 0x55c27225dc10 2024-04-26T01:39:51.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f30fd600700 1 -- 172.21.15.28:0/3422193370 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f30f8028030 con 0x55c27225dc10 2024-04-26T01:39:51.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f31206f3b80 10 client.24922 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:39:51.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f31206f3b80 10 client.24922 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:39:51.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f31206f3b80 20 client.24922 populate_metadata read hostname 'smithi028' 2024-04-26T01:39:51.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x55c2722e4570 0x55c2722e3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:51.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f31206f3b80 4 client.24922 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:39:51.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.983+0000 7f31206f3b80 1 -- 172.21.15.28:0/3422193370 --> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] -- command(tid 0: {"prefix": "status"}) -- 0x55c2721b86d0 con 0x55c2722e4570 2024-04-26T01:39:51.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.984+0000 7f310ea00700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x55c2722e4570 0x55c2722e3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:51.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.984+0000 7f310ea00700 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x55c2722e4570 0x55c2722e3280 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=560304b9684ba8f2 server_cookie=fcad7f7ad2669fe0 in_seq=0 out_seq=0 2024-04-26T01:39:51.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.984+0000 7f30fd600700 10 client.24922 ms_handle_connect on v2:172.21.15.53:6838/955096275 2024-04-26T01:39:51.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.984+0000 7f30fd600700 1 -- 172.21.15.28:0/3422193370 <== mds.1 v2:172.21.15.53:6838/955096275 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55c2721b86d0 con 0x55c2722e4570 2024-04-26T01:39:51.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.984+0000 7f30fd600700 10 client.24922 handle_command_reply: tid=0 2024-04-26T01:39:51.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.984+0000 7f31206f3b80 1 client.24922 shutdown 2024-04-26T01:39:51.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 -- 172.21.15.28:0/3422193370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 msgr2=0x55c27227ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c27227ecc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55c27227a570 tx=0x7f30f80047c0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 -- 172.21.15.28:0/3422193370 shutdown_connections 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c272268ab0 0x55c27227f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x55c2722e4570 0x55c2722e3280 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27225d2f0 0x55c27227e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 --2- 172.21.15.28:0/3422193370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c27225dc10 0x55c27227ecc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f30fd600700 0 client.24922 ms_handle_reset on v2:172.21.15.53:6838/955096275 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 -- 172.21.15.28:0/3422193370 >> 172.21.15.28:0/3422193370 conn(0x55c27218c550 msgr2=0x55c272267620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 -- 172.21.15.28:0/3422193370 shutdown_connections 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 1 -- 172.21.15.28:0/3422193370 wait complete. 2024-04-26T01:39:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.985+0000 7f31206f3b80 20 client.24922 trim_cache size 0 max 16384 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24509, 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 158.60934648, 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T01:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 196.12479721899999 2024-04-26T01:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:39:51.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 -- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f30f00a3c40 msgr2=0x7f30f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f30f00a3c40 0x7f30f00a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f311010b580 tx=0x7f30f8009260 comp rx=0 tx=0).stop 2024-04-26T01:39:51.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 -- 172.21.15.28:0/28046845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3110104ee0 msgr2=0x7f311019b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3110104ee0 0x7f311019b370 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f310000e9a0 tx=0x7f310000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 -- 172.21.15.28:0/28046845 shutdown_connections 2024-04-26T01:39:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f30f00a3c40 0x7f30f00a6100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3110109fc0 0x7f311019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3110104ee0 0x7f311019b370 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 --2- 172.21.15.28:0/28046845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110105890 0x7f311019b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 -- 172.21.15.28:0/28046845 >> 172.21.15.28:0/28046845 conn(0x7f3110100770 msgr2=0x7f3110101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 -- 172.21.15.28:0/28046845 shutdown_connections 2024-04-26T01:39:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:51.986+0000 7f3115e00700 1 -- 172.21.15.28:0/28046845 wait complete. 2024-04-26T01:39:52.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:39:53.309 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:53.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:53 smithi028 ceph-mon[28247]: pgmap v424: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:53.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:53 smithi169 ceph-mon[32851]: pgmap v424: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:53 smithi053 ceph-mon[32593]: pgmap v424: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:39:54.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:54 smithi169 ceph-mon[32851]: pgmap v425: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:54.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:54 smithi053 ceph-mon[32593]: pgmap v425: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:55.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:54 smithi028 ceph-mon[28247]: pgmap v425: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:55.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.596+0000 7fc647a00700 1 -- 172.21.15.28:0/2548376824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc640109c80 msgr2=0x7fc64010dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.596+0000 7fc647a00700 1 --2- 172.21.15.28:0/2548376824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc640109c80 0x7fc64010dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0099c0 tx=0x7fc62c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.597+0000 7fc647a00700 1 -- 172.21.15.28:0/2548376824 shutdown_connections 2024-04-26T01:39:55.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.597+0000 7fc647a00700 1 --2- 172.21.15.28:0/2548376824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc64010e3b0 0x7fc640115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.597+0000 7fc647a00700 1 --2- 172.21.15.28:0/2548376824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6401092d0 0x7fc6401096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.597+0000 7fc647a00700 1 --2- 172.21.15.28:0/2548376824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc640109c80 0x7fc64010dcd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.597+0000 7fc647a00700 1 -- 172.21.15.28:0/2548376824 >> 172.21.15.28:0/2548376824 conn(0x7fc640076210 msgr2=0x7fc640107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:55.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.597+0000 7fc647a00700 1 -- 172.21.15.28:0/2548376824 shutdown_connections 2024-04-26T01:39:55.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.597+0000 7fc647a00700 1 -- 172.21.15.28:0/2548376824 wait complete. 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.598+0000 7fc647a00700 1 Processor -- start 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.598+0000 7fc647a00700 1 -- start start 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.598+0000 7fc647a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401092d0 0x7fc64019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc647a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc640109c80 0x7fc64019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc647a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc64010e3b0 0x7fc6401a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc647a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc640119220 con 0x7fc64010e3b0 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc647a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401190a0 con 0x7fc6401092d0 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc647a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401193a0 con 0x7fc640109c80 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401092d0 0x7fc64019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401092d0 0x7fc64019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:49726/0 (socket says 172.21.15.28:49726) 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645000700 1 -- 172.21.15.28:0/3604137088 learned_addr learned my addr 172.21.15.28:0/3604137088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc63fe00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc640109c80 0x7fc64019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc64010e3b0 0x7fc6401a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645000700 1 -- 172.21.15.28:0/3604137088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc640109c80 msgr2=0x7fc64019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645000700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc640109c80 0x7fc64019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645000700 1 -- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc64010e3b0 msgr2=0x7fc6401a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645000700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc64010e3b0 0x7fc6401a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.599+0000 7fc645000700 1 -- 172.21.15.28:0/3604137088 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6401a4760 con 0x7fc6401092d0 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.600+0000 7fc645000700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401092d0 0x7fc64019f750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc63400d950 tx=0x7fc63400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.600+0000 7fc63d600700 1 -- 172.21.15.28:0/3604137088 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634016070 con 0x7fc6401092d0 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.600+0000 7fc63d600700 1 -- 172.21.15.28:0/3604137088 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc634009df0 con 0x7fc6401092d0 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.600+0000 7fc647a00700 1 -- 172.21.15.28:0/3604137088 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6401a4a50 con 0x7fc6401092d0 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.600+0000 7fc63d600700 1 -- 172.21.15.28:0/3604137088 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63400f3c0 con 0x7fc6401092d0 2024-04-26T01:39:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.600+0000 7fc647a00700 1 -- 172.21.15.28:0/3604137088 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6401ac2d0 con 0x7fc6401092d0 2024-04-26T01:39:55.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.603+0000 7fc63d600700 1 -- 172.21.15.28:0/3604137088 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc634019070 con 0x7fc6401092d0 2024-04-26T01:39:55.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.605+0000 7fc652377b80 1 Processor -- start 2024-04-26T01:39:55.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.605+0000 7fc652377b80 1 -- start start 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.605+0000 7fc652377b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.605+0000 7fc652377b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a40a0 0x55f32379d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.605+0000 7fc652377b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.605+0000 7fc652377b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3236efe40 con 0x55f3237a3780 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.605+0000 7fc652377b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f3236966e0 con 0x55f3237a40a0 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.605+0000 7fc652377b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f3236ebe00 con 0x55f32379def0 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a40a0 0x55f32379d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a40a0 0x55f32379d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:49738/0 (socket says 172.21.15.28:49738) 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645000700 1 -- 172.21.15.28:0/2100721886 learned_addr learned my addr 172.21.15.28:0/2100721886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645a00700 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc63d600700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6280a3c40 0x7fc6280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc63fe00700 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645000700 1 -- 172.21.15.28:0/2100721886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 msgr2=0x55f32379e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:55.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645000700 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645000700 1 -- 172.21.15.28:0/2100721886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 msgr2=0x55f3237a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:55.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645000700 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc645000700 1 -- 172.21.15.28:0/2100721886 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f32368c630 con 0x55f3237a40a0 2024-04-26T01:39:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.606+0000 7fc63fe00700 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:39:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.607+0000 7fc645000700 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a40a0 0x55f32379d9b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc6340987a0 tx=0x7fc63409be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ee79e13f50b98e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.607+0000 7fc645a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6280a3c40 0x7fc6280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.607+0000 7fc63d600700 1 -- 172.21.15.28:0/3604137088 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fc6340d00b0 con 0x7fc6401092d0 2024-04-26T01:39:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.607+0000 7fc632000700 1 -- 172.21.15.28:0/2100721886 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63409ca40 con 0x55f3237a40a0 2024-04-26T01:39:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.607+0000 7fc632000700 1 -- 172.21.15.28:0/2100721886 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6340a4fb0 con 0x55f3237a40a0 2024-04-26T01:39:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.607+0000 7fc632000700 1 -- 172.21.15.28:0/2100721886 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634010710 con 0x55f3237a40a0 2024-04-26T01:39:55.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 -- 172.21.15.28:0/2100721886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a40a0 msgr2=0x55f32379d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc645a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6280a3c40 0x7fc6280a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc638009530 tx=0x7fc638009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:55.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a40a0 0x55f32379d9b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc6340987a0 tx=0x7fc63409be80 comp rx=0 tx=0).stop 2024-04-26T01:39:55.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 -- 172.21.15.28:0/2100721886 shutdown_connections 2024-04-26T01:39:55.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 --2- 172.21.15.28:0/2100721886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a40a0 0x55f32379d9b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 -- 172.21.15.28:0/2100721886 >> 172.21.15.28:0/2100721886 conn(0x55f3236cbf00 msgr2=0x55f3237988a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 -- 172.21.15.28:0/2100721886 shutdown_connections 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.608+0000 7fc652377b80 1 -- 172.21.15.28:0/2100721886 wait complete. 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.609+0000 7fc652377b80 1 -- start start 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.609+0000 7fc652377b80 1 Processor -- start 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.609+0000 7fc652377b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32379def0 0x55f3237a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.609+0000 7fc652377b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3237a3780 0x55f3237a6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.609+0000 7fc652377b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 0x55f3237a7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.609+0000 7fc652377b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3237aa920 con 0x55f32379def0 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.609+0000 7fc652377b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f3236a9810 con 0x55f3237a7510 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.609+0000 7fc652377b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f3237aaaa0 con 0x55f3237a3780 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc652377b80 10 client.0 _ll_register_callbacks cb 0x55f3236886d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:39:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32379def0 0x55f3237a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32379def0 0x55f3237a9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37094/0 (socket says 172.21.15.28:37094) 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645000700 1 -- 172.21.15.28:0/529079625 learned_addr learned my addr 172.21.15.28:0/529079625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc63fe00700 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3237a3780 0x55f3237a6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645a00700 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 0x55f3237a7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645000700 1 -- 172.21.15.28:0/529079625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3237a3780 msgr2=0x55f3237a6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645000700 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3237a3780 0x55f3237a6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645000700 1 -- 172.21.15.28:0/529079625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 msgr2=0x55f3237a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645000700 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 0x55f3237a7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.610+0000 7fc645000700 1 -- 172.21.15.28:0/529079625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f32363d880 con 0x55f32379def0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc645000700 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32379def0 0x55f3237a9cd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc63409a5f0 tx=0x7fc6340d0330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a026e3243191506 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc626a00700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc626a00700 1 -- 172.21.15.28:0/529079625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634020070 con 0x55f32379def0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc626a00700 1 -- 172.21.15.28:0/529079625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fc634010710 con 0x55f32379def0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc652377b80 1 -- 172.21.15.28:0/529079625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6340a4fb0 con 0x55f32379def0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc626a00700 1 -- 172.21.15.28:0/529079625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc63409c340 con 0x55f32379def0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc652377b80 10 client.14790 fetch_fsmap learned FSMap version 24 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc652377b80 1 -- 172.21.15.28:0/529079625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f323686aa0 con 0x55f32379def0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.611+0000 7fc626a00700 1 -- 172.21.15.28:0/529079625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6340a4fb0 con 0x55f32379def0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.612+0000 7fc626a00700 1 -- 172.21.15.28:0/529079625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fc6340a4b20 con 0x55f32379def0 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.612+0000 7fc652377b80 10 client.14790 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:39:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.612+0000 7fc652377b80 10 client.14790 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.612+0000 7fc652377b80 20 client.14790 populate_metadata read hostname 'smithi028' 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.612+0000 7fc652377b80 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3237b7350 0x55f3237b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.612+0000 7fc652377b80 4 client.14790 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.612+0000 7fc652377b80 1 -- 172.21.15.28:0/529079625 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3236fd630 con 0x55f3237b7350 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.612+0000 7fc63fe00700 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3237b7350 0x55f3237b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.613+0000 7fc63fe00700 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3237b7350 0x55f3237b87c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a1dda88137688d1 server_cookie=8f03b206bb2d279e in_seq=0 out_seq=0 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.613+0000 7fc626a00700 10 client.14790 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc626a00700 1 -- 172.21.15.28:0/529079625 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3236fd630 con 0x55f3237b7350 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc626a00700 10 client.14790 handle_command_reply: tid=0 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 client.14790 shutdown 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 -- 172.21.15.28:0/529079625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32379def0 msgr2=0x55f3237a9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32379def0 0x55f3237a9cd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc63409a5f0 tx=0x7fc6340d0330 comp rx=0 tx=0).stop 2024-04-26T01:39:55.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 -- 172.21.15.28:0/529079625 shutdown_connections 2024-04-26T01:39:55.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 0x55f3237a7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32379def0 0x55f3237a9cd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3237b7350 0x55f3237b87c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 --2- 172.21.15.28:0/529079625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3237a3780 0x55f3237a6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 -- 172.21.15.28:0/529079625 >> 172.21.15.28:0/529079625 conn(0x55f3236cbf00 msgr2=0x55f323797e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc626a00700 0 client.14790 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:39:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.614+0000 7fc652377b80 1 -- 172.21.15.28:0/529079625 shutdown_connections 2024-04-26T01:39:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.615+0000 7fc652377b80 1 -- 172.21.15.28:0/529079625 wait complete. 2024-04-26T01:39:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.615+0000 7fc652377b80 20 client.14790 trim_cache size 0 max 16384 2024-04-26T01:39:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.637+0000 7fc652377b80 1 Processor -- start 2024-04-26T01:39:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc652377b80 1 -- start start 2024-04-26T01:39:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc652377b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc652377b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a3780 0x55f3237a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc652377b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a7510 0x55f3237985d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc652377b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3237b6b00 con 0x55f3237a7510 2024-04-26T01:39:55.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc652377b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f3237b71d0 con 0x55f3237a3780 2024-04-26T01:39:55.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc652377b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f3237b78a0 con 0x55f32379def0 2024-04-26T01:39:55.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc645000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc645000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54136/0 (socket says 172.21.15.28:54136) 2024-04-26T01:39:55.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc645000700 1 -- 172.21.15.28:0/2584053787 learned_addr learned my addr 172.21.15.28:0/2584053787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:55.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.638+0000 7fc645a00700 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a7510 0x55f3237985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc63fe00700 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a3780 0x55f3237a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc645000700 1 -- 172.21.15.28:0/2584053787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a3780 msgr2=0x55f3237a6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc645000700 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a3780 0x55f3237a6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc645000700 1 -- 172.21.15.28:0/2584053787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a7510 msgr2=0x55f3237985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc645000700 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a7510 0x55f3237985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc645000700 1 -- 172.21.15.28:0/2584053787 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3237a9910 con 0x55f32379def0 2024-04-26T01:39:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc645000700 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379d9b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc6340d3ee0 tx=0x7fc63409a850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d95a61fc1447b648 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc631600700 1 -- 172.21.15.28:0/2584053787 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634020070 con 0x55f32379def0 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc631600700 1 -- 172.21.15.28:0/2584053787 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6340a4fb0 con 0x55f32379def0 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.639+0000 7fc631600700 1 -- 172.21.15.28:0/2584053787 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6340a06f0 con 0x55f32379def0 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.640+0000 7fc652377b80 1 -- 172.21.15.28:0/2584053787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 msgr2=0x55f32379d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.640+0000 7fc652377b80 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379d9b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc6340d3ee0 tx=0x7fc63409a850 comp rx=0 tx=0).stop 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.640+0000 7fc652377b80 1 -- 172.21.15.28:0/2584053787 shutdown_connections 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.640+0000 7fc652377b80 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a7510 0x55f3237985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.640+0000 7fc652377b80 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f32379d9b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.640+0000 7fc652377b80 1 --2- 172.21.15.28:0/2584053787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a3780 0x55f3237a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.640+0000 7fc652377b80 1 -- 172.21.15.28:0/2584053787 >> 172.21.15.28:0/2584053787 conn(0x55f3236cbf00 msgr2=0x55f323794ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.640+0000 7fc652377b80 1 -- 172.21.15.28:0/2584053787 shutdown_connections 2024-04-26T01:39:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.641+0000 7fc652377b80 1 -- 172.21.15.28:0/2584053787 wait complete. 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.641+0000 7fc652377b80 1 -- start start 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.641+0000 7fc652377b80 1 Processor -- start 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc652377b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f3237c1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc652377b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237bcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc652377b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 0x55f3237bd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc652377b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3237b8640 con 0x55f3237a3780 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc652377b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f3237b7f70 con 0x55f3237a7510 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc652377b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f3237b8d10 con 0x55f32379def0 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc652377b80 10 client.0 _ll_register_callbacks cb 0x55f3236886d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc63fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237bcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc63fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237bcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37110/0 (socket says 172.21.15.28:37110) 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc63fe00700 1 -- 172.21.15.28:0/1684099896 learned_addr learned my addr 172.21.15.28:0/1684099896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc645a00700 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 0x55f3237bd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.642+0000 7fc645000700 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f3237c1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc63fe00700 1 -- 172.21.15.28:0/1684099896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 msgr2=0x55f3237c1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc63fe00700 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f3237c1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc63fe00700 1 -- 172.21.15.28:0/1684099896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 msgr2=0x55f3237bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:39:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc63fe00700 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 0x55f3237bd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc63fe00700 1 -- 172.21.15.28:0/1684099896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3237bd6f0 con 0x55f3237a3780 2024-04-26T01:39:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc63fe00700 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237bcbf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0099c0 tx=0x7fc62c0051f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b7fa99e26bf7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc626a00700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:39:55.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc626a00700 1 -- 172.21.15.28:0/1684099896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0055b0 con 0x55f3237a3780 2024-04-26T01:39:55.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc626a00700 1 -- 172.21.15.28:0/1684099896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fc62c0037c0 con 0x55f3237a3780 2024-04-26T01:39:55.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.643+0000 7fc652377b80 1 -- 172.21.15.28:0/1684099896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6340a4fb0 con 0x55f3237a3780 2024-04-26T01:39:55.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc626a00700 1 -- 172.21.15.28:0/1684099896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc62c0039c0 con 0x55f3237a3780 2024-04-26T01:39:55.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc652377b80 10 client.14796 fetch_fsmap learned FSMap version 24 2024-04-26T01:39:55.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc652377b80 1 -- 172.21.15.28:0/1684099896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f32381f2c0 con 0x55f3237a3780 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc626a00700 1 -- 172.21.15.28:0/1684099896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6340a4fb0 con 0x55f3237a3780 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc626a00700 1 -- 172.21.15.28:0/1684099896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fc62c029130 con 0x55f3237a3780 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc652377b80 10 client.14796 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc652377b80 10 client.14796 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc652377b80 20 client.14796 populate_metadata read hostname 'smithi028' 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc652377b80 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f323823eb0 0x55f323822bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc652377b80 4 client.14796 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.644+0000 7fc652377b80 1 -- 172.21.15.28:0/1684099896 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "status"}) -- 0x55f3236f76d0 con 0x55f323823eb0 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.645+0000 7fc645000700 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f323823eb0 0x55f323822bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.645+0000 7fc645000700 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f323823eb0 0x55f323822bc0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2991ec81b0354fdf server_cookie=c41121bd713de51 in_seq=0 out_seq=0 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.645+0000 7fc626a00700 10 client.14796 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc626a00700 1 -- 172.21.15.28:0/1684099896 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f3236f76d0 con 0x55f323823eb0 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc626a00700 10 client.14796 handle_command_reply: tid=0 2024-04-26T01:39:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 client.14796 shutdown 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 -- 172.21.15.28:0/1684099896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 msgr2=0x55f3237bcbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237bcbf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0099c0 tx=0x7fc62c0051f0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 -- 172.21.15.28:0/1684099896 shutdown_connections 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3237a7510 0x55f3237bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f323823eb0 0x55f323822bc0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f32379def0 0x55f3237c1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc626a00700 0 client.14796 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 --2- 172.21.15.28:0/1684099896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3237a3780 0x55f3237bcbf0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 -- 172.21.15.28:0/1684099896 >> 172.21.15.28:0/1684099896 conn(0x55f3236cbf00 msgr2=0x55f3237a6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 -- 172.21.15.28:0/1684099896 shutdown_connections 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 1 -- 172.21.15.28:0/1684099896 wait complete. 2024-04-26T01:39:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.646+0000 7fc652377b80 20 client.14796 trim_cache size 0 max 16384 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24523, 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 178.35227162499999, 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T01:39:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 213.562564047 2024-04-26T01:39:55.647 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:39:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 -- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6280a3c40 msgr2=0x7fc6280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6280a3c40 0x7fc6280a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc638009530 tx=0x7fc638009040 comp rx=0 tx=0).stop 2024-04-26T01:39:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 -- 172.21.15.28:0/3604137088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401092d0 msgr2=0x7fc64019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401092d0 0x7fc64019f750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc63400d950 tx=0x7fc63400dc60 comp rx=0 tx=0).stop 2024-04-26T01:39:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 -- 172.21.15.28:0/3604137088 shutdown_connections 2024-04-26T01:39:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc6280a3c40 0x7fc6280a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc64010e3b0 0x7fc6401a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401092d0 0x7fc64019f750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 --2- 172.21.15.28:0/3604137088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc640109c80 0x7fc64019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 -- 172.21.15.28:0/3604137088 >> 172.21.15.28:0/3604137088 conn(0x7fc640076210 msgr2=0x7fc6400768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.647+0000 7fc647a00700 1 -- 172.21.15.28:0/3604137088 shutdown_connections 2024-04-26T01:39:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:55.648+0000 7fc647a00700 1 -- 172.21.15.28:0/3604137088 wait complete. 2024-04-26T01:39:56.492 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T01:39:56.492 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T01:39:56.503 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:39:56.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T01:39:56.955 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:57.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:56 smithi169 ceph-mon[32851]: pgmap v426: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:56 smithi053 ceph-mon[32593]: pgmap v426: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:57.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:56 smithi028 ceph-mon[28247]: pgmap v426: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:39:58.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:58 smithi028 ceph-mon[28247]: pgmap v427: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:59.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:58 smithi169 ceph-mon[32851]: pgmap v427: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:59.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:58 smithi053 ceph-mon[32593]: pgmap v427: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:39:59.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.286+0000 7fce16800700 1 -- 172.21.15.28:0/1916282332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce1010a9b0 msgr2=0x7fce1010ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.286+0000 7fce16800700 1 --2- 172.21.15.28:0/1916282332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce1010a9b0 0x7fce1010ce40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fce08009f90 tx=0x7fce0800b330 comp rx=0 tx=0).stop 2024-04-26T01:39:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.286+0000 7fce16800700 1 -- 172.21.15.28:0/1916282332 shutdown_connections 2024-04-26T01:39:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.286+0000 7fce16800700 1 --2- 172.21.15.28:0/1916282332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce1010a9b0 0x7fce1010ce40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.286+0000 7fce16800700 1 --2- 172.21.15.28:0/1916282332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce10076a70 0x7fce10074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.286+0000 7fce16800700 1 --2- 172.21.15.28:0/1916282332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce100754a0 0x7fce10075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.286+0000 7fce16800700 1 -- 172.21.15.28:0/1916282332 >> 172.21.15.28:0/1916282332 conn(0x7fce100fe080 msgr2=0x7fce101004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.287+0000 7fce16800700 1 -- 172.21.15.28:0/1916282332 shutdown_connections 2024-04-26T01:39:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.287+0000 7fce16800700 1 -- 172.21.15.28:0/1916282332 wait complete. 2024-04-26T01:39:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce16800700 1 Processor -- start 2024-04-26T01:39:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce16800700 1 -- start start 2024-04-26T01:39:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce16800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce100754a0 0x7fce1019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce16800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce10076a70 0x7fce1019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce16800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce1010a9b0 0x7fce101a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce16800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce10110770 con 0x7fce100754a0 2024-04-26T01:39:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce16800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fce101105f0 con 0x7fce1010a9b0 2024-04-26T01:39:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce16800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fce101108f0 con 0x7fce10076a70 2024-04-26T01:39:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.288+0000 7fce0f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce100754a0 0x7fce1019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce100754a0 0x7fce1019ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34826/0 (socket says 172.21.15.28:34826) 2024-04-26T01:39:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0f400700 1 -- 172.21.15.28:0/2859280868 learned_addr learned my addr 172.21.15.28:0/2859280868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:39:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce07e00700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce10076a70 0x7fce1019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0fe00700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce1010a9b0 0x7fce101a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0f400700 1 -- 172.21.15.28:0/2859280868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce10076a70 msgr2=0x7fce1019b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0f400700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce10076a70 0x7fce1019b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0f400700 1 -- 172.21.15.28:0/2859280868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce1010a9b0 msgr2=0x7fce101a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0f400700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce1010a9b0 0x7fce101a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0f400700 1 -- 172.21.15.28:0/2859280868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce101a29b0 con 0x7fce100754a0 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce07e00700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce10076a70 0x7fce1019b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.289+0000 7fce0f400700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce100754a0 0x7fce1019ad50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fce0000e9a0 tx=0x7fce0000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.290+0000 7fce0cc00700 1 -- 172.21.15.28:0/2859280868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00004a30 con 0x7fce100754a0 2024-04-26T01:39:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.290+0000 7fce0cc00700 1 -- 172.21.15.28:0/2859280868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce0000ccc0 con 0x7fce100754a0 2024-04-26T01:39:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.290+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce101a2ca0 con 0x7fce100754a0 2024-04-26T01:39:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.290+0000 7fce0cc00700 1 -- 172.21.15.28:0/2859280868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00005610 con 0x7fce100754a0 2024-04-26T01:39:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.290+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce101a3190 con 0x7fce100754a0 2024-04-26T01:39:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.291+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce10069730 con 0x7fce100754a0 2024-04-26T01:39:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.293+0000 7fce0cc00700 1 -- 172.21.15.28:0/2859280868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fce0000c810 con 0x7fce100754a0 2024-04-26T01:39:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.296+0000 7fce0cc00700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcdf40a3c40 0x7fcdf40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.296+0000 7fce0cc00700 1 -- 172.21.15.28:0/2859280868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fce000d05a0 con 0x7fce100754a0 2024-04-26T01:39:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.296+0000 7fce07e00700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcdf40a3c40 0x7fcdf40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.297+0000 7fce07e00700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcdf40a3c40 0x7fcdf40a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009750 tx=0x7fcdf8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.301+0000 7fce0cc00700 1 -- 172.21.15.28:0/2859280868 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce0009cf30 con 0x7fce100754a0 2024-04-26T01:39:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.556+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fce10062e80 con 0x7fcdf40a3c40 2024-04-26T01:39:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.560+0000 7fce0cc00700 1 -- 172.21.15.28:0/2859280868 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fce10062e80 con 0x7fcdf40a3c40 2024-04-26T01:39:59.557 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-04-26T01:39:59.557 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-04-26T01:39:59.557 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-04-26T01:39:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcdf40a3c40 msgr2=0x7fcdf40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcdf40a3c40 0x7fcdf40a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009750 tx=0x7fcdf8009310 comp rx=0 tx=0).stop 2024-04-26T01:39:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce100754a0 msgr2=0x7fce1019ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce100754a0 0x7fce1019ad50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fce0000e9a0 tx=0x7fce0000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 shutdown_connections 2024-04-26T01:39:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcdf40a3c40 0x7fcdf40a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce1010a9b0 0x7fce101a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce100754a0 0x7fce1019ad50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 --2- 172.21.15.28:0/2859280868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce10076a70 0x7fce1019b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.563+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 >> 172.21.15.28:0/2859280868 conn(0x7fce100fe080 msgr2=0x7fce101068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.564+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 shutdown_connections 2024-04-26T01:39:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:39:59.564+0000 7fce16800700 1 -- 172.21.15.28:0/2859280868 wait complete. 2024-04-26T01:40:00.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:39:59 smithi169 ceph-mon[32851]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:00.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:39:59 smithi053 ceph-mon[32593]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:00.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:39:59 smithi028 ceph-mon[28247]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:00.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T01:40:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:01.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:00 smithi169 ceph-mon[32851]: pgmap v428: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:01.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:00 smithi169 ceph-mon[32851]: overall HEALTH_OK 2024-04-26T01:40:01.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:00 smithi028 ceph-mon[28247]: pgmap v428: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:01.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:00 smithi028 ceph-mon[28247]: overall HEALTH_OK 2024-04-26T01:40:01.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:00 smithi053 ceph-mon[32593]: pgmap v428: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:01.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:00 smithi053 ceph-mon[32593]: overall HEALTH_OK 2024-04-26T01:40:03.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 -- 172.21.15.28:0/778470894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c108760 msgr2=0x7fcc7c10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 --2- 172.21.15.28:0/778470894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c108760 0x7fcc7c10ab50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400b0a0 tx=0x7fcc7400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:02 smithi028 ceph-mon[28247]: pgmap v429: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 -- 172.21.15.28:0/778470894 shutdown_connections 2024-04-26T01:40:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 --2- 172.21.15.28:0/778470894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c108760 0x7fcc7c10ab50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 --2- 172.21.15.28:0/778470894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc7c103500 0x7fcc7c1058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 --2- 172.21.15.28:0/778470894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c105e30 0x7fcc7c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 -- 172.21.15.28:0/778470894 >> 172.21.15.28:0/778470894 conn(0x7fcc7c0fce50 msgr2=0x7fcc7c0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 -- 172.21.15.28:0/778470894 shutdown_connections 2024-04-26T01:40:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.192+0000 7fcc7be00700 1 -- 172.21.15.28:0/778470894 wait complete. 2024-04-26T01:40:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.193+0000 7fcc7be00700 1 Processor -- start 2024-04-26T01:40:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7be00700 1 -- start start 2024-04-26T01:40:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc7c103500 0x7fcc7c102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c105e30 0x7fcc7c1014b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7be00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c108760 0x7fcc7c1019f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7be00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c10e450 con 0x7fcc7c105e30 2024-04-26T01:40:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7be00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c10e2d0 con 0x7fcc7c108760 2024-04-26T01:40:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7be00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c10e5d0 con 0x7fcc7c103500 2024-04-26T01:40:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c105e30 0x7fcc7c1014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7aa00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc7c103500 0x7fcc7c102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c105e30 0x7fcc7c1014b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34846/0 (socket says 172.21.15.28:34846) 2024-04-26T01:40:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7b400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c108760 0x7fcc7c1019f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.194+0000 7fcc7a000700 1 -- 172.21.15.28:0/1985821601 learned_addr learned my addr 172.21.15.28:0/1985821601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.195+0000 7fcc7b400700 1 -- 172.21.15.28:0/1985821601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc7c103500 msgr2=0x7fcc7c102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.195+0000 7fcc7b400700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc7c103500 0x7fcc7c102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.195+0000 7fcc7b400700 1 -- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c105e30 msgr2=0x7fcc7c1014b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.195+0000 7fcc7b400700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c105e30 0x7fcc7c1014b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.195+0000 7fcc7b400700 1 -- 172.21.15.28:0/1985821601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c1022a0 con 0x7fcc7c108760 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.195+0000 7fcc7a000700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c105e30 0x7fcc7c1014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.195+0000 7fcc7b400700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c108760 0x7fcc7c1019f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcc74006130 tx=0x7fcc74004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.195+0000 7fcc73400700 1 -- 172.21.15.28:0/1985821601 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74021070 con 0x7fcc7c108760 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.196+0000 7fcc73400700 1 -- 172.21.15.28:0/1985821601 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc74007b80 con 0x7fcc7c108760 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.196+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7c1a2b70 con 0x7fcc7c108760 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.196+0000 7fcc73400700 1 -- 172.21.15.28:0/1985821601 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74005040 con 0x7fcc7c108760 2024-04-26T01:40:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.196+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7c1a30b0 con 0x7fcc7c108760 2024-04-26T01:40:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.197+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc7c069730 con 0x7fcc7c108760 2024-04-26T01:40:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.199+0000 7fcc73400700 1 -- 172.21.15.28:0/1985821601 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcc7401d450 con 0x7fcc7c108760 2024-04-26T01:40:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.200+0000 7fcc73400700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcc600a3cc0 0x7fcc600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.200+0000 7fcc73400700 1 -- 172.21.15.28:0/1985821601 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fcc7400f020 con 0x7fcc7c108760 2024-04-26T01:40:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.201+0000 7fcc7aa00700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcc600a3cc0 0x7fcc600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.201+0000 7fcc7aa00700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcc600a3cc0 0x7fcc600a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009c60 tx=0x7fcc6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.205+0000 7fcc73400700 1 -- 172.21.15.28:0/1985821601 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc7402a030 con 0x7fcc7c108760 2024-04-26T01:40:03.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:02 smithi053 ceph-mon[32593]: pgmap v429: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:03.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:02 smithi169 ceph-mon[32851]: pgmap v429: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.449+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fcc7c062e80 con 0x7fcc600a3cc0 2024-04-26T01:40:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.485+0000 7fcc73400700 1 -- 172.21.15.28:0/1985821601 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fcc7c062e80 con 0x7fcc600a3cc0 2024-04-26T01:40:03.483 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T01:40:03.483 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (3m) 45s ago 3m 18.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 984a7114bd05 2024-04-26T01:40:03.483 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi053 running (3m) 65s ago 3m 21.5M - 19.0.0-3156-gd69f13d8 da4b004c09e8 b9e03549be77 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi169 running (3m) 66s ago 3m 18.6M - 19.0.0-3156-gd69f13d8 da4b004c09e8 813d4dceafef 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (3m) 45s ago 3m 13.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 83f859f87345 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi053 running (3m) 65s ago 3m 13.7M - 19.0.0-3156-gd69f13d8 da4b004c09e8 998214597ce2 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi169 running (3m) 66s ago 3m 20.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 a2f9a35ac466 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (3m) 45s ago 3m 18.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 5127c7787501 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi053 running (3m) 65s ago 3m 13.7M - 19.0.0-3156-gd69f13d8 da4b004c09e8 57bcc154dc47 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi169 running (3m) 66s ago 3m 27.6M - 19.0.0-3156-gd69f13d8 da4b004c09e8 36c61d61a670 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (4m) 45s ago 4m 13.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 2492a7c9273a 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi053 running (4m) 65s ago 4m 13.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 7498e416a85a 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi169 running (4m) 66s ago 4m 13.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 5dc46c50602c 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (12m) 45s ago 12m 514M - 19.0.0-3156-gd69f13d8 da4b004c09e8 2e5fee773f32 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi053 *:8443,8765 running (10m) 65s ago 10m 456M - 19.0.0-3156-gd69f13d8 da4b004c09e8 4c2228c33d22 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi169 *:8443,8765 running (10m) 66s ago 10m 454M - 19.0.0-3156-gd69f13d8 da4b004c09e8 caca5a9b12e1 2024-04-26T01:40:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (12m) 45s ago 12m 49.2M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 8553833424df 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi053 running (10m) 65s ago 10m 43.8M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 2cf2123eb492 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi169 running (10m) 66s ago 10m 43.0M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 edd4d4fc4da9 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (9m) 45s ago 9m 56.9M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 c6c032f6c583 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi053 running (9m) 65s ago 9m 48.1M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 61b26e19c9dd 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi169 running (8m) 66s ago 8m 58.1M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 f562c3a0d468 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (8m) 45s ago 8m 43.6M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 d5ba3546dd6d 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi053 running (7m) 65s ago 7m 44.0M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 01d309ad14b9 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi169 running (7m) 66s ago 7m 57.4M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 2f286c8b402b 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (6m) 45s ago 6m 53.4M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 2ecfbd5a52be 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi053 running (6m) 65s ago 6m 56.6M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 77b4b24cc1e6 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi169 running (6m) 66s ago 6m 46.8M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 44d8ccfc3774 2024-04-26T01:40:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (5m) 45s ago 5m 42.9M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 c9c117b985fc 2024-04-26T01:40:03.486 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi053 running (5m) 65s ago 4m 56.6M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 e6c45ab53b47 2024-04-26T01:40:03.486 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi169 running (4m) 66s ago 4m 55.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 93a3436c6667 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcc600a3cc0 msgr2=0x7fcc600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcc600a3cc0 0x7fcc600a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009c60 tx=0x7fcc6c009260 comp rx=0 tx=0).stop 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c108760 msgr2=0x7fcc7c1019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c108760 0x7fcc7c1019f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcc74006130 tx=0x7fcc74004ce0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 shutdown_connections 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fcc600a3cc0 0x7fcc600a6180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c108760 0x7fcc7c1019f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc7c103500 0x7fcc7c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 --2- 172.21.15.28:0/1985821601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc7c105e30 0x7fcc7c1014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 >> 172.21.15.28:0/1985821601 conn(0x7fcc7c0fce50 msgr2=0x7fcc7c103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 shutdown_connections 2024-04-26T01:40:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:03.489+0000 7fcc7be00700 1 -- 172.21.15.28:0/1985821601 wait complete. 2024-04-26T01:40:04.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:03 smithi053 ceph-mon[32593]: from='client.24911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:04.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:03 smithi028 ceph-mon[28247]: from='client.24911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:04.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T01:40:04.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:03 smithi169 ceph-mon[32851]: from='client.24911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:04.769 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:05.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:04 smithi053 ceph-mon[32593]: pgmap v430: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:05.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:04 smithi028 ceph-mon[28247]: pgmap v430: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:05.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:04 smithi169 ceph-mon[32851]: pgmap v430: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.965+0000 7f46fd800700 1 -- 172.21.15.28:0/2564264032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46f81026b0 msgr2=0x7f46f8102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.965+0000 7f46fd800700 1 --2- 172.21.15.28:0/2564264032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46f81026b0 0x7f46f8102b30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009f90 tx=0x7f46f000b330 comp rx=0 tx=0).stop 2024-04-26T01:40:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.966+0000 7f46fd800700 1 -- 172.21.15.28:0/2564264032 shutdown_connections 2024-04-26T01:40:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.966+0000 7f46fd800700 1 --2- 172.21.15.28:0/2564264032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46f81026b0 0x7f46f8102b30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.966+0000 7f46fd800700 1 --2- 172.21.15.28:0/2564264032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8101390 0x7f46f81017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.966+0000 7f46fd800700 1 --2- 172.21.15.28:0/2564264032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46f8101cf0 0x7f46f8102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.966+0000 7f46fd800700 1 -- 172.21.15.28:0/2564264032 >> 172.21.15.28:0/2564264032 conn(0x7f46f80fcf30 msgr2=0x7f46f80ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.966+0000 7f46fd800700 1 -- 172.21.15.28:0/2564264032 shutdown_connections 2024-04-26T01:40:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.966+0000 7f46fd800700 1 -- 172.21.15.28:0/2564264032 wait complete. 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.967+0000 7f46fd800700 1 Processor -- start 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.967+0000 7f46fd800700 1 -- start start 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.967+0000 7f46fd800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46f8101390 0x7f46f819ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.967+0000 7f46fd800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8101cf0 0x7f46f819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.967+0000 7f46fd800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46f81026b0 0x7f46f81a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.967+0000 7f46fd800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8110760 con 0x7f46f8101cf0 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.967+0000 7f46fd800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f46f81105e0 con 0x7f46f81026b0 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.967+0000 7f46fd800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f46f81108e0 con 0x7f46f8101390 2024-04-26T01:40:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f6000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8101cf0 0x7f46f819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:06.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f6000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8101cf0 0x7f46f819b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34868/0 (socket says 172.21.15.28:34868) 2024-04-26T01:40:06.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f6000700 1 -- 172.21.15.28:0/3395782288 learned_addr learned my addr 172.21.15.28:0/3395782288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:06.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f6a00700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46f8101390 0x7f46f819ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:06.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f7400700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46f81026b0 0x7f46f81a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f7400700 1 -- 172.21.15.28:0/3395782288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46f8101390 msgr2=0x7f46f819ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f7400700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46f8101390 0x7f46f819ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f7400700 1 -- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8101cf0 msgr2=0x7f46f819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f7400700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8101cf0 0x7f46f819b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f7400700 1 -- 172.21.15.28:0/3395782288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f81a2a10 con 0x7f46f81026b0 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.968+0000 7f46f6000700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8101cf0 0x7f46f819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.969+0000 7f46f7400700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46f81026b0 0x7f46f81a2370 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f46f000b6c0 tx=0x7f46f000cfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.969+0000 7f46ef400700 1 -- 172.21.15.28:0/3395782288 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0021070 con 0x7f46f81026b0 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.969+0000 7f46ef400700 1 -- 172.21.15.28:0/3395782288 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46f00092e0 con 0x7f46f81026b0 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.969+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f81a2ca0 con 0x7f46f81026b0 2024-04-26T01:40:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.969+0000 7f46ef400700 1 -- 172.21.15.28:0/3395782288 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f00052a0 con 0x7f46f81026b0 2024-04-26T01:40:06.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.969+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f81a3150 con 0x7f46f81026b0 2024-04-26T01:40:06.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.971+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f8069730 con 0x7f46f81026b0 2024-04-26T01:40:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.974+0000 7f46ef400700 1 -- 172.21.15.28:0/3395782288 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f46f0024070 con 0x7f46f81026b0 2024-04-26T01:40:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.975+0000 7f46ef400700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f46dc0a3c40 0x7f46dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.975+0000 7f46ef400700 1 -- 172.21.15.28:0/3395782288 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f46f000f020 con 0x7f46f81026b0 2024-04-26T01:40:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.976+0000 7f46f6a00700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f46dc0a3c40 0x7f46dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.976+0000 7f46f6a00700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f46dc0a3c40 0x7f46dc0a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009750 tx=0x7f46e8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:06.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:06.980+0000 7f46ef400700 1 -- 172.21.15.28:0/3395782288 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46f002b020 con 0x7f46f81026b0 2024-04-26T01:40:07.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:06 smithi053 ceph-mon[32593]: pgmap v431: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.222+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f46f8062e80 con 0x7f46dc0a3c40 2024-04-26T01:40:07.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:06 smithi028 ceph-mon[28247]: pgmap v431: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:07.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.239+0000 7f46ef400700 1 -- 172.21.15.28:0/3395782288 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f46f8062e80 con 0x7f46dc0a3c40 2024-04-26T01:40:07.236 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T01:40:07.236 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 70s ago 4m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l;count:12 2024-04-26T01:40:07.236 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 70s ago 10m smithi028=x;smithi053=y;smithi169=z;count:3 2024-04-26T01:40:07.236 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 70s ago 10m smithi028:172.21.15.28=a;smithi053:172.21.15.53=b;smithi169:172.21.15.169=c;count:3 2024-04-26T01:40:07.236 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 70s ago - 2024-04-26T01:40:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f46dc0a3c40 msgr2=0x7f46dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f46dc0a3c40 0x7f46dc0a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009750 tx=0x7f46e8009310 comp rx=0 tx=0).stop 2024-04-26T01:40:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46f81026b0 msgr2=0x7f46f81a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46f81026b0 0x7f46f81a2370 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f46f000b6c0 tx=0x7f46f000cfa0 comp rx=0 tx=0).stop 2024-04-26T01:40:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 shutdown_connections 2024-04-26T01:40:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f46dc0a3c40 0x7f46dc0a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:07.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46f81026b0 0x7f46f81a2370 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:07.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46f8101390 0x7f46f819ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:07.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 --2- 172.21.15.28:0/3395782288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f8101cf0 0x7f46f819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:07.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 >> 172.21.15.28:0/3395782288 conn(0x7f46f80fcf30 msgr2=0x7f46f80fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:07.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 shutdown_connections 2024-04-26T01:40:07.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:07.242+0000 7f46fd800700 1 -- 172.21.15.28:0/3395782288 wait complete. 2024-04-26T01:40:07.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:06 smithi169 ceph-mon[32851]: pgmap v431: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:08.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T01:40:08.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:07 smithi053 ceph-mon[32593]: from='client.24917 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:08.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:07 smithi028 ceph-mon[28247]: from='client.24917 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:08.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:07 smithi169 ceph-mon[32851]: from='client.24917 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:08.497 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:09.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:08 smithi053 ceph-mon[32593]: pgmap v432: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:09.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:08 smithi028 ceph-mon[28247]: pgmap v432: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:09.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:08 smithi169 ceph-mon[32851]: pgmap v432: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 -- 172.21.15.28:0/3117326515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410ac80 msgr2=0x7f7f94111530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 --2- 172.21.15.28:0/3117326515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410ac80 0x7f7f94111530 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0099c0 tx=0x7f7f8c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 -- 172.21.15.28:0/3117326515 shutdown_connections 2024-04-26T01:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 --2- 172.21.15.28:0/3117326515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410ac80 0x7f7f94111530 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 --2- 172.21.15.28:0/3117326515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f941090e0 0x7f7f94109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 --2- 172.21.15.28:0/3117326515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f9410a2e0 0x7f7f9410a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 -- 172.21.15.28:0/3117326515 >> 172.21.15.28:0/3117326515 conn(0x7f7f940780b0 msgr2=0x7f7f9407a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 -- 172.21.15.28:0/3117326515 shutdown_connections 2024-04-26T01:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.806+0000 7f7f99400700 1 -- 172.21.15.28:0/3117326515 wait complete. 2024-04-26T01:40:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.807+0000 7f7f99400700 1 Processor -- start 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.807+0000 7f7f99400700 1 -- start start 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f99400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f941090e0 0x7f7f9419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f99400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410a2e0 0x7f7f9419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f99400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f9410ac80 0x7f7f941a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f99400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f94114be0 con 0x7f7f9410a2e0 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f99400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7f94114a60 con 0x7f7f9410ac80 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f99400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7f94114d60 con 0x7f7f941090e0 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f89e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410a2e0 0x7f7f9419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f92800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f941090e0 0x7f7f9419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f92800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f941090e0 0x7f7f9419f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59962/0 (socket says 172.21.15.28:59962) 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f89e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410a2e0 0x7f7f9419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49186/0 (socket says 172.21.15.28:49186) 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f92800700 1 -- 172.21.15.28:0/3193048590 learned_addr learned my addr 172.21.15.28:0/3193048590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.808+0000 7f7f93200700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f9410ac80 0x7f7f941a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f92800700 1 -- 172.21.15.28:0/3193048590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f9410ac80 msgr2=0x7f7f941a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f92800700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f9410ac80 0x7f7f941a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f92800700 1 -- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410a2e0 msgr2=0x7f7f9419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f92800700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410a2e0 0x7f7f9419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f92800700 1 -- 172.21.15.28:0/3193048590 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f941a6dd0 con 0x7f7f941090e0 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f92800700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f941090e0 0x7f7f9419f170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009f40 tx=0x7f7f8400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f8be00700 1 -- 172.21.15.28:0/3193048590 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f840092e0 con 0x7f7f941090e0 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f941a7060 con 0x7f7f941090e0 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f8be00700 1 -- 172.21.15.28:0/3193048590 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f8400c9c0 con 0x7f7f941090e0 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f8be00700 1 -- 172.21.15.28:0/3193048590 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f840044c0 con 0x7f7f941090e0 2024-04-26T01:40:10.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.809+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f941a75a0 con 0x7f7f941090e0 2024-04-26T01:40:10.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.810+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f94069730 con 0x7f7f941090e0 2024-04-26T01:40:10.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.813+0000 7f7f8be00700 1 -- 172.21.15.28:0/3193048590 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7f8401b030 con 0x7f7f941090e0 2024-04-26T01:40:10.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.814+0000 7f7f8be00700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7f780a3c40 0x7f7f780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:10.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.814+0000 7f7f8be00700 1 -- 172.21.15.28:0/3193048590 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f7f840d0cc0 con 0x7f7f941090e0 2024-04-26T01:40:10.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.814+0000 7f7f89e00700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7f780a3c40 0x7f7f780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.815+0000 7f7f89e00700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7f780a3c40 0x7f7f780a6100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009c60 tx=0x7f7f7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:10.816+0000 7f7f8be00700 1 -- 172.21.15.28:0/3193048590 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7f84022020 con 0x7f7f941090e0 2024-04-26T01:40:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.053+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f7f94062e80 con 0x7f7f780a3c40 2024-04-26T01:40:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.057+0000 7f7f8be00700 1 -- 172.21.15.28:0/3193048590 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f7f94062e80 con 0x7f7f780a3c40 2024-04-26T01:40:11.054 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-04-26T01:40:11.054 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-04-26T01:40:11.054 INFO:teuthology.orchestra.run.smithi028.stdout:smithi053 172.21.15.53 2024-04-26T01:40:11.054 INFO:teuthology.orchestra.run.smithi028.stdout:smithi169 172.21.15.169 2024-04-26T01:40:11.054 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-04-26T01:40:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7f780a3c40 msgr2=0x7f7f780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7f780a3c40 0x7f7f780a6100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009c60 tx=0x7f7f7c009260 comp rx=0 tx=0).stop 2024-04-26T01:40:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f941090e0 msgr2=0x7f7f9419f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f941090e0 0x7f7f9419f170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009f40 tx=0x7f7f8400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 shutdown_connections 2024-04-26T01:40:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f7f780a3c40 0x7f7f780a6100 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f9410ac80 0x7f7f941a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f941090e0 0x7f7f9419f170 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 --2- 172.21.15.28:0/3193048590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f9410a2e0 0x7f7f9419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 >> 172.21.15.28:0/3193048590 conn(0x7f7f940780b0 msgr2=0x7f7f9410d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 shutdown_connections 2024-04-26T01:40:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:11.060+0000 7f7f99400700 1 -- 172.21.15.28:0/3193048590 wait complete. 2024-04-26T01:40:11.212 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:10 smithi169 ceph-mon[32851]: pgmap v433: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:11.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:10 smithi053 ceph-mon[32593]: pgmap v433: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:11.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:10 smithi028 ceph-mon[28247]: pgmap v433: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:11.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T01:40:12.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:11 smithi053 ceph-mon[32593]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:12.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:11 smithi028 ceph-mon[28247]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:12.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:11 smithi169 ceph-mon[32851]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:13.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:12 smithi053 ceph-mon[32593]: pgmap v434: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:13.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:12 smithi028 ceph-mon[28247]: pgmap v434: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:13.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:12 smithi169 ceph-mon[32851]: pgmap v434: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:14.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.711+0000 7f086b600700 1 -- 172.21.15.28:0/1301846024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864106850 msgr2=0x7f086410d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:14.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.711+0000 7f086b600700 1 --2- 172.21.15.28:0/1301846024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864106850 0x7f086410d100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f085c0099c0 tx=0x7f085c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.711+0000 7f086b600700 1 -- 172.21.15.28:0/1301846024 shutdown_connections 2024-04-26T01:40:14.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.711+0000 7f086b600700 1 --2- 172.21.15.28:0/1301846024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864106850 0x7f086410d100 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.711+0000 7f086b600700 1 --2- 172.21.15.28:0/1301846024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864104cb0 0x7f08641050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.711+0000 7f086b600700 1 --2- 172.21.15.28:0/1301846024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0864105eb0 0x7f0864106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.711+0000 7f086b600700 1 -- 172.21.15.28:0/1301846024 >> 172.21.15.28:0/1301846024 conn(0x7f0864100250 msgr2=0x7f0864102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:14.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.712+0000 7f086b600700 1 -- 172.21.15.28:0/1301846024 shutdown_connections 2024-04-26T01:40:14.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.712+0000 7f086b600700 1 -- 172.21.15.28:0/1301846024 wait complete. 2024-04-26T01:40:14.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f086b600700 1 Processor -- start 2024-04-26T01:40:14.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f086b600700 1 -- start start 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f086b600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0864104cb0 0x7f086419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f086b600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864105eb0 0x7f086419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f086b600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864106850 0x7f08641a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f086b600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0864110780 con 0x7f0864106850 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f086b600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0864110600 con 0x7f0864105eb0 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f086b600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0864110900 con 0x7f0864104cb0 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f0863e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864105eb0 0x7f086419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f0868c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0864104cb0 0x7f086419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f0863e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864105eb0 0x7f086419b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53404/0 (socket says 172.21.15.28:53404) 2024-04-26T01:40:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f0868c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0864104cb0 0x7f086419adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59980/0 (socket says 172.21.15.28:59980) 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.713+0000 7f0863e00700 1 -- 172.21.15.28:0/4083724307 learned_addr learned my addr 172.21.15.28:0/4083724307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.714+0000 7f0869600700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864106850 0x7f08641a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.714+0000 7f0868c00700 1 -- 172.21.15.28:0/4083724307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864105eb0 msgr2=0x7f086419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.714+0000 7f0868c00700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864105eb0 0x7f086419b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.714+0000 7f0868c00700 1 -- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864106850 msgr2=0x7f08641a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.714+0000 7f0868c00700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864106850 0x7f08641a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.714+0000 7f0868c00700 1 -- 172.21.15.28:0/4083724307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08641a2a20 con 0x7f0864104cb0 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.714+0000 7f0863e00700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864105eb0 0x7f086419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.714+0000 7f0868c00700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0864104cb0 0x7f086419adc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f085800ef10 tx=0x7f085800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.715+0000 7f0861600700 1 -- 172.21.15.28:0/4083724307 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580092e0 con 0x7f0864104cb0 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.715+0000 7f0861600700 1 -- 172.21.15.28:0/4083724307 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f085800c9c0 con 0x7f0864104cb0 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.715+0000 7f0861600700 1 -- 172.21.15.28:0/4083724307 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858004440 con 0x7f0864104cb0 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.715+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08641a2cb0 con 0x7f0864104cb0 2024-04-26T01:40:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.715+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08641a31a0 con 0x7f0864104cb0 2024-04-26T01:40:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.716+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0864069730 con 0x7f0864104cb0 2024-04-26T01:40:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.718+0000 7f0861600700 1 -- 172.21.15.28:0/4083724307 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f085801b030 con 0x7f0864104cb0 2024-04-26T01:40:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.720+0000 7f0861600700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f084c0a3c40 0x7f084c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.720+0000 7f0861600700 1 -- 172.21.15.28:0/4083724307 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f08580d0d30 con 0x7f0864104cb0 2024-04-26T01:40:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.720+0000 7f0863e00700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f084c0a3c40 0x7f084c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.721+0000 7f0863e00700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f084c0a3c40 0x7f084c0a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0850011550 tx=0x7f0850009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.722+0000 7f0861600700 1 -- 172.21.15.28:0/4083724307 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0858022020 con 0x7f0864104cb0 2024-04-26T01:40:14.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.962+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 --> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f0864062e80 con 0x7f084c0a3c40 2024-04-26T01:40:14.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.965+0000 7f0861600700 1 -- 172.21.15.28:0/4083724307 <== mgr.14152 v2:172.21.15.28:6800/3431203710 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f0864062e80 con 0x7f084c0a3c40 2024-04-26T01:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T01:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 5m ago Has a FileSystem, LVM detected 2024-04-26T01:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout:smithi053 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300G8480DGN 447G No 5m ago Has a FileSystem, LVM detected 2024-04-26T01:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout:smithi169 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300LN400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T01:40:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f084c0a3c40 msgr2=0x7f084c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f084c0a3c40 0x7f084c0a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0850011550 tx=0x7f0850009260 comp rx=0 tx=0).stop 2024-04-26T01:40:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0864104cb0 msgr2=0x7f086419adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0864104cb0 0x7f086419adc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f085800ef10 tx=0x7f085800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 shutdown_connections 2024-04-26T01:40:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f084c0a3c40 0x7f084c0a6100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864106850 0x7f08641a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0864104cb0 0x7f086419adc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 --2- 172.21.15.28:0/4083724307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0864105eb0 0x7f086419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:14.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.968+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 >> 172.21.15.28:0/4083724307 conn(0x7f0864100250 msgr2=0x7f08641091b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:14.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.969+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 shutdown_connections 2024-04-26T01:40:14.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:14.969+0000 7f086b600700 1 -- 172.21.15.28:0/4083724307 wait complete. 2024-04-26T01:40:15.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:14 smithi053 ceph-mon[32593]: pgmap v435: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:15.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:14 smithi028 ceph-mon[28247]: pgmap v435: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:15.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:14 smithi169 ceph-mon[32851]: pgmap v435: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:15.850 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T01:40:15.861 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:40:15.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T01:40:16.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:15 smithi028 ceph-mon[28247]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:16.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:15 smithi053 ceph-mon[32593]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:16.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:15 smithi169 ceph-mon[32851]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:16.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:16 smithi028 ceph-mon[28247]: pgmap v436: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:16 smithi169 ceph-mon[32851]: pgmap v436: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:17.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:16 smithi053 ceph-mon[32593]: pgmap v436: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:18.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.578+0000 7f388a600700 1 -- 172.21.15.28:0/956741940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 msgr2=0x7f3884108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.578+0000 7f388a600700 1 --2- 172.21.15.28:0/956741940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 0x7f3884108250 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f38700098f0 tx=0x7f3870009c00 comp rx=0 tx=0).stop 2024-04-26T01:40:18.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.579+0000 7f388a600700 1 -- 172.21.15.28:0/956741940 shutdown_connections 2024-04-26T01:40:18.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.579+0000 7f388a600700 1 --2- 172.21.15.28:0/956741940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3884108790 0x7f388410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.579+0000 7f388a600700 1 --2- 172.21.15.28:0/956741940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3884103530 0x7f3884105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.579+0000 7f388a600700 1 --2- 172.21.15.28:0/956741940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 0x7f3884108250 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.579+0000 7f388a600700 1 -- 172.21.15.28:0/956741940 >> 172.21.15.28:0/956741940 conn(0x7f38840fcf40 msgr2=0x7f38840ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.579+0000 7f388a600700 1 -- 172.21.15.28:0/956741940 shutdown_connections 2024-04-26T01:40:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.579+0000 7f388a600700 1 -- 172.21.15.28:0/956741940 wait complete. 2024-04-26T01:40:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.580+0000 7f388a600700 1 Processor -- start 2024-04-26T01:40:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.580+0000 7f388a600700 1 -- start start 2024-04-26T01:40:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.580+0000 7f388a600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3884103530 0x7f3884102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.580+0000 7f388a600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 0x7f38841014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.580+0000 7f388a600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3884108790 0x7f3884101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.580+0000 7f388a600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f388410e5a0 con 0x7f3884108790 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.580+0000 7f388a600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f388410e420 con 0x7f3884105e60 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f388a600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f388410e720 con 0x7f3884103530 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3884108790 0x7f3884101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3884103530 0x7f3884102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3884108790 0x7f3884101a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49224/0 (socket says 172.21.15.28:49224) 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883e00700 1 -- 172.21.15.28:0/17892757 learned_addr learned my addr 172.21.15.28:0/17892757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3882a00700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 0x7f38841014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883e00700 1 -- 172.21.15.28:0/17892757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3884103530 msgr2=0x7f3884102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883e00700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3884103530 0x7f3884102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883e00700 1 -- 172.21.15.28:0/17892757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 msgr2=0x7f38841014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883e00700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 0x7f38841014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3883e00700 1 -- 172.21.15.28:0/17892757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38841022d0 con 0x7f3884108790 2024-04-26T01:40:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.581+0000 7f3882a00700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 0x7f38841014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.582+0000 7f3883e00700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3884108790 0x7f3884101a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f387c00ef90 tx=0x7f387c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.582+0000 7f387be00700 1 -- 172.21.15.28:0/17892757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c0092e0 con 0x7f3884108790 2024-04-26T01:40:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.582+0000 7f387be00700 1 -- 172.21.15.28:0/17892757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f387c004bb0 con 0x7f3884108790 2024-04-26T01:40:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.582+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38841a2e20 con 0x7f3884108790 2024-04-26T01:40:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.582+0000 7f387be00700 1 -- 172.21.15.28:0/17892757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c004280 con 0x7f3884108790 2024-04-26T01:40:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.582+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38841a32d0 con 0x7f3884108790 2024-04-26T01:40:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.584+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3884069730 con 0x7f3884108790 2024-04-26T01:40:18.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.586+0000 7f387be00700 1 -- 172.21.15.28:0/17892757 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f387c01b030 con 0x7f3884108790 2024-04-26T01:40:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.587+0000 7f387be00700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f38680a3cc0 0x7f38680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.587+0000 7f387be00700 1 -- 172.21.15.28:0/17892757 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f387c0cfea0 con 0x7f3884108790 2024-04-26T01:40:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.587+0000 7f3883400700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f38680a3cc0 0x7f38680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.588+0000 7f3883400700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f38680a3cc0 0x7f38680a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3874005fd0 tx=0x7f3874009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.592+0000 7f387be00700 1 -- 172.21.15.28:0/17892757 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f387c022020 con 0x7f3884108790 2024-04-26T01:40:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.878+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f3884063650 con 0x7f3884108790 2024-04-26T01:40:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.879+0000 7f387be00700 1 -- 172.21.15.28:0/17892757 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3045 (secure 0 0 0) 0x7f387c010040 con 0x7f3884108790 2024-04-26T01:40:18.876 INFO:teuthology.orchestra.run.smithi028.stdout:e24 2024-04-26T01:40:18.876 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:40:18.876 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:40:18.876 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-04-26T01:40:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:40:18.876 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-04-26T01:40:18.876 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 24 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-26T01:36:57.152667+0000 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-26T01:37:21.237658+0000 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-04-26T01:40:18.877 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 3 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24523,1=24509,2=24505} 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3] 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-04-26T01:40:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24523 members: 24505,24523,24509 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{0:24523} state up:active seq 10 addr [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{0:24499} state up:standby-replay seq 1 addr [v2:172.21.15.169:6834/1802608598,v1:172.21.15.169:6835/1802608598] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{1:24509} state up:active seq 11 addr [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{1:24497} state up:standby-replay seq 1 addr [v2:172.21.15.28:6840/2299937376,v1:172.21.15.28:6841/2299937376] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{2:24505} state up:active seq 16 addr [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.879 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{2:24491} state up:standby-replay seq 1 addr [v2:172.21.15.28:6838/2114874192,v1:172.21.15.28:6839/2114874192] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14358} state up:standby seq 1 addr [v2:172.21.15.28:6834/2607969943,v1:172.21.15.28:6835/2607969943] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:14370} state up:standby seq 1 addr [v2:172.21.15.28:6836/447949088,v1:172.21.15.28:6837/447949088] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{-1:24422} state up:standby seq 1 addr [v2:172.21.15.53:6832/332324514,v1:172.21.15.53:6833/332324514] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{-1:24452} state up:standby seq 1 addr [v2:172.21.15.53:6834/2175861856,v1:172.21.15.53:6835/2175861856] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{-1:24457} state up:standby seq 1 addr [v2:172.21.15.169:6832/451902326,v1:172.21.15.169:6833/451902326] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.880 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{-1:24470} state up:standby seq 1 addr [v2:172.21.15.53:6836/2195700399,v1:172.21.15.53:6837/2195700399] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:40:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.881+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f38680a3cc0 msgr2=0x7f38680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.881+0000 7f388a600700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f38680a3cc0 0x7f38680a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3874005fd0 tx=0x7f3874009310 comp rx=0 tx=0).stop 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.881+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3884108790 msgr2=0x7f3884101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.881+0000 7f388a600700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3884108790 0x7f3884101a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f387c00ef90 tx=0x7f387c00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.882+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 shutdown_connections 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.882+0000 7f388a600700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f38680a3cc0 0x7f38680a6180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.882+0000 7f388a600700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3884108790 0x7f3884101a20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.882+0000 7f388a600700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3884103530 0x7f3884102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.882+0000 7f388a600700 1 --2- 172.21.15.28:0/17892757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3884105e60 0x7f38841014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.882+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 >> 172.21.15.28:0/17892757 conn(0x7f38840fcf40 msgr2=0x7f3884103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.882+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 shutdown_connections 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:18.882+0000 7f388a600700 1 -- 172.21.15.28:0/17892757 wait complete. 2024-04-26T01:40:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-26T01:40:19.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:18 smithi028 ceph-mon[28247]: pgmap v437: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:19.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:18 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/17892757' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:40:19.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:18 smithi169 ceph-mon[32851]: pgmap v437: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:19.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:18 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/17892757' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:40:19.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:18 smithi053 ceph-mon[32593]: pgmap v437: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:19.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:18 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/17892757' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:40:19.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T01:40:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:20.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:19 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:20.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:19 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:20.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:19 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:21.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:21 smithi028 ceph-mon[28247]: pgmap v438: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:21.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:21 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:21.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:21 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:21.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:21 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:21.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:21 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:21.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:21 smithi028 ceph-mon[28247]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:21.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:21 smithi169 ceph-mon[32851]: pgmap v438: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:21.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:21 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:21.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:21 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:21.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:21 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:21.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:21 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:21.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:21 smithi169 ceph-mon[32851]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:21.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:21 smithi053 ceph-mon[32593]: pgmap v438: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:21.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:21 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:21.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:21 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:21 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:21 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:21 smithi053 ceph-mon[32593]: from='mgr.14152 172.21.15.28:0/3697388231' entity='mgr.x' 2024-04-26T01:40:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 -- 172.21.15.28:0/3315904028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 msgr2=0x7fada0104020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 --2- 172.21.15.28:0/3315904028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 0x7fada0104020 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fad8c0099c0 tx=0x7fad8c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:40:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 -- 172.21.15.28:0/3315904028 shutdown_connections 2024-04-26T01:40:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 --2- 172.21.15.28:0/3315904028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fada0104560 0x7fada010ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 --2- 172.21.15.28:0/3315904028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada01029c0 0x7fada0102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 --2- 172.21.15.28:0/3315904028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 0x7fada0104020 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 -- 172.21.15.28:0/3315904028 >> 172.21.15.28:0/3315904028 conn(0x7fada00fdf80 msgr2=0x7fada01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 -- 172.21.15.28:0/3315904028 shutdown_connections 2024-04-26T01:40:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.754+0000 7fada6800700 1 -- 172.21.15.28:0/3315904028 wait complete. 2024-04-26T01:40:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.755+0000 7fada6800700 1 Processor -- start 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.755+0000 7fada6800700 1 -- start start 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.755+0000 7fada6800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada01029c0 0x7fada0198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.755+0000 7fada6800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 0x7fada0199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.755+0000 7fada6800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fada0104560 0x7fada01a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.755+0000 7fada6800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fada010e4b0 con 0x7fada0103bc0 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.755+0000 7fada6800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fada010e330 con 0x7fada01029c0 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.755+0000 7fada6800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fada010e630 con 0x7fada0104560 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada5400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada01029c0 0x7fada0198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada5400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada01029c0 0x7fada0198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47678/0 (socket says 172.21.15.28:47678) 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada5400700 1 -- 172.21.15.28:0/906940747 learned_addr learned my addr 172.21.15.28:0/906940747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada4a00700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 0x7fada0199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada5e00700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fada0104560 0x7fada01a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada4a00700 1 -- 172.21.15.28:0/906940747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fada0104560 msgr2=0x7fada01a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada4a00700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fada0104560 0x7fada01a00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada4a00700 1 -- 172.21.15.28:0/906940747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada01029c0 msgr2=0x7fada0198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada4a00700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada01029c0 0x7fada0198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada4a00700 1 -- 172.21.15.28:0/906940747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada01a0760 con 0x7fada0103bc0 2024-04-26T01:40:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.756+0000 7fada5400700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada01029c0 0x7fada0198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.757+0000 7fada4a00700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 0x7fada0199040 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fad8c0099c0 tx=0x7fad8c003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.757+0000 7fad9e000700 1 -- 172.21.15.28:0/906940747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c009e60 con 0x7fada0103bc0 2024-04-26T01:40:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.757+0000 7fad9e000700 1 -- 172.21.15.28:0/906940747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad8c00f460 con 0x7fada0103bc0 2024-04-26T01:40:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.757+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada01a09f0 con 0x7fada0103bc0 2024-04-26T01:40:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.757+0000 7fad9e000700 1 -- 172.21.15.28:0/906940747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00bea0 con 0x7fada0103bc0 2024-04-26T01:40:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.757+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada01a0ea0 con 0x7fada0103bc0 2024-04-26T01:40:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.758+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada0069730 con 0x7fada0103bc0 2024-04-26T01:40:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.762+0000 7fad9e000700 1 -- 172.21.15.28:0/906940747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fad8c00f610 con 0x7fada0103bc0 2024-04-26T01:40:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.763+0000 7fad9e000700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fad880a3cc0 0x7fad880a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.764+0000 7fad9e000700 1 -- 172.21.15.28:0/906940747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fad8c0d11e0 con 0x7fada0103bc0 2024-04-26T01:40:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.764+0000 7fada5400700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fad880a3cc0 0x7fad880a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.765+0000 7fada5400700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fad880a3cc0 0x7fad880a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fad940105c0 tx=0x7fad9400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:22.766+0000 7fad9e000700 1 -- 172.21.15.28:0/906940747 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad8c013070 con 0x7fada0103bc0 2024-04-26T01:40:22.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.000+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fada002df80 con 0x7fada0103bc0 2024-04-26T01:40:22.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.002+0000 7fad9e000700 1 -- 172.21.15.28:0/906940747 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5277 (secure 0 0 0) 0x7fad8c09dc00 con 0x7fada0103bc0 2024-04-26T01:40:23.000 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 90 2024-04-26T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:fsid ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-26T01:27:15.883926+0000 2024-04-26T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-26T01:37:25.472266+0000 2024-04-26T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 26 2024-04-26T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-04-26T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/3881623886,v1:172.21.15.28:6803/3881623886] [v2:172.21.15.28:6804/3881623886,v1:172.21.15.28:6805/3881623886] exists,up f0017e56-53b2-4568-ab76-85ef9640ce85 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6800/1246761036,v1:172.21.15.53:6801/1246761036] [v2:172.21.15.53:6802/1246761036,v1:172.21.15.53:6803/1246761036] exists,up 4bfc13cc-6c4d-45ec-8b43-6a346569692f 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6800/3360829572,v1:172.21.15.169:6801/3360829572] [v2:172.21.15.169:6802/3360829572,v1:172.21.15.169:6803/3360829572] exists,up 147b31a0-5128-4e08-a252-ada0832abff2 2024-04-26T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/1623836559,v1:172.21.15.28:6811/1623836559] [v2:172.21.15.28:6812/1623836559,v1:172.21.15.28:6813/1623836559] exists,up 0929291f-f56e-4c6b-813d-886624278d0e 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 33 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6808/3129809581,v1:172.21.15.53:6809/3129809581] [v2:172.21.15.53:6810/3129809581,v1:172.21.15.53:6811/3129809581] exists,up 3adbfa0d-bced-4cd1-bf74-06e2fd42a015 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 38 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6808/1028981135,v1:172.21.15.169:6809/1028981135] [v2:172.21.15.169:6810/1028981135,v1:172.21.15.169:6811/1028981135] exists,up f9211a41-97a0-4477-8dfb-7354f6020bfd 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 46 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/4283307315,v1:172.21.15.28:6819/4283307315] [v2:172.21.15.28:6820/4283307315,v1:172.21.15.28:6821/4283307315] exists,up 05c575f6-3cd7-46be-9c3e-66fcc47b4bf2 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 51 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6816/1268967758,v1:172.21.15.53:6817/1268967758] [v2:172.21.15.53:6818/1268967758,v1:172.21.15.53:6819/1268967758] exists,up 630a21f2-701f-4b41-8d73-88c86c371714 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 59 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6816/3120946551,v1:172.21.15.169:6817/3120946551] [v2:172.21.15.169:6818/3120946551,v1:172.21.15.169:6819/3120946551] exists,up 1e807449-7760-4d62-801e-bf02535de567 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 67 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/2760989310,v1:172.21.15.28:6827/2760989310] [v2:172.21.15.28:6828/2760989310,v1:172.21.15.28:6829/2760989310] exists,up 41a7832e-3641-4f51-b26d-c928e268c72f 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 72 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6824/786160308,v1:172.21.15.53:6825/786160308] [v2:172.21.15.53:6826/786160308,v1:172.21.15.53:6827/786160308] exists,up ef2c5cbd-b772-4910-b007-c50899067c2a 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 80 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6824/2005675057,v1:172.21.15.169:6825/2005675057] [v2:172.21.15.169:6826/2005675057,v1:172.21.15.169:6827/2005675057] exists,up 673532a1-0742-4a52-82bf-8dd61d64a955 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.f [8,11] 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.10 [2,4] 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.16 [7,11] 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1a [8,4] 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1b [2,1] 2024-04-26T01:40:23.003 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.2c [4,9] 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.34 [4,9] 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.38 [3,9] 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3962708174 expires 2024-04-27T01:28:23.601445+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/259754378 expires 2024-04-27T01:28:23.601445+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/2991743583 expires 2024-04-27T01:28:23.601445+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3106064028 expires 2024-04-27T01:28:23.601445+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1803582359 expires 2024-04-27T01:27:49.755979+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/615407590 expires 2024-04-27T01:27:49.755979+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/1017286588 expires 2024-04-27T01:27:49.755979+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2119832118 expires 2024-04-27T01:27:49.755979+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2991743583 expires 2024-04-27T01:28:23.601445+0000 2024-04-26T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/1017286588 expires 2024-04-27T01:27:49.755979+0000 2024-04-26T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fad880a3cc0 msgr2=0x7fad880a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fad880a3cc0 0x7fad880a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fad940105c0 tx=0x7fad9400b4a0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 msgr2=0x7fada0199040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 0x7fada0199040 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fad8c0099c0 tx=0x7fad8c003e60 comp rx=0 tx=0).stop 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 shutdown_connections 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fad880a3cc0 0x7fad880a6180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fada0104560 0x7fada01a00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada01029c0 0x7fada0198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 --2- 172.21.15.28:0/906940747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada0103bc0 0x7fada0199040 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 >> 172.21.15.28:0/906940747 conn(0x7fada00fdf80 msgr2=0x7fada0106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.006+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 shutdown_connections 2024-04-26T01:40:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:23.007+0000 7fada6800700 1 -- 172.21.15.28:0/906940747 wait complete. 2024-04-26T01:40:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:23 smithi169 ceph-mon[32851]: pgmap v439: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:23 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/906940747' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:40:23.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:23 smithi053 ceph-mon[32593]: pgmap v439: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:23.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:23 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/906940747' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:40:23.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:23 smithi028 ceph-mon[28247]: pgmap v439: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:23.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:23 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/906940747' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:40:23.835 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T01:40:23.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:24.334 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:25.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:25 smithi169 ceph-mon[32851]: pgmap v440: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:25.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:25 smithi053 ceph-mon[32593]: pgmap v440: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:25.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:25 smithi028 ceph-mon[28247]: pgmap v440: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 -- 172.21.15.28:0/4242977635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 msgr2=0x7fe0e410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/4242977635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 0x7fe0e410f8b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009990 tx=0x7fe0dc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 -- 172.21.15.28:0/4242977635 shutdown_connections 2024-04-26T01:40:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/4242977635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 0x7fe0e410f8b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/4242977635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0e4108a50 0x7fe0e4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/4242977635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4102a90 0x7fe0e4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 -- 172.21.15.28:0/4242977635 >> 172.21.15.28:0/4242977635 conn(0x7fe0e4079ba0 msgr2=0x7fe0e4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 -- 172.21.15.28:0/4242977635 shutdown_connections 2024-04-26T01:40:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.679+0000 7fe0eaa00700 1 -- 172.21.15.28:0/4242977635 wait complete. 2024-04-26T01:40:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.680+0000 7fe0eaa00700 1 Processor -- start 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.680+0000 7fe0eaa00700 1 -- start start 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0eaa00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0e4102a90 0x7fe0e4074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0eaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 0x7fe0e4075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0eaa00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4108a50 0x7fe0e40757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0eaa00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4112c00 con 0x7fe0e4103450 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0eaa00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4112a80 con 0x7fe0e4108a50 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0eaa00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4112d80 con 0x7fe0e4102a90 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0db400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 0x7fe0e4075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0e3e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0e4102a90 0x7fe0e4074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0db400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 0x7fe0e4075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35582/0 (socket says 172.21.15.28:35582) 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0db400700 1 -- 172.21.15.28:0/3588998880 learned_addr learned my addr 172.21.15.28:0/3588998880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.681+0000 7fe0e8a00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4108a50 0x7fe0e40757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0db400700 1 -- 172.21.15.28:0/3588998880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0e4102a90 msgr2=0x7fe0e4074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0db400700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0e4102a90 0x7fe0e4074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0db400700 1 -- 172.21.15.28:0/3588998880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4108a50 msgr2=0x7fe0e40757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0db400700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4108a50 0x7fe0e40757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0db400700 1 -- 172.21.15.28:0/3588998880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e41abaa0 con 0x7fe0e4103450 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0e3e00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0e4102a90 0x7fe0e4074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0e8a00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4108a50 0x7fe0e40757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0db400700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 0x7fe0e4075280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00e9a0 tx=0x7fe0cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0e1600700 1 -- 172.21.15.28:0/3588998880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00ca10 con 0x7fe0e4103450 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0e1600700 1 -- 172.21.15.28:0/3588998880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0cc00cbb0 con 0x7fe0e4103450 2024-04-26T01:40:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e41abd30 con 0x7fe0e4103450 2024-04-26T01:40:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.682+0000 7fe0e1600700 1 -- 172.21.15.28:0/3588998880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc018680 con 0x7fe0e4103450 2024-04-26T01:40:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.683+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e41ac1c0 con 0x7fe0e4103450 2024-04-26T01:40:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.684+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e4104180 con 0x7fe0e4103450 2024-04-26T01:40:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.686+0000 7fe0e1600700 1 -- 172.21.15.28:0/3588998880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe0cc010430 con 0x7fe0e4103450 2024-04-26T01:40:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.687+0000 7fe0e1600700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe0c80a3c40 0x7fe0c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.687+0000 7fe0e1600700 1 -- 172.21.15.28:0/3588998880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fe0cc0d05f0 con 0x7fe0e4103450 2024-04-26T01:40:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.688+0000 7fe0e3e00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe0c80a3c40 0x7fe0c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.688+0000 7fe0e3e00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe0c80a3c40 0x7fe0c80a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4003e30 tx=0x7fe0d400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.692+0000 7fe0e1600700 1 -- 172.21.15.28:0/3588998880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0cc09cf80 con 0x7fe0e4103450 2024-04-26T01:40:26.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.975+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0e402e140 con 0x7fe0e4103450 2024-04-26T01:40:26.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.977+0000 7fe0e1600700 1 -- 172.21.15.28:0/3588998880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11861 (secure 0 0 0) 0x7fe0cc09c880 con 0x7fe0e4103450 2024-04-26T01:40:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:40:26.975 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:37:21.237658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:40:26.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe0c80a3c40 msgr2=0x7fe0c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:26.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe0c80a3c40 0x7fe0c80a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4003e30 tx=0x7fe0d400b040 comp rx=0 tx=0).stop 2024-04-26T01:40:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 msgr2=0x7fe0e4075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 0x7fe0e4075280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00e9a0 tx=0x7fe0cc00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 shutdown_connections 2024-04-26T01:40:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fe0c80a3c40 0x7fe0c80a6100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4108a50 0x7fe0e40757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0e4102a90 0x7fe0e4074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 --2- 172.21.15.28:0/3588998880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e4103450 0x7fe0e4075280 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.980+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 >> 172.21.15.28:0/3588998880 conn(0x7fe0e4079ba0 msgr2=0x7fe0e41017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.981+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 shutdown_connections 2024-04-26T01:40:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:26.981+0000 7fe0eaa00700 1 -- 172.21.15.28:0/3588998880 wait complete. 2024-04-26T01:40:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-26T01:40:27.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:27 smithi169 ceph-mon[32851]: pgmap v441: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:27.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:27 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3588998880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:27.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:27 smithi053 ceph-mon[32593]: pgmap v441: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:27.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:27 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3588998880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:27.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:27 smithi028 ceph-mon[28247]: pgmap v441: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:27.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:27 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3588998880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:27.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:28.359 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:29.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:29 smithi169 ceph-mon[32851]: pgmap v442: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:29.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:29 smithi053 ceph-mon[32593]: pgmap v442: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:29.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:29 smithi028 ceph-mon[28247]: pgmap v442: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 -- 172.21.15.28:0/2923695187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 msgr2=0x7f93a4111620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 --2- 172.21.15.28:0/2923695187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 0x7f93a4111620 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f939c00b0a0 tx=0x7f939c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 -- 172.21.15.28:0/2923695187 shutdown_connections 2024-04-26T01:40:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 --2- 172.21.15.28:0/2923695187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 0x7f93a4111620 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 --2- 172.21.15.28:0/2923695187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a4074e70 0x7f93a4075250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 --2- 172.21.15.28:0/2923695187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a4075790 0x7f93a410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 -- 172.21.15.28:0/2923695187 >> 172.21.15.28:0/2923695187 conn(0x7f93a40fe4c0 msgr2=0x7f93a41008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 -- 172.21.15.28:0/2923695187 shutdown_connections 2024-04-26T01:40:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.621+0000 7f93aac00700 1 -- 172.21.15.28:0/2923695187 wait complete. 2024-04-26T01:40:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aac00700 1 Processor -- start 2024-04-26T01:40:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aac00700 1 -- start start 2024-04-26T01:40:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aac00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a4074e70 0x7f93a419b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aac00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a4075790 0x7f93a419b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aac00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 0x7f93a419fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aac00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4114da0 con 0x7f93a4075790 2024-04-26T01:40:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aac00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4114c20 con 0x7f93a410f1b0 2024-04-26T01:40:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aac00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4114f20 con 0x7f93a4074e70 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.623+0000 7f93aa200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 0x7f93a419fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93a8e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a4075790 0x7f93a419b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93aa200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 0x7f93a419fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54196/0 (socket says 172.21.15.28:54196) 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93a9800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a4074e70 0x7f93a419b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93aa200700 1 -- 172.21.15.28:0/2710107792 learned_addr learned my addr 172.21.15.28:0/2710107792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93aa200700 1 -- 172.21.15.28:0/2710107792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a4074e70 msgr2=0x7f93a419b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93aa200700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a4074e70 0x7f93a419b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93aa200700 1 -- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a4075790 msgr2=0x7f93a419b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93aa200700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a4075790 0x7f93a419b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.624+0000 7f93aa200700 1 -- 172.21.15.28:0/2710107792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a41a0260 con 0x7f93a410f1b0 2024-04-26T01:40:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.625+0000 7f93aa200700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 0x7f93a419fb20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f939c00b070 tx=0x7f939c004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.625+0000 7f93a2000700 1 -- 172.21.15.28:0/2710107792 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c021070 con 0x7f93a410f1b0 2024-04-26T01:40:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.625+0000 7f93a2000700 1 -- 172.21.15.28:0/2710107792 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f939c007e30 con 0x7f93a410f1b0 2024-04-26T01:40:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.625+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a41a04f0 con 0x7f93a410f1b0 2024-04-26T01:40:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.625+0000 7f93a2000700 1 -- 172.21.15.28:0/2710107792 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c0046d0 con 0x7f93a410f1b0 2024-04-26T01:40:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.625+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a41a7dd0 con 0x7f93a410f1b0 2024-04-26T01:40:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.626+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93a4069730 con 0x7f93a410f1b0 2024-04-26T01:40:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.628+0000 7f93a2000700 1 -- 172.21.15.28:0/2710107792 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f939c007980 con 0x7f93a410f1b0 2024-04-26T01:40:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.630+0000 7f93a2000700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f938c0a3cc0 0x7f938c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.630+0000 7f93a2000700 1 -- 172.21.15.28:0/2710107792 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f939c00f020 con 0x7f93a410f1b0 2024-04-26T01:40:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.630+0000 7f93a9800700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f938c0a3cc0 0x7f938c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.631+0000 7f93a9800700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f938c0a3cc0 0x7f938c0a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9398009680 tx=0x7f9398009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.634+0000 7f93a2000700 1 -- 172.21.15.28:0/2710107792 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f939c02a020 con 0x7f93a410f1b0 2024-04-26T01:40:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.914+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93a402df80 con 0x7f93a410f1b0 2024-04-26T01:40:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.915+0000 7f93a2000700 1 -- 172.21.15.28:0/2710107792 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11861 (secure 0 0 0) 0x7f939c09e1e0 con 0x7f93a410f1b0 2024-04-26T01:40:30.913 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:40:30.913 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:37:21.237658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:40:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f938c0a3cc0 msgr2=0x7f938c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f938c0a3cc0 0x7f938c0a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9398009680 tx=0x7f9398009260 comp rx=0 tx=0).stop 2024-04-26T01:40:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 msgr2=0x7f93a419fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 0x7f93a419fb20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f939c00b070 tx=0x7f939c004cf0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 shutdown_connections 2024-04-26T01:40:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f938c0a3cc0 0x7f938c0a6180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93a410f1b0 0x7f93a419fb20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a4074e70 0x7f93a419b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 --2- 172.21.15.28:0/2710107792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a4075790 0x7f93a419b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 >> 172.21.15.28:0/2710107792 conn(0x7f93a40fe4c0 msgr2=0x7f93a40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 shutdown_connections 2024-04-26T01:40:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:30.919+0000 7f93aac00700 1 -- 172.21.15.28:0/2710107792 wait complete. 2024-04-26T01:40:30.918 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-26T01:40:31.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:31 smithi169 ceph-mon[32851]: pgmap v443: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:31.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:31 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2710107792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:31.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:31 smithi028 ceph-mon[28247]: pgmap v443: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:31.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:31 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2710107792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:31.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:31 smithi053 ceph-mon[32593]: pgmap v443: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:31.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:31 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2710107792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:31.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:40:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:33.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:33 smithi028 ceph-mon[28247]: pgmap v444: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:33.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:33 smithi169 ceph-mon[32851]: pgmap v444: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:33.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:33 smithi053 ceph-mon[32593]: pgmap v444: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 -- 172.21.15.28:0/3180306553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73c108a30 msgr2=0x7ff73c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 --2- 172.21.15.28:0/3180306553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73c108a30 0x7ff73c108e10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009990 tx=0x7ff72c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 -- 172.21.15.28:0/3180306553 shutdown_connections 2024-04-26T01:40:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 --2- 172.21.15.28:0/3180306553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff73c06b0f0 0x7ff73c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 --2- 172.21.15.28:0/3180306553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73c108a30 0x7ff73c108e10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 --2- 172.21.15.28:0/3180306553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff73c06a750 0x7ff73c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 -- 172.21.15.28:0/3180306553 >> 172.21.15.28:0/3180306553 conn(0x7ff73c0788f0 msgr2=0x7ff73c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 -- 172.21.15.28:0/3180306553 shutdown_connections 2024-04-26T01:40:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.439+0000 7ff741c00700 1 -- 172.21.15.28:0/3180306553 wait complete. 2024-04-26T01:40:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.440+0000 7ff741c00700 1 Processor -- start 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.440+0000 7ff741c00700 1 -- start start 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff741c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff73c06a750 0x7ff73c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff741c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73c06b0f0 0x7ff73c19b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff741c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff73c108a30 0x7ff73c19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff741c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c112c00 con 0x7ff73c06b0f0 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff741c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c112a80 con 0x7ff73c06a750 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff741c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c112d80 con 0x7ff73c108a30 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73aa00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff73c06a750 0x7ff73c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff732000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73c06b0f0 0x7ff73c19b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73aa00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff73c06a750 0x7ff73c19b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54214/0 (socket says 172.21.15.28:54214) 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73aa00700 1 -- 172.21.15.28:0/222244451 learned_addr learned my addr 172.21.15.28:0/222244451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73b400700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff73c108a30 0x7ff73c19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73aa00700 1 -- 172.21.15.28:0/222244451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff73c108a30 msgr2=0x7ff73c19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73aa00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff73c108a30 0x7ff73c19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73aa00700 1 -- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73c06b0f0 msgr2=0x7ff73c19b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73aa00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73c06b0f0 0x7ff73c19b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.441+0000 7ff73aa00700 1 -- 172.21.15.28:0/222244451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c1a0360 con 0x7ff73c06a750 2024-04-26T01:40:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.442+0000 7ff73aa00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff73c06a750 0x7ff73c19b350 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff72c000c00 tx=0x7ff72c0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.442+0000 7ff733e00700 1 -- 172.21.15.28:0/222244451 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c027070 con 0x7ff73c06a750 2024-04-26T01:40:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.442+0000 7ff733e00700 1 -- 172.21.15.28:0/222244451 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff72c022470 con 0x7ff73c06a750 2024-04-26T01:40:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.442+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73c1a05f0 con 0x7ff73c06a750 2024-04-26T01:40:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.442+0000 7ff733e00700 1 -- 172.21.15.28:0/222244451 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c017650 con 0x7ff73c06a750 2024-04-26T01:40:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.442+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff73c1a7ed0 con 0x7ff73c06a750 2024-04-26T01:40:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.443+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff73c069730 con 0x7ff73c06a750 2024-04-26T01:40:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.445+0000 7ff733e00700 1 -- 172.21.15.28:0/222244451 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff72c02a070 con 0x7ff73c06a750 2024-04-26T01:40:34.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.447+0000 7ff733e00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff7200a3cc0 0x7ff7200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:34.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.447+0000 7ff733e00700 1 -- 172.21.15.28:0/222244451 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7ff72c0d1070 con 0x7ff73c06a750 2024-04-26T01:40:34.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.447+0000 7ff732000700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff7200a3cc0 0x7ff7200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:34.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.448+0000 7ff732000700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff7200a3cc0 0x7ff7200a6180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff73c10fd40 tx=0x7ff724009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.452+0000 7ff733e00700 1 -- 172.21.15.28:0/222244451 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff72c013070 con 0x7ff73c06a750 2024-04-26T01:40:34.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.685+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff73c1a80a0 con 0x7ff73c06a750 2024-04-26T01:40:34.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.687+0000 7ff733e00700 1 -- 172.21.15.28:0/222244451 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22630 (secure 0 0 0) 0x7ff72c09daf0 con 0x7ff73c06a750 2024-04-26T01:40:34.685 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:40:34.685 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:37:25.472266+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":7,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:40:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.690+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff7200a3cc0 msgr2=0x7ff7200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.690+0000 7ff741c00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff7200a3cc0 0x7ff7200a6180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff73c10fd40 tx=0x7ff724009260 comp rx=0 tx=0).stop 2024-04-26T01:40:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.690+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff73c06a750 msgr2=0x7ff73c19b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.690+0000 7ff741c00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff73c06a750 0x7ff73c19b350 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff72c000c00 tx=0x7ff72c0057c0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.691+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 shutdown_connections 2024-04-26T01:40:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.691+0000 7ff741c00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7ff7200a3cc0 0x7ff7200a6180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.691+0000 7ff741c00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff73c108a30 0x7ff73c19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.691+0000 7ff741c00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff73c06a750 0x7ff73c19b350 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.691+0000 7ff741c00700 1 --2- 172.21.15.28:0/222244451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73c06b0f0 0x7ff73c19b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.691+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 >> 172.21.15.28:0/222244451 conn(0x7ff73c0788f0 msgr2=0x7ff73c0ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.691+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 shutdown_connections 2024-04-26T01:40:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:34.691+0000 7ff741c00700 1 -- 172.21.15.28:0/222244451 wait complete. 2024-04-26T01:40:35.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:36:57.152667+0000', 'modified': '2024-04-26T01:37:21.237658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24523, 'mds_1': 24509, 'mds_2': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24491': {'gid': 24491, 'name': 'g', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/2114874192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2114874192}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2114874192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24497': {'gid': 24497, 'name': 'a', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/2299937376', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 2299937376}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 2299937376}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.169:6835/1802608598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6834', 'nonce': 1802608598}, {'type': 'v1', 'addr': '172.21.15.169:6835', 'nonce': 1802608598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.169:6837/3777955238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6836', 'nonce': 3777955238}, {'type': 'v1', 'addr': '172.21.15.169:6837', 'nonce': 3777955238}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.53:6839/955096275', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6838', 'nonce': 955096275}, {'type': 'v1', 'addr': '172.21.15.53:6839', 'nonce': 955096275}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.169:6839/2157805186', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6838', 'nonce': 2157805186}, {'type': 'v1', 'addr': '172.21.15.169:6839', 'nonce': 2157805186}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24523, 'qdb_cluster': [24505, 24523, 24509]} 2024-04-26T01:40:35.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:40:35.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-26T01:40:35.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:35 smithi028 ceph-mon[28247]: pgmap v445: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:35.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:35 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/222244451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:40:35.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:35 smithi169 ceph-mon[32851]: pgmap v445: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:35.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:35 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/222244451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:40:35.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:35 smithi053 ceph-mon[32593]: pgmap v445: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:35.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:35 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/222244451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:40:35.991 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:37.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:37 smithi169 ceph-mon[32851]: pgmap v446: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:37.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:37 smithi053 ceph-mon[32593]: pgmap v446: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:37.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:37 smithi028 ceph-mon[28247]: pgmap v446: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 -- 172.21.15.28:0/3538380498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950108a50 msgr2=0x7f3950108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 --2- 172.21.15.28:0/3538380498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950108a50 0x7f3950108e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3940009990 tx=0x7f3940009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 -- 172.21.15.28:0/3538380498 shutdown_connections 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 --2- 172.21.15.28:0/3538380498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3950103450 0x7f395010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 --2- 172.21.15.28:0/3538380498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950108a50 0x7f3950108e30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 --2- 172.21.15.28:0/3538380498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3950102a90 0x7f3950102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 -- 172.21.15.28:0/3538380498 >> 172.21.15.28:0/3538380498 conn(0x7f3950079ba0 msgr2=0x7f3950079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 -- 172.21.15.28:0/3538380498 shutdown_connections 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.332+0000 7f3954a00700 1 -- 172.21.15.28:0/3538380498 wait complete. 2024-04-26T01:40:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.334+0000 7f3954a00700 1 Processor -- start 2024-04-26T01:40:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.334+0000 7f3954a00700 1 -- start start 2024-04-26T01:40:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f3954a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950102a90 0x7f395019b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f3954a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3950103450 0x7f395019b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f3954a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3950108a50 0x7f395019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f3954a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3950112c30 con 0x7f3950103450 2024-04-26T01:40:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f3954a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3950112ab0 con 0x7f3950102a90 2024-04-26T01:40:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f3954a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3950112db0 con 0x7f3950108a50 2024-04-26T01:40:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f394de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950102a90 0x7f395019b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f394de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950102a90 0x7f395019b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54238/0 (socket says 172.21.15.28:54238) 2024-04-26T01:40:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f394de00700 1 -- 172.21.15.28:0/3090977822 learned_addr learned my addr 172.21.15.28:0/3090977822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f394d400700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3950103450 0x7f395019b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.335+0000 7f394e800700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3950108a50 0x7f395019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f394de00700 1 -- 172.21.15.28:0/3090977822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3950108a50 msgr2=0x7f395019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f394de00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3950108a50 0x7f395019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f394de00700 1 -- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3950103450 msgr2=0x7f395019b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f394de00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3950103450 0x7f395019b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f394de00700 1 -- 172.21.15.28:0/3090977822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39501a03c0 con 0x7f3950102a90 2024-04-26T01:40:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f394de00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950102a90 0x7f395019b3b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3940000c00 tx=0x7f39400057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f3946a00700 1 -- 172.21.15.28:0/3090977822 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940027070 con 0x7f3950102a90 2024-04-26T01:40:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f3946a00700 1 -- 172.21.15.28:0/3090977822 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3940022470 con 0x7f3950102a90 2024-04-26T01:40:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f3954a00700 1 -- 172.21.15.28:0/3090977822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39501a0650 con 0x7f3950102a90 2024-04-26T01:40:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f3946a00700 1 -- 172.21.15.28:0/3090977822 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940017650 con 0x7f3950102a90 2024-04-26T01:40:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.336+0000 7f3954a00700 1 -- 172.21.15.28:0/3090977822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39501a7f30 con 0x7f3950102a90 2024-04-26T01:40:38.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.339+0000 7f3946a00700 1 -- 172.21.15.28:0/3090977822 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f394002a070 con 0x7f3950102a90 2024-04-26T01:40:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f395f419b80 1 Processor -- start 2024-04-26T01:40:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f3946a00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f39340a3cc0 0x7f39340a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f395f419b80 1 -- start start 2024-04-26T01:40:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f3946a00700 1 -- 172.21.15.28:0/3090977822 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f39400d10d0 con 0x7f3950102a90 2024-04-26T01:40:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f394e800700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f39340a3cc0 0x7f39340a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f395f419b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de17822f0 0x560de17826d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f395f419b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de1782c10 0x560de178d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f395f419b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f395f419b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560de16d5e40 con 0x560de1782c10 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f395f419b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560de167c6e0 con 0x560de178dab0 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.342+0000 7f395f419b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x560de16d1e00 con 0x560de17822f0 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de17822f0 0x560de17826d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de1782c10 0x560de178d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de17822f0 0x560de17826d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:41558/0 (socket says 172.21.15.28:41558) 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 -- 172.21.15.28:0/4167506407 learned_addr learned my addr 172.21.15.28:0/4167506407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394e800700 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394e800700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f39340a3cc0 0x7f39340a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3948009fd0 tx=0x7f394800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 -- 172.21.15.28:0/4167506407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 msgr2=0x560de178fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 -- 172.21.15.28:0/4167506407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de1782c10 msgr2=0x560de178d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de1782c10 0x560de178d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 -- 172.21.15.28:0/4167506407 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560de1672630 con 0x560de17822f0 2024-04-26T01:40:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394d400700 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de1782c10 0x560de178d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.343+0000 7f394de00700 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de17822f0 0x560de17826d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f394009d4a0 tx=0x7f3940099860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e165ded648978bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.344+0000 7f393b400700 1 -- 172.21.15.28:0/4167506407 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394009dc10 con 0x560de17822f0 2024-04-26T01:40:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.344+0000 7f393b400700 1 -- 172.21.15.28:0/4167506407 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f394009d510 con 0x560de17822f0 2024-04-26T01:40:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.344+0000 7f393b400700 1 -- 172.21.15.28:0/4167506407 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394009d770 con 0x560de17822f0 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.344+0000 7f395f419b80 1 -- 172.21.15.28:0/4167506407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de17822f0 msgr2=0x560de17826d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.344+0000 7f395f419b80 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de17822f0 0x560de17826d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f394009d4a0 tx=0x7f3940099860 comp rx=0 tx=0).stop 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 -- 172.21.15.28:0/4167506407 shutdown_connections 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de17822f0 0x560de17826d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 --2- 172.21.15.28:0/4167506407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de1782c10 0x560de178d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 -- 172.21.15.28:0/4167506407 >> 172.21.15.28:0/4167506407 conn(0x560de16b1550 msgr2=0x560de177e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 -- 172.21.15.28:0/4167506407 shutdown_connections 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 -- 172.21.15.28:0/4167506407 wait complete. 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 -- start start 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.345+0000 7f395f419b80 1 Processor -- start 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f395f419b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de178d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f395f419b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de178a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f395f419b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f395f419b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560de17903e0 con 0x560de17822f0 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f395f419b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560de168f810 con 0x560de178dab0 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f395f419b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x560de1790560 con 0x560de1782c10 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f395f419b80 10 client.0 _ll_register_callbacks cb 0x560de166e6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f394de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de178d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f394de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de178d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39968/0 (socket says 172.21.15.28:39968) 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f394e800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f394de00700 1 -- 172.21.15.28:0/480684973 learned_addr learned my addr 172.21.15.28:0/480684973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.346+0000 7f394d400700 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de178a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f394e800700 1 -- 172.21.15.28:0/480684973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 msgr2=0x560de178a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f394e800700 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de178a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f394e800700 1 -- 172.21.15.28:0/480684973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 msgr2=0x560de178d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f394e800700 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de178d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f394e800700 1 -- 172.21.15.28:0/480684973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560de1623880 con 0x560de178dab0 2024-04-26T01:40:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f394de00700 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de178d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f394e800700 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178a9e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3948009d70 tx=0x7f3948004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30313d072076edc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f392fe00700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f392fe00700 1 -- 172.21.15.28:0/480684973 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948026070 con 0x560de178dab0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f392fe00700 1 -- 172.21.15.28:0/480684973 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f3948017aa0 con 0x560de178dab0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.347+0000 7f395f419b80 1 -- 172.21.15.28:0/480684973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f394009d510 con 0x560de178dab0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f392fe00700 1 -- 172.21.15.28:0/480684973 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3948032050 con 0x560de178dab0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f395f419b80 10 client.24971 fetch_fsmap learned FSMap version 24 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f395f419b80 1 -- 172.21.15.28:0/480684973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560de166caa0 con 0x560de178dab0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f392fe00700 1 -- 172.21.15.28:0/480684973 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f394009d510 con 0x560de178dab0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f392fe00700 1 -- 172.21.15.28:0/480684973 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3948015420 con 0x560de178dab0 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f395f419b80 10 client.24971 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:40:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f395f419b80 10 client.24971 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f395f419b80 20 client.24971 populate_metadata read hostname 'smithi028' 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f395f419b80 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x560de179d080 0x560de179e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f395f419b80 4 client.24971 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.348+0000 7f395f419b80 1 -- 172.21.15.28:0/480684973 --> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560de16e3630 con 0x560de179d080 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.349+0000 7f394de00700 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x560de179d080 0x560de179e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.349+0000 7f394de00700 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x560de179d080 0x560de179e4f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1cb3689b1a642e8b server_cookie=67bfe429af3ae329 in_seq=0 out_seq=0 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.349+0000 7f392fe00700 10 client.24971 ms_handle_connect on v2:172.21.15.169:6836/3777955238 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f392fe00700 1 -- 172.21.15.28:0/480684973 <== mds.2 v2:172.21.15.169:6836/3777955238 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560de16e3630 con 0x560de179d080 2024-04-26T01:40:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f392fe00700 10 client.24971 handle_command_reply: tid=0 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f395f419b80 1 client.24971 shutdown 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f395f419b80 1 -- 172.21.15.28:0/480684973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 msgr2=0x560de178a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f395f419b80 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178a9e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3948009d70 tx=0x7f3948004ad0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f395f419b80 1 -- 172.21.15.28:0/480684973 shutdown_connections 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f395f419b80 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x560de179d080 0x560de179e4f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f395f419b80 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de178a9e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f395f419b80 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de178d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f392fe00700 0 client.24971 ms_handle_reset on v2:172.21.15.169:6836/3777955238 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.350+0000 7f395f419b80 1 --2- 172.21.15.28:0/480684973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de178a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.351+0000 7f395f419b80 1 -- 172.21.15.28:0/480684973 >> 172.21.15.28:0/480684973 conn(0x560de16b1550 msgr2=0x560de177d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.351+0000 7f395f419b80 1 -- 172.21.15.28:0/480684973 shutdown_connections 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.351+0000 7f395f419b80 1 -- 172.21.15.28:0/480684973 wait complete. 2024-04-26T01:40:38.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.351+0000 7f395f419b80 20 client.24971 trim_cache size 0 max 16384 2024-04-26T01:40:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.373+0000 7f395f419b80 1 Processor -- start 2024-04-26T01:40:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.373+0000 7f395f419b80 1 -- start start 2024-04-26T01:40:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.373+0000 7f395f419b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de1785b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.373+0000 7f395f419b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de177e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.373+0000 7f395f419b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de177e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.373+0000 7f395f419b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560de179c830 con 0x560de17822f0 2024-04-26T01:40:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.373+0000 7f395f419b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560de179cf00 con 0x560de178dab0 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.373+0000 7f395f419b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x560de179d5d0 con 0x560de1782c10 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de1785b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de1785b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39982/0 (socket says 172.21.15.28:39982) 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 -- 172.21.15.28:0/686123906 learned_addr learned my addr 172.21.15.28:0/686123906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394e800700 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de177e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394d400700 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de177e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 -- 172.21.15.28:0/686123906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 msgr2=0x560de177e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de177e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 -- 172.21.15.28:0/686123906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 msgr2=0x560de177e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de177e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 -- 172.21.15.28:0/686123906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560de1788a10 con 0x560de17822f0 2024-04-26T01:40:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.374+0000 7f394de00700 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de1785b60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3940099970 tx=0x7f394001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2f5a35eadb7e43 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f393aa00700 1 -- 172.21.15.28:0/686123906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394009d510 con 0x560de17822f0 2024-04-26T01:40:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f393aa00700 1 -- 172.21.15.28:0/686123906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f394009dc10 con 0x560de17822f0 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f393aa00700 1 -- 172.21.15.28:0/686123906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394009d970 con 0x560de17822f0 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f395f419b80 1 -- 172.21.15.28:0/686123906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 msgr2=0x560de1785b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f395f419b80 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de1785b60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3940099970 tx=0x7f394001a040 comp rx=0 tx=0).stop 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f395f419b80 1 -- 172.21.15.28:0/686123906 shutdown_connections 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f395f419b80 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de178dab0 0x560de177e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f395f419b80 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de17822f0 0x560de1785b60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f395f419b80 1 --2- 172.21.15.28:0/686123906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de177e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.375+0000 7f395f419b80 1 -- 172.21.15.28:0/686123906 >> 172.21.15.28:0/686123906 conn(0x560de16b1550 msgr2=0x560de1785450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.376+0000 7f395f419b80 1 -- 172.21.15.28:0/686123906 shutdown_connections 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.376+0000 7f395f419b80 1 -- 172.21.15.28:0/686123906 wait complete. 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.376+0000 7f395f419b80 1 -- start start 2024-04-26T01:40:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.376+0000 7f395f419b80 1 Processor -- start 2024-04-26T01:40:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f395f419b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de17822f0 0x560de17a7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f395f419b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de17a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f395f419b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de178dab0 0x560de1787ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f395f419b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560de179e370 con 0x560de178dab0 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f395f419b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560de179dca0 con 0x560de17822f0 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f395f419b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x560de179ea40 con 0x560de1782c10 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f395f419b80 10 client.0 _ll_register_callbacks cb 0x560de166e6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f394e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de178dab0 0x560de1787ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f394d400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de17a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f394e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de178dab0 0x560de1787ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39992/0 (socket says 172.21.15.28:39992) 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f394d400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de17a7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:41600/0 (socket says 172.21.15.28:41600) 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f394e800700 1 -- 172.21.15.28:0/3227220109 learned_addr learned my addr 172.21.15.28:0/3227220109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.377+0000 7f394de00700 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de17822f0 0x560de17a7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f394d400700 1 -- 172.21.15.28:0/3227220109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de17822f0 msgr2=0x560de17a7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f394d400700 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de17822f0 0x560de17a7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f394d400700 1 -- 172.21.15.28:0/3227220109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de178dab0 msgr2=0x560de1787ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f394d400700 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de178dab0 0x560de1787ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f394d400700 1 -- 172.21.15.28:0/3227220109 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560de17884f0 con 0x560de1782c10 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f394de00700 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de17822f0 0x560de17a7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f394e800700 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de178dab0 0x560de1787ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f394d400700 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de17a7ea0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x560de17852c0 tx=0x7f393c00bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4234e073322504c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f392fe00700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f392fe00700 1 -- 172.21.15.28:0/3227220109 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c004830 con 0x560de1782c10 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f395f419b80 1 -- 172.21.15.28:0/3227220109 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f394009dc10 con 0x560de1782c10 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.378+0000 7f392fe00700 1 -- 172.21.15.28:0/3227220109 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f393c0049d0 con 0x560de1782c10 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f392fe00700 1 -- 172.21.15.28:0/3227220109 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f393c004bd0 con 0x560de1782c10 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f395f419b80 10 client.25018 fetch_fsmap learned FSMap version 24 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f395f419b80 1 -- 172.21.15.28:0/3227220109 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560de1804260 con 0x560de1782c10 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f392fe00700 1 -- 172.21.15.28:0/3227220109 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f394009dc10 con 0x560de1782c10 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f392fe00700 1 -- 172.21.15.28:0/3227220109 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f393c01c580 con 0x560de1782c10 2024-04-26T01:40:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f395f419b80 10 client.25018 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:40:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f395f419b80 10 client.25018 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:40:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f395f419b80 20 client.25018 populate_metadata read hostname 'smithi028' 2024-04-26T01:40:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f395f419b80 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x560de1808d70 0x560de1807a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f395f419b80 4 client.25018 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:40:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.379+0000 7f395f419b80 1 -- 172.21.15.28:0/3227220109 --> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] -- command(tid 0: {"prefix": "status"}) -- 0x560de16dd6d0 con 0x560de1808d70 2024-04-26T01:40:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.380+0000 7f394de00700 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x560de1808d70 0x560de1807a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.380+0000 7f394de00700 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x560de1808d70 0x560de1807a80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e95b73b940aadffa server_cookie=c8cec4d95a835031 in_seq=0 out_seq=0 2024-04-26T01:40:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.380+0000 7f392fe00700 10 client.25018 ms_handle_connect on v2:172.21.15.169:6836/3777955238 2024-04-26T01:40:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.380+0000 7f392fe00700 1 -- 172.21.15.28:0/3227220109 <== mds.2 v2:172.21.15.169:6836/3777955238 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560de16dd6d0 con 0x560de1808d70 2024-04-26T01:40:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.380+0000 7f392fe00700 10 client.25018 handle_command_reply: tid=0 2024-04-26T01:40:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.380+0000 7f395f419b80 1 client.25018 shutdown 2024-04-26T01:40:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 -- 172.21.15.28:0/3227220109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 msgr2=0x560de17a7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de17a7ea0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x560de17852c0 tx=0x7f393c00bf90 comp rx=0 tx=0).stop 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 -- 172.21.15.28:0/3227220109 shutdown_connections 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x560de1808d70 0x560de1807a80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560de178dab0 0x560de1787ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560de17822f0 0x560de17a7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 --2- 172.21.15.28:0/3227220109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560de1782c10 0x560de17a7ea0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f392fe00700 0 client.25018 ms_handle_reset on v2:172.21.15.169:6836/3777955238 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 -- 172.21.15.28:0/3227220109 >> 172.21.15.28:0/3227220109 conn(0x560de16b1550 msgr2=0x560de1788d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 -- 172.21.15.28:0/3227220109 shutdown_connections 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 1 -- 172.21.15.28:0/3227220109 wait complete. 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.381+0000 7f395f419b80 20 client.25018 trim_cache size 0 max 16384 2024-04-26T01:40:38.381 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24505, 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 203.98385517899999, 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 262.18992435500002 2024-04-26T01:40:38.382 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:40:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 -- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f39340a3cc0 msgr2=0x7f39340a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f39340a3cc0 0x7f39340a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3948009fd0 tx=0x7f394800b040 comp rx=0 tx=0).stop 2024-04-26T01:40:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 -- 172.21.15.28:0/3090977822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950102a90 msgr2=0x7f395019b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950102a90 0x7f395019b3b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3940000c00 tx=0x7f39400057c0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 -- 172.21.15.28:0/3090977822 shutdown_connections 2024-04-26T01:40:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f39340a3cc0 0x7f39340a6180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3950108a50 0x7f395019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950102a90 0x7f395019b3b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 --2- 172.21.15.28:0/3090977822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3950103450 0x7f395019b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 -- 172.21.15.28:0/3090977822 >> 172.21.15.28:0/3090977822 conn(0x7f3950079ba0 msgr2=0x7f395010df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 -- 172.21.15.28:0/3090977822 shutdown_connections 2024-04-26T01:40:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:38.382+0000 7f3954a00700 1 -- 172.21.15.28:0/3090977822 wait complete. 2024-04-26T01:40:39.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:40:39.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:39 smithi028 ceph-mon[28247]: pgmap v447: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:39.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:39 smithi169 ceph-mon[32851]: pgmap v447: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:39.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:39 smithi053 ceph-mon[32593]: pgmap v447: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:41.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:41 smithi028 ceph-mon[28247]: pgmap v448: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:41.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:41 smithi169 ceph-mon[32851]: pgmap v448: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:41.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:41 smithi053 ceph-mon[32593]: pgmap v448: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 -- 172.21.15.28:0/1534793833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c01092d0 msgr2=0x7fa9c01096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 --2- 172.21.15.28:0/1534793833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c01092d0 0x7fa9c01096b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00099c0 tx=0x7fa9b0009cd0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 -- 172.21.15.28:0/1534793833 shutdown_connections 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 --2- 172.21.15.28:0/1534793833 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9c010e3b0 0x7fa9c0115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 --2- 172.21.15.28:0/1534793833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c01092d0 0x7fa9c01096b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 --2- 172.21.15.28:0/1534793833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c0109c80 0x7fa9c010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 -- 172.21.15.28:0/1534793833 >> 172.21.15.28:0/1534793833 conn(0x7fa9c0076210 msgr2=0x7fa9c0107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 -- 172.21.15.28:0/1534793833 shutdown_connections 2024-04-26T01:40:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.146+0000 7fa9c5000700 1 -- 172.21.15.28:0/1534793833 wait complete. 2024-04-26T01:40:42.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.148+0000 7fa9c5000700 1 Processor -- start 2024-04-26T01:40:42.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.148+0000 7fa9c5000700 1 -- start start 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.148+0000 7fa9c5000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c01092d0 0x7fa9c01a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.148+0000 7fa9c5000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c0109c80 0x7fa9c01a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9c5000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9c010e3b0 0x7fa9c019f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9c5000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0119290 con 0x7fa9c0109c80 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9c5000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0119110 con 0x7fa9c01092d0 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9c5000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0119410 con 0x7fa9c010e3b0 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9bde00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c01092d0 0x7fa9c01a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9bde00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c01092d0 0x7fa9c01a53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35894/0 (socket says 172.21.15.28:35894) 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9bde00700 1 -- 172.21.15.28:0/281023582 learned_addr learned my addr 172.21.15.28:0/281023582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c0109c80 0x7fa9c01a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9be800700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9c010e3b0 0x7fa9c019f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9bde00700 1 -- 172.21.15.28:0/281023582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9c010e3b0 msgr2=0x7fa9c019f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9bde00700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9c010e3b0 0x7fa9c019f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9bde00700 1 -- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c0109c80 msgr2=0x7fa9c01a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9bde00700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c0109c80 0x7fa9c01a5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.149+0000 7fa9bde00700 1 -- 172.21.15.28:0/281023582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c019fcf0 con 0x7fa9c01092d0 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.150+0000 7fa9bde00700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c01092d0 0x7fa9c01a53d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0006010 tx=0x7fa9b0005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.150+0000 7fa9b6a00700 1 -- 172.21.15.28:0/281023582 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0026070 con 0x7fa9c01092d0 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.150+0000 7fa9c5000700 1 -- 172.21.15.28:0/281023582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c019ff80 con 0x7fa9c01092d0 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.150+0000 7fa9b6a00700 1 -- 172.21.15.28:0/281023582 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9b0005680 con 0x7fa9c01092d0 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.150+0000 7fa9b6a00700 1 -- 172.21.15.28:0/281023582 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b000fb20 con 0x7fa9c01092d0 2024-04-26T01:40:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.150+0000 7fa9c5000700 1 -- 172.21.15.28:0/281023582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c01ac2a0 con 0x7fa9c01092d0 2024-04-26T01:40:42.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.154+0000 7fa9b6a00700 1 -- 172.21.15.28:0/281023582 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa9b0029070 con 0x7fa9c01092d0 2024-04-26T01:40:42.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.155+0000 7fa9cf92db80 1 Processor -- start 2024-04-26T01:40:42.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.155+0000 7fa9cf92db80 1 -- start start 2024-04-26T01:40:42.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.155+0000 7fa9b6a00700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa9a40a3cc0 0x7fa9a40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.155+0000 7fa9b6a00700 1 -- 172.21.15.28:0/281023582 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fa9b00d0b50 con 0x7fa9c01092d0 2024-04-26T01:40:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.155+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9be800700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa9a40a3cc0 0x7fa9a40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f20a0 0x564e074eb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074ec330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e0743de40 con 0x564e074f1780 2024-04-26T01:40:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564e073e46e0 con 0x564e074ebef0 2024-04-26T01:40:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x564e07439e00 con 0x564e074f20a0 2024-04-26T01:40:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9b7e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f20a0 0x564e074eb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9b7e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f20a0 0x564e074eb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59262/0 (socket says 172.21.15.28:59262) 2024-04-26T01:40:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9b7e00700 1 -- 172.21.15.28:0/558658849 learned_addr learned my addr 172.21.15.28:0/558658849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9be800700 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074ec330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.156+0000 7fa9bde00700 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9b7e00700 1 -- 172.21.15.28:0/558658849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 msgr2=0x564e074ec330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:42.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074ec330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9b7e00700 1 -- 172.21.15.28:0/558658849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 msgr2=0x564e074f1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9b7e00700 1 -- 172.21.15.28:0/558658849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e073da630 con 0x564e074f20a0 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9be800700 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074ec330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9be800700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa9a40a3cc0 0x7fa9a40a6180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800f870 tx=0x7fa9b800fc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f20a0 0x564e074eb9b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009cf0 tx=0x7fa9a8011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3ed45146cf341c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9af400700 1 -- 172.21.15.28:0/558658849 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8011d80 con 0x564e074f20a0 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9af400700 1 -- 172.21.15.28:0/558658849 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a8004500 con 0x564e074f20a0 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.157+0000 7fa9af400700 1 -- 172.21.15.28:0/558658849 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8005280 con 0x564e074f20a0 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 -- 172.21.15.28:0/558658849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f20a0 msgr2=0x564e074eb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f20a0 0x564e074eb9b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009cf0 tx=0x7fa9a8011440 comp rx=0 tx=0).stop 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 -- 172.21.15.28:0/558658849 shutdown_connections 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074ec330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/558658849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f20a0 0x564e074eb9b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 -- 172.21.15.28:0/558658849 >> 172.21.15.28:0/558658849 conn(0x564e07419f00 msgr2=0x564e074e68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:42.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 -- 172.21.15.28:0/558658849 shutdown_connections 2024-04-26T01:40:42.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.158+0000 7fa9cf92db80 1 -- 172.21.15.28:0/558658849 wait complete. 2024-04-26T01:40:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.159+0000 7fa9cf92db80 1 -- start start 2024-04-26T01:40:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.159+0000 7fa9cf92db80 1 Processor -- start 2024-04-26T01:40:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f1780 0x564e074f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f5460 0x564e074f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e074f8920 con 0x564e074f5460 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564e073f7810 con 0x564e074ebef0 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x564e074f8aa0 con 0x564e074f1780 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9cf92db80 10 client.0 _ll_register_callbacks cb 0x564e073d66d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9be800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f5460 0x564e074f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9be800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f5460 0x564e074f58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55704/0 (socket says 172.21.15.28:55704) 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9be800700 1 -- 172.21.15.28:0/2805717810 learned_addr learned my addr 172.21.15.28:0/2805717810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9bde00700 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.160+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f1780 0x564e074f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9be800700 1 -- 172.21.15.28:0/2805717810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f1780 msgr2=0x564e074f4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9be800700 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f1780 0x564e074f4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9be800700 1 -- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 msgr2=0x564e074f7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9be800700 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074f7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9be800700 1 -- 172.21.15.28:0/2805717810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e0738b880 con 0x564e074f5460 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9bde00700 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9be800700 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f5460 0x564e074f58e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80060b0 tx=0x7fa9b8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2ea3f18163bfc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:40:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9a3e00700 1 -- 172.21.15.28:0/2805717810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8027030 con 0x564e074f5460 2024-04-26T01:40:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9a3e00700 1 -- 172.21.15.28:0/2805717810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fa9b8008c80 con 0x564e074f5460 2024-04-26T01:40:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.161+0000 7fa9cf92db80 1 -- 172.21.15.28:0/2805717810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9a8004500 con 0x564e074f5460 2024-04-26T01:40:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9a3e00700 1 -- 172.21.15.28:0/2805717810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa9b8003a00 con 0x564e074f5460 2024-04-26T01:40:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9cf92db80 10 client.14886 fetch_fsmap learned FSMap version 24 2024-04-26T01:40:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9cf92db80 1 -- 172.21.15.28:0/2805717810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e073d4aa0 con 0x564e074f5460 2024-04-26T01:40:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9a3e00700 1 -- 172.21.15.28:0/2805717810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a8004500 con 0x564e074f5460 2024-04-26T01:40:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9a3e00700 1 -- 172.21.15.28:0/2805717810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fa9b80045f0 con 0x564e074f5460 2024-04-26T01:40:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9cf92db80 10 client.14886 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:40:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9cf92db80 10 client.14886 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9cf92db80 20 client.14886 populate_metadata read hostname 'smithi028' 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x564e07505a40 0x564e07506eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9cf92db80 4 client.14886 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.162+0000 7fa9cf92db80 1 -- 172.21.15.28:0/2805717810 --> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e0744b630 con 0x564e07505a40 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.163+0000 7fa9bde00700 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x564e07505a40 0x564e07506eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.163+0000 7fa9bde00700 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x564e07505a40 0x564e07506eb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a6f0a517e60db95e server_cookie=d4839ec948506d39 in_seq=0 out_seq=0 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.163+0000 7fa9a3e00700 10 client.14886 ms_handle_connect on v2:172.21.15.53:6838/955096275 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.164+0000 7fa9a3e00700 1 -- 172.21.15.28:0/2805717810 <== mds.1 v2:172.21.15.53:6838/955096275 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e0744b630 con 0x564e07505a40 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.164+0000 7fa9a3e00700 10 client.14886 handle_command_reply: tid=0 2024-04-26T01:40:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.164+0000 7fa9cf92db80 1 client.14886 shutdown 2024-04-26T01:40:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.164+0000 7fa9cf92db80 1 -- 172.21.15.28:0/2805717810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f5460 msgr2=0x564e074f58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.164+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f5460 0x564e074f58e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80060b0 tx=0x7fa9b8004b40 comp rx=0 tx=0).stop 2024-04-26T01:40:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 1 -- 172.21.15.28:0/2805717810 shutdown_connections 2024-04-26T01:40:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x564e07505a40 0x564e07506eb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f5460 0x564e074f58e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074ebef0 0x564e074f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/2805717810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f1780 0x564e074f4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9a3e00700 0 client.14886 ms_handle_reset on v2:172.21.15.53:6838/955096275 2024-04-26T01:40:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 1 -- 172.21.15.28:0/2805717810 >> 172.21.15.28:0/2805717810 conn(0x564e07419f00 msgr2=0x564e074e5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 1 -- 172.21.15.28:0/2805717810 shutdown_connections 2024-04-26T01:40:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 1 -- 172.21.15.28:0/2805717810 wait complete. 2024-04-26T01:40:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.165+0000 7fa9cf92db80 20 client.14886 trim_cache size 0 max 16384 2024-04-26T01:40:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.188+0000 7fa9cf92db80 1 Processor -- start 2024-04-26T01:40:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.188+0000 7fa9cf92db80 1 -- start start 2024-04-26T01:40:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.188+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074ebef0 0x564e074ec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.188+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.188+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f5460 0x564e074e65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.188+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e075051f0 con 0x564e074f1780 2024-04-26T01:40:42.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.188+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564e075058c0 con 0x564e074f5460 2024-04-26T01:40:42.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.188+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x564e07505f90 con 0x564e074ebef0 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9be800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f5460 0x564e074e65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55718/0 (socket says 172.21.15.28:55718) 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 -- 172.21.15.28:0/3176800888 learned_addr learned my addr 172.21.15.28:0/3176800888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9bde00700 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074ebef0 0x564e074ec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 -- 172.21.15.28:0/3176800888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074ebef0 msgr2=0x564e074ec2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074ebef0 0x564e074ec2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 -- 172.21.15.28:0/3176800888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f5460 msgr2=0x564e074e65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f5460 0x564e074e65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 -- 172.21.15.28:0/3176800888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e074ecae0 con 0x564e074f1780 2024-04-26T01:40:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9be800700 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f5460 0x564e074e65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.189+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f7c20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009c10 tx=0x7fa9a8004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6029b4bba38bdb95 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.190+0000 7fa9aea00700 1 -- 172.21.15.28:0/3176800888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8028020 con 0x564e074f1780 2024-04-26T01:40:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.190+0000 7fa9aea00700 1 -- 172.21.15.28:0/3176800888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a8011b80 con 0x564e074f1780 2024-04-26T01:40:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.190+0000 7fa9aea00700 1 -- 172.21.15.28:0/3176800888 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8005070 con 0x564e074f1780 2024-04-26T01:40:42.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.190+0000 7fa9cf92db80 1 -- 172.21.15.28:0/3176800888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 msgr2=0x564e074f7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.190+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f7c20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009c10 tx=0x7fa9a8004da0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 -- 172.21.15.28:0/3176800888 shutdown_connections 2024-04-26T01:40:42.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f5460 0x564e074e65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074ebef0 0x564e074ec2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/3176800888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074f1780 0x564e074f7c20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 -- 172.21.15.28:0/3176800888 >> 172.21.15.28:0/3176800888 conn(0x564e07419f00 msgr2=0x564e074f4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 -- 172.21.15.28:0/3176800888 shutdown_connections 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 -- 172.21.15.28:0/3176800888 wait complete. 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 -- start start 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.191+0000 7fa9cf92db80 1 Processor -- start 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074ebef0 0x564e07510330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f1780 0x564e07510870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9cf92db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f5460 0x564e0750b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e07506d30 con 0x564e074ebef0 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564e07506660 con 0x564e074f1780 2024-04-26T01:40:42.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9cf92db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x564e07507400 con 0x564e074f5460 2024-04-26T01:40:42.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9cf92db80 10 client.0 _ll_register_callbacks cb 0x564e073d66d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:40:42.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9bde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074ebef0 0x564e07510330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9bde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074ebef0 0x564e07510330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55734/0 (socket says 172.21.15.28:55734) 2024-04-26T01:40:42.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9bde00700 1 -- 172.21.15.28:0/986648047 learned_addr learned my addr 172.21.15.28:0/986648047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:42.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f1780 0x564e07510870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.192+0000 7fa9be800700 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f5460 0x564e0750b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9bde00700 1 -- 172.21.15.28:0/986648047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f5460 msgr2=0x564e0750b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9bde00700 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f5460 0x564e0750b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9bde00700 1 -- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f1780 msgr2=0x564e07510870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9bde00700 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f1780 0x564e07510870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9bde00700 1 -- 172.21.15.28:0/986648047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e0750bd20 con 0x564e074ebef0 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f1780 0x564e07510870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9bde00700 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074ebef0 0x564e07510330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009fd0 tx=0x7fa9b0009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd06a9550cd13bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9a3e00700 1 -- 172.21.15.28:0/986648047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b00ba070 con 0x564e074ebef0 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9a3e00700 1 -- 172.21.15.28:0/986648047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fa9b00ce710 con 0x564e074ebef0 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.193+0000 7fa9cf92db80 1 -- 172.21.15.28:0/986648047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9a8011b80 con 0x564e074ebef0 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9a3e00700 1 -- 172.21.15.28:0/986648047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa9b009ce70 con 0x564e074ebef0 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9cf92db80 10 client.14892 fetch_fsmap learned FSMap version 24 2024-04-26T01:40:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9cf92db80 1 -- 172.21.15.28:0/986648047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e0756d810 con 0x564e074ebef0 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9a3e00700 1 -- 172.21.15.28:0/986648047 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a8011b80 con 0x564e074ebef0 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9a3e00700 1 -- 172.21.15.28:0/986648047 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fa9b007bce0 con 0x564e074ebef0 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9cf92db80 10 client.14892 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9cf92db80 10 client.14892 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9cf92db80 20 client.14892 populate_metadata read hostname 'smithi028' 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x564e07572440 0x564e07571150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9cf92db80 4 client.14892 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.194+0000 7fa9cf92db80 1 -- 172.21.15.28:0/986648047 --> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] -- command(tid 0: {"prefix": "status"}) -- 0x564e074456d0 con 0x564e07572440 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.195+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x564e07572440 0x564e07571150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.195+0000 7fa9b7e00700 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x564e07572440 0x564e07571150 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee3061fb9c23cba8 server_cookie=e8758545d52ed72b in_seq=0 out_seq=0 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.195+0000 7fa9a3e00700 10 client.14892 ms_handle_connect on v2:172.21.15.53:6838/955096275 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.195+0000 7fa9a3e00700 1 -- 172.21.15.28:0/986648047 <== mds.1 v2:172.21.15.53:6838/955096275 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564e074456d0 con 0x564e07572440 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9a3e00700 10 client.14892 handle_command_reply: tid=0 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 client.14892 shutdown 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 -- 172.21.15.28:0/986648047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074ebef0 msgr2=0x564e07510330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074ebef0 0x564e07510330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009fd0 tx=0x7fa9b0009f40 comp rx=0 tx=0).stop 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 -- 172.21.15.28:0/986648047 shutdown_connections 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e074f5460 0x564e0750b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x564e07572440 0x564e07571150 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e074ebef0 0x564e07510330 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 --2- 172.21.15.28:0/986648047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564e074f1780 0x564e07510870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 -- 172.21.15.28:0/986648047 >> 172.21.15.28:0/986648047 conn(0x564e07419f00 msgr2=0x564e074f7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9a3e00700 0 client.14892 ms_handle_reset on v2:172.21.15.53:6838/955096275 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 -- 172.21.15.28:0/986648047 shutdown_connections 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 1 -- 172.21.15.28:0/986648047 wait complete. 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.196+0000 7fa9cf92db80 20 client.14892 trim_cache size 0 max 16384 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:40:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24509, 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 208.82043670100001, 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 246.335887224 2024-04-26T01:40:42.197 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 -- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa9a40a3cc0 msgr2=0x7fa9a40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa9a40a3cc0 0x7fa9a40a6180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800f870 tx=0x7fa9b800fc30 comp rx=0 tx=0).stop 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 -- 172.21.15.28:0/281023582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c01092d0 msgr2=0x7fa9c01a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c01092d0 0x7fa9c01a53d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0006010 tx=0x7fa9b0005280 comp rx=0 tx=0).stop 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 -- 172.21.15.28:0/281023582 shutdown_connections 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa9a40a3cc0 0x7fa9a40a6180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9c010e3b0 0x7fa9c019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c01092d0 0x7fa9c01a53d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 --2- 172.21.15.28:0/281023582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c0109c80 0x7fa9c01a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 -- 172.21.15.28:0/281023582 >> 172.21.15.28:0/281023582 conn(0x7fa9c0076210 msgr2=0x7fa9c0077dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 -- 172.21.15.28:0/281023582 shutdown_connections 2024-04-26T01:40:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:42.197+0000 7fa9c5000700 1 -- 172.21.15.28:0/281023582 wait complete. 2024-04-26T01:40:43.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:40:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:43.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:43 smithi169 ceph-mon[32851]: pgmap v449: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:43.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:43 smithi053 ceph-mon[32593]: pgmap v449: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:43.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:43 smithi028 ceph-mon[28247]: pgmap v449: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:40:45.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:45 smithi028 ceph-mon[28247]: pgmap v450: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:45.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:45 smithi169 ceph-mon[32851]: pgmap v450: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:45.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:45 smithi053 ceph-mon[32593]: pgmap v450: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 -- 172.21.15.28:0/343434802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf201019d0 msgr2=0x7fdf2010f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 --2- 172.21.15.28:0/343434802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf201019d0 0x7fdf2010f910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800b0a0 tx=0x7fdf1800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 -- 172.21.15.28:0/343434802 shutdown_connections 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 --2- 172.21.15.28:0/343434802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf201019d0 0x7fdf2010f910 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 --2- 172.21.15.28:0/343434802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf20108ad0 0x7fdf20108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 --2- 172.21.15.28:0/343434802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf20101030 0x7fdf20101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 -- 172.21.15.28:0/343434802 >> 172.21.15.28:0/343434802 conn(0x7fdf20077fb0 msgr2=0x7fdf200ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 -- 172.21.15.28:0/343434802 shutdown_connections 2024-04-26T01:40:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.914+0000 7fdf28000700 1 -- 172.21.15.28:0/343434802 wait complete. 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.915+0000 7fdf28000700 1 Processor -- start 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf28000700 1 -- start start 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf28000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf20101030 0x7fdf20102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf28000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf201019d0 0x7fdf20102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf28000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf20108ad0 0x7fdf20102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf28000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf20112be0 con 0x7fdf20108ad0 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf28000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdf20112a60 con 0x7fdf20101030 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf28000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdf20112d60 con 0x7fdf201019d0 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf24c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf201019d0 0x7fdf20102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf26000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf20108ad0 0x7fdf20102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf25600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf20101030 0x7fdf20102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf24c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf201019d0 0x7fdf20102750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59304/0 (socket says 172.21.15.28:59304) 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.916+0000 7fdf24c00700 1 -- 172.21.15.28:0/1470093238 learned_addr learned my addr 172.21.15.28:0/1470093238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.917+0000 7fdf26000700 1 -- 172.21.15.28:0/1470093238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf201019d0 msgr2=0x7fdf20102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.917+0000 7fdf26000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf201019d0 0x7fdf20102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.917+0000 7fdf26000700 1 -- 172.21.15.28:0/1470093238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf20101030 msgr2=0x7fdf20102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.917+0000 7fdf26000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf20101030 0x7fdf20102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.917+0000 7fdf26000700 1 -- 172.21.15.28:0/1470093238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf201a75e0 con 0x7fdf20108ad0 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.917+0000 7fdf24c00700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf201019d0 0x7fdf20102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.917+0000 7fdf25600700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf20101030 0x7fdf20102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.917+0000 7fdf26000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf20108ad0 0x7fdf20102c90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800b070 tx=0x7fdf18004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.918+0000 7fdf1e000700 1 -- 172.21.15.28:0/1470093238 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf18021070 con 0x7fdf20108ad0 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.918+0000 7fdf1e000700 1 -- 172.21.15.28:0/1470093238 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf18007e30 con 0x7fdf20108ad0 2024-04-26T01:40:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.918+0000 7fdf28000700 1 -- 172.21.15.28:0/1470093238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf201a7870 con 0x7fdf20108ad0 2024-04-26T01:40:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.918+0000 7fdf1e000700 1 -- 172.21.15.28:0/1470093238 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf180045d0 con 0x7fdf20108ad0 2024-04-26T01:40:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.918+0000 7fdf28000700 1 -- 172.21.15.28:0/1470093238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf201a7d70 con 0x7fdf20108ad0 2024-04-26T01:40:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.921+0000 7fdf1e000700 1 -- 172.21.15.28:0/1470093238 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdf18003c00 con 0x7fdf20108ad0 2024-04-26T01:40:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.923+0000 7fdf32a01b80 1 Processor -- start 2024-04-26T01:40:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.923+0000 7fdf1e000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdf080a3cc0 0x7fdf080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.923+0000 7fdf32a01b80 1 -- start start 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.923+0000 7fdf1e000700 1 -- 172.21.15.28:0/1470093238 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fdf1800f020 con 0x7fdf20108ad0 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.923+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255d9fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.923+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da00a0 0x55c255d999b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.923+0000 7fdf24c00700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdf080a3cc0 0x7fdf080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255d9a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c255cebe40 con 0x55c255da00a0 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c255c926e0 con 0x55c255d99ef0 2024-04-26T01:40:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c255ce7e00 con 0x55c255d9f780 2024-04-26T01:40:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf25600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255d9fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf24c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da00a0 0x55c255d999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf25600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255d9fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59314/0 (socket says 172.21.15.28:59314) 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf25600700 1 -- 172.21.15.28:0/1389684578 learned_addr learned my addr 172.21.15.28:0/1389684578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf26000700 1 --2- 172.21.15.28:0/1389684578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255d9a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf25600700 1 -- 172.21.15.28:0/1389684578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 msgr2=0x55c255d9a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf25600700 1 --2- 172.21.15.28:0/1389684578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255d9a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf25600700 1 -- 172.21.15.28:0/1389684578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da00a0 msgr2=0x55c255d999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf25600700 1 --2- 172.21.15.28:0/1389684578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da00a0 0x55c255d999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.924+0000 7fdf25600700 1 -- 172.21.15.28:0/1389684578 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c255c88630 con 0x55c255d9f780 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.925+0000 7fdf24c00700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdf080a3cc0 0x7fdf080a6180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdf200ff100 tx=0x7fdf0c013d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.925+0000 7fdf25600700 1 --2- 172.21.15.28:0/1389684578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255d9fb60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf140115f0 tx=0x7fdf14011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d358143129ce5270 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.925+0000 7fdf12a00700 1 -- 172.21.15.28:0/1389684578 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1401e070 con 0x55c255d9f780 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.925+0000 7fdf12a00700 1 -- 172.21.15.28:0/1389684578 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf14019ae0 con 0x55c255d9f780 2024-04-26T01:40:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.925+0000 7fdf12a00700 1 -- 172.21.15.28:0/1389684578 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1400ea60 con 0x55c255d9f780 2024-04-26T01:40:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1389684578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 msgr2=0x55c255d9fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1389684578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255d9fb60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf140115f0 tx=0x7fdf14011900 comp rx=0 tx=0).stop 2024-04-26T01:40:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1389684578 shutdown_connections 2024-04-26T01:40:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1389684578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255d9a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1389684578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255d9fb60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1389684578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da00a0 0x55c255d999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1389684578 >> 172.21.15.28:0/1389684578 conn(0x55c255cc7f00 msgr2=0x55c255d948a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1389684578 shutdown_connections 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.926+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1389684578 wait complete. 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.927+0000 7fdf32a01b80 1 -- start start 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.927+0000 7fdf32a01b80 1 Processor -- start 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255da5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255da30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da35e0 0x55c255da3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c255da6920 con 0x55c255da35e0 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c255ca5810 con 0x55c255d99ef0 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c255da6aa0 con 0x55c255d9f780 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf32a01b80 10 client.0 _ll_register_callbacks cb 0x55c255c846d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf26000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da35e0 0x55c255da3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf25600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255da5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf24c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255da30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf25600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255da5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35966/0 (socket says 172.21.15.28:35966) 2024-04-26T01:40:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf26000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da35e0 0x55c255da3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55764/0 (socket says 172.21.15.28:55764) 2024-04-26T01:40:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf25600700 1 -- 172.21.15.28:0/3288081305 learned_addr learned my addr 172.21.15.28:0/3288081305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf24c00700 1 -- 172.21.15.28:0/3288081305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 msgr2=0x55c255da5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.928+0000 7fdf24c00700 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255da5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf24c00700 1 -- 172.21.15.28:0/3288081305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da35e0 msgr2=0x55c255da3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf24c00700 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da35e0 0x55c255da3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf24c00700 1 -- 172.21.15.28:0/3288081305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c255c39880 con 0x55c255d9f780 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf25600700 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255da5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf26000700 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da35e0 0x55c255da3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf24c00700 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255da30a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c013510 tx=0x7fdf0c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33663f402decc085 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf07400700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf07400700 1 -- 172.21.15.28:0/3288081305 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c024070 con 0x55c255d9f780 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf32a01b80 1 -- 172.21.15.28:0/3288081305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf14019ae0 con 0x55c255d9f780 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.929+0000 7fdf07400700 1 -- 172.21.15.28:0/3288081305 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fdf0c010c20 con 0x55c255d9f780 2024-04-26T01:40:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf07400700 1 -- 172.21.15.28:0/3288081305 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdf0c010e20 con 0x55c255d9f780 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf32a01b80 10 client.25048 fetch_fsmap learned FSMap version 24 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf32a01b80 1 -- 172.21.15.28:0/3288081305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c255c82aa0 con 0x55c255d9f780 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf07400700 1 -- 172.21.15.28:0/3288081305 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf14019ae0 con 0x55c255d9f780 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf07400700 1 -- 172.21.15.28:0/3288081305 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fdf0c01f070 con 0x55c255d9f780 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf32a01b80 10 client.25048 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf32a01b80 10 client.25048 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf32a01b80 20 client.25048 populate_metadata read hostname 'smithi028' 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c255db3b40 0x55c255db4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf32a01b80 4 client.25048 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.930+0000 7fdf32a01b80 1 -- 172.21.15.28:0/3288081305 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c255cf9630 con 0x55c255db3b40 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.931+0000 7fdf25600700 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c255db3b40 0x55c255db4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.931+0000 7fdf25600700 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c255db3b40 0x55c255db4fb0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed905bf20acb2fc5 server_cookie=4cf9660fdf53062e in_seq=0 out_seq=0 2024-04-26T01:40:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.931+0000 7fdf07400700 10 client.25048 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.932+0000 7fdf07400700 1 -- 172.21.15.28:0/3288081305 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c255cf9630 con 0x55c255db3b40 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.932+0000 7fdf07400700 10 client.25048 handle_command_reply: tid=0 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.932+0000 7fdf32a01b80 1 client.25048 shutdown 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 -- 172.21.15.28:0/3288081305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 msgr2=0x55c255da30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255da30a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c013510 tx=0x7fdf0c0047c0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 -- 172.21.15.28:0/3288081305 shutdown_connections 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255da35e0 0x55c255da3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255da5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c255db3b40 0x55c255db4fb0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/3288081305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255d9f780 0x55c255da30a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf07400700 0 client.25048 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 -- 172.21.15.28:0/3288081305 >> 172.21.15.28:0/3288081305 conn(0x55c255cc7f00 msgr2=0x55c255d93e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 -- 172.21.15.28:0/3288081305 shutdown_connections 2024-04-26T01:40:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 1 -- 172.21.15.28:0/3288081305 wait complete. 2024-04-26T01:40:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.933+0000 7fdf32a01b80 20 client.25048 trim_cache size 0 max 16384 2024-04-26T01:40:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.956+0000 7fdf32a01b80 1 Processor -- start 2024-04-26T01:40:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.956+0000 7fdf32a01b80 1 -- start start 2024-04-26T01:40:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255d999b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255d945d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255dbe410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c255db32f0 con 0x55c255d9f780 2024-04-26T01:40:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c255db39c0 con 0x55c255d99ef0 2024-04-26T01:40:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c255db4090 con 0x55c255da35e0 2024-04-26T01:40:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf24c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255d945d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf25600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255d999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf24c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255d945d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55780/0 (socket says 172.21.15.28:55780) 2024-04-26T01:40:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf24c00700 1 -- 172.21.15.28:0/2913662169 learned_addr learned my addr 172.21.15.28:0/2913662169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.957+0000 7fdf26000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255dbe410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf26000700 1 -- 172.21.15.28:0/2913662169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 msgr2=0x55c255d999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf26000700 1 --2- 172.21.15.28:0/2913662169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255d999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf26000700 1 -- 172.21.15.28:0/2913662169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 msgr2=0x55c255d945d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf26000700 1 --2- 172.21.15.28:0/2913662169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255d945d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf26000700 1 -- 172.21.15.28:0/2913662169 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c255db02e0 con 0x55c255da35e0 2024-04-26T01:40:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf26000700 1 --2- 172.21.15.28:0/2913662169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255dbe410 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdf1809d670 tx=0x7fdf180be040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aad3f089d5799af server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf12000700 1 -- 172.21.15.28:0/2913662169 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1809d970 con 0x55c255da35e0 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf12000700 1 -- 172.21.15.28:0/2913662169 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf180a1d20 con 0x55c255da35e0 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.958+0000 7fdf12000700 1 -- 172.21.15.28:0/2913662169 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1809dbd0 con 0x55c255da35e0 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 -- 172.21.15.28:0/2913662169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 msgr2=0x55c255dbe410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/2913662169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255dbe410 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdf1809d670 tx=0x7fdf180be040 comp rx=0 tx=0).stop 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 -- 172.21.15.28:0/2913662169 shutdown_connections 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/2913662169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255dbe410 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/2913662169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255d999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/2913662169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255d945d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 -- 172.21.15.28:0/2913662169 >> 172.21.15.28:0/2913662169 conn(0x55c255cc7f00 msgr2=0x55c255d9a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 -- 172.21.15.28:0/2913662169 shutdown_connections 2024-04-26T01:40:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.959+0000 7fdf32a01b80 1 -- 172.21.15.28:0/2913662169 wait complete. 2024-04-26T01:40:45.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.960+0000 7fdf32a01b80 1 -- start start 2024-04-26T01:40:45.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.960+0000 7fdf32a01b80 1 Processor -- start 2024-04-26T01:40:45.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.960+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255dbe430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255dbe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf32a01b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255db93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c255db4e30 con 0x55c255d9f780 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c255db4760 con 0x55c255d99ef0 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf32a01b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c255db5500 con 0x55c255da35e0 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf32a01b80 10 client.0 _ll_register_callbacks cb 0x55c255c846d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf24c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255dbe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf24c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255dbe970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55794/0 (socket says 172.21.15.28:55794) 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf24c00700 1 -- 172.21.15.28:0/1472072343 learned_addr learned my addr 172.21.15.28:0/1472072343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf25600700 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255dbe430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf26000700 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255db93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf24c00700 1 -- 172.21.15.28:0/1472072343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 msgr2=0x55c255db93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf24c00700 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255db93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf24c00700 1 -- 172.21.15.28:0/1472072343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 msgr2=0x55c255dbe430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:40:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf24c00700 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255dbe430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.961+0000 7fdf24c00700 1 -- 172.21.15.28:0/1472072343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c255db5f10 con 0x55c255d9f780 2024-04-26T01:40:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.962+0000 7fdf24c00700 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255dbe970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55c255d957d0 tx=0x7fdf0c0052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfcd69402e691d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.962+0000 7fdf07400700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:40:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.962+0000 7fdf07400700 1 -- 172.21.15.28:0/1472072343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c024040 con 0x55c255d9f780 2024-04-26T01:40:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.962+0000 7fdf07400700 1 -- 172.21.15.28:0/1472072343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fdf0c004c40 con 0x55c255d9f780 2024-04-26T01:40:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.962+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1472072343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf180a1d20 con 0x55c255d9f780 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.962+0000 7fdf07400700 1 -- 172.21.15.28:0/1472072343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdf0c009260 con 0x55c255d9f780 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.962+0000 7fdf32a01b80 10 client.14910 fetch_fsmap learned FSMap version 24 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.962+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1472072343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c255e1b390 con 0x55c255d9f780 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf07400700 1 -- 172.21.15.28:0/1472072343 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf180a1d20 con 0x55c255d9f780 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf07400700 1 -- 172.21.15.28:0/1472072343 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fdf0c02ac80 con 0x55c255d9f780 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf32a01b80 10 client.14910 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf32a01b80 10 client.14910 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf32a01b80 20 client.14910 populate_metadata read hostname 'smithi028' 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c255e1feb0 0x55c255e1ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf32a01b80 4 client.14910 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1472072343 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "status"}) -- 0x55c255cf36d0 con 0x55c255e1feb0 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.963+0000 7fdf25600700 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c255e1feb0 0x55c255e1ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.964+0000 7fdf25600700 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c255e1feb0 0x55c255e1ebc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ff957d41606ec71 server_cookie=f705239608b5dae0 in_seq=0 out_seq=0 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.964+0000 7fdf07400700 10 client.14910 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.964+0000 7fdf07400700 1 -- 172.21.15.28:0/1472072343 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c255cf36d0 con 0x55c255e1feb0 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.964+0000 7fdf07400700 10 client.14910 handle_command_reply: tid=0 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.964+0000 7fdf32a01b80 1 client.14910 shutdown 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.964+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1472072343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 msgr2=0x55c255dbe970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.964+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255dbe970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55c255d957d0 tx=0x7fdf0c0052b0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1472072343 shutdown_connections 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c255e1feb0 0x55c255e1ebc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c255da35e0 0x55c255db93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c255d99ef0 0x55c255dbe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 1 --2- 172.21.15.28:0/1472072343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c255d9f780 0x55c255dbe970 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1472072343 >> 172.21.15.28:0/1472072343 conn(0x55c255cc7f00 msgr2=0x55c255da0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf07400700 0 client.14910 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1472072343 shutdown_connections 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 1 -- 172.21.15.28:0/1472072343 wait complete. 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.965+0000 7fdf32a01b80 20 client.14910 trim_cache size 0 max 16384 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-26T01:40:45.965 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24523, 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 228.670933453, 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 263.88122526199999 2024-04-26T01:40:45.966 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 -- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdf080a3cc0 msgr2=0x7fdf080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdf080a3cc0 0x7fdf080a6180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdf200ff100 tx=0x7fdf0c013d30 comp rx=0 tx=0).stop 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 -- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf20108ad0 msgr2=0x7fdf20102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf20108ad0 0x7fdf20102c90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800b070 tx=0x7fdf18004d00 comp rx=0 tx=0).stop 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 -- 172.21.15.28:0/1470093238 shutdown_connections 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fdf080a3cc0 0x7fdf080a6180 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf20108ad0 0x7fdf20102c90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf20101030 0x7fdf20102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 --2- 172.21.15.28:0/1470093238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf201019d0 0x7fdf20102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 -- 172.21.15.28:0/1470093238 >> 172.21.15.28:0/1470093238 conn(0x7fdf20077fb0 msgr2=0x7fdf200ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 -- 172.21.15.28:0/1470093238 shutdown_connections 2024-04-26T01:40:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:45.966+0000 7fdf28000700 1 -- 172.21.15.28:0/1470093238 wait complete. 2024-04-26T01:40:46.931 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T01:40:46.941 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T01:40:46.941 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T01:40:46.941 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T01:40:46.941 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-26T01:40:46.942 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:40:46.942 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:40:46.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 -- 172.21.15.28:0/2348490040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc310107ab0 msgr2=0x7fc310109f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 --2- 172.21.15.28:0/2348490040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc310107ab0 0x7fc310109f00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc308009860 tx=0x7fc30802a340 comp rx=0 tx=0).stop 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 -- 172.21.15.28:0/2348490040 shutdown_connections 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 --2- 172.21.15.28:0/2348490040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc310107ab0 0x7fc310109f00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 --2- 172.21.15.28:0/2348490040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310105180 0x7fc310107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 --2- 172.21.15.28:0/2348490040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc310102850 0x7fc310104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 -- 172.21.15.28:0/2348490040 >> 172.21.15.28:0/2348490040 conn(0x7fc3100fc240 msgr2=0x7fc3100fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 -- 172.21.15.28:0/2348490040 shutdown_connections 2024-04-26T01:40:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.160+0000 7fc316200640 1 -- 172.21.15.28:0/2348490040 wait complete. 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.161+0000 7fc316200640 1 Processor -- start 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.161+0000 7fc316200640 1 -- start start 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc316200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc310102850 0x7fc31019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc316200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc310105180 0x7fc31019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc316200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310107ab0 0x7fc3101a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc316200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010d920 con 0x7fc310102850 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc316200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010d7a0 con 0x7fc310107ab0 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc316200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010daa0 con 0x7fc310105180 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc314e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc310102850 0x7fc31019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc314e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc310102850 0x7fc31019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55802/0 (socket says 172.21.15.28:55802) 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc314e00640 1 -- 172.21.15.28:0/2173138873 learned_addr learned my addr 172.21.15.28:0/2173138873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc30fe00640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc310105180 0x7fc31019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc315800640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310107ab0 0x7fc3101a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc314e00640 1 -- 172.21.15.28:0/2173138873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc310105180 msgr2=0x7fc31019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc314e00640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc310105180 0x7fc31019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc314e00640 1 -- 172.21.15.28:0/2173138873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310107ab0 msgr2=0x7fc3101a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc314e00640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310107ab0 0x7fc3101a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.162+0000 7fc314e00640 1 -- 172.21.15.28:0/2173138873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3101a1dd0 con 0x7fc310102850 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.163+0000 7fc314e00640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc310102850 0x7fc31019a1b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc30400efb0 tx=0x7fc30400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.163+0000 7fc30d600640 1 -- 172.21.15.28:0/2173138873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30401b070 con 0x7fc310102850 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.163+0000 7fc30d600640 1 -- 172.21.15.28:0/2173138873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30400ba20 con 0x7fc310102850 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.163+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3101a20c0 con 0x7fc310102850 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.163+0000 7fc30d600640 1 -- 172.21.15.28:0/2173138873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304016b30 con 0x7fc310102850 2024-04-26T01:40:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.163+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3101a2660 con 0x7fc310102850 2024-04-26T01:40:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.165+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3101095a0 con 0x7fc310102850 2024-04-26T01:40:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.167+0000 7fc30d600640 1 -- 172.21.15.28:0/2173138873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc30401e070 con 0x7fc310102850 2024-04-26T01:40:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.169+0000 7fc30d600640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2f80a3c20 0x7fc2f80a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.169+0000 7fc30d600640 1 -- 172.21.15.28:0/2173138873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fc3040d0300 con 0x7fc310102850 2024-04-26T01:40:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.169+0000 7fc30fe00640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2f80a3c20 0x7fc2f80a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.170+0000 7fc30fe00640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2f80a3c20 0x7fc2f80a60e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc31019b6c0 tx=0x7fc2fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.174+0000 7fc30d600640 1 -- 172.21.15.28:0/2173138873 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc30409c640 con 0x7fc310102850 2024-04-26T01:40:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:47.434+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fc310062b50 con 0x7fc310102850 2024-04-26T01:40:47.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:47 smithi169 ceph-mon[32851]: pgmap v451: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:47.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:47 smithi053 ceph-mon[32593]: pgmap v451: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:47.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:47 smithi028 ceph-mon[28247]: pgmap v451: 97 pgs: 97 active+clean; 645 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T01:40:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.370+0000 7fc30d600640 1 -- 172.21.15.28:0/2173138873 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fc304098030 con 0x7fc310102850 2024-04-26T01:40:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.374+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2f80a3c20 msgr2=0x7fc2f80a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.374+0000 7fc316200640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2f80a3c20 0x7fc2f80a60e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc31019b6c0 tx=0x7fc2fc005f90 comp rx=0 tx=0).stop 2024-04-26T01:40:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.374+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc310102850 msgr2=0x7fc31019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.375+0000 7fc316200640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc310102850 0x7fc31019a1b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc30400efb0 tx=0x7fc30400c800 comp rx=0 tx=0).stop 2024-04-26T01:40:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.375+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 shutdown_connections 2024-04-26T01:40:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.375+0000 7fc316200640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fc2f80a3c20 0x7fc2f80a60e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.375+0000 7fc316200640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310107ab0 0x7fc3101a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.375+0000 7fc316200640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc310105180 0x7fc31019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.375+0000 7fc316200640 1 --2- 172.21.15.28:0/2173138873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc310102850 0x7fc31019a1b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.375+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 >> 172.21.15.28:0/2173138873 conn(0x7fc3100fc240 msgr2=0x7fc310101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.377+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 shutdown_connections 2024-04-26T01:40:48.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.378+0000 7fc316200640 1 -- 172.21.15.28:0/2173138873 wait complete. 2024-04-26T01:40:48.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-26T01:40:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 -- 172.21.15.28:0/3034714629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 msgr2=0x7fa144108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 --2- 172.21.15.28:0/3034714629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 0x7fa144108a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa138009900 tx=0x7fa13802a400 comp rx=0 tx=0).stop 2024-04-26T01:40:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 -- 172.21.15.28:0/3034714629 shutdown_connections 2024-04-26T01:40:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 --2- 172.21.15.28:0/3034714629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14410a190 0x7fa144110a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 --2- 172.21.15.28:0/3034714629 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1441097d0 0x7fa144109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 --2- 172.21.15.28:0/3034714629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 0x7fa144108a00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 -- 172.21.15.28:0/3034714629 >> 172.21.15.28:0/3034714629 conn(0x7fa1440775a0 msgr2=0x7fa1440799c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 -- 172.21.15.28:0/3034714629 shutdown_connections 2024-04-26T01:40:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.595+0000 7fa14ba00640 1 -- 172.21.15.28:0/3034714629 wait complete. 2024-04-26T01:40:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.596+0000 7fa14ba00640 1 Processor -- start 2024-04-26T01:40:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.596+0000 7fa14ba00640 1 -- start start 2024-04-26T01:40:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.596+0000 7fa14ba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 0x7fa14419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.596+0000 7fa14ba00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1441097d0 0x7fa14419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.596+0000 7fa14ba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14410a190 0x7fa1441a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.596+0000 7fa14ba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa144114160 con 0x7fa14410a190 2024-04-26T01:40:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.596+0000 7fa14ba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa144113fe0 con 0x7fa1441085e0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.596+0000 7fa14ba00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa1441142e0 con 0x7fa1441097d0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 0x7fa14419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 0x7fa14419e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36028/0 (socket says 172.21.15.28:36028) 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 -- 172.21.15.28:0/1559692614 learned_addr learned my addr 172.21.15.28:0/1559692614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149a00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14410a190 0x7fa1441a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa143e00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1441097d0 0x7fa14419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 -- 172.21.15.28:0/1559692614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1441097d0 msgr2=0x7fa14419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1441097d0 0x7fa14419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 -- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14410a190 msgr2=0x7fa1441a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14410a190 0x7fa1441a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 -- 172.21.15.28:0/1559692614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1441a6390 con 0x7fa1441085e0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.597+0000 7fa149000640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 0x7fa14419e770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa138031500 tx=0x7fa138003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.598+0000 7fa141600640 1 -- 172.21.15.28:0/1559692614 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13803f070 con 0x7fa1441085e0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.598+0000 7fa141600640 1 -- 172.21.15.28:0/1559692614 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa138031c50 con 0x7fa1441085e0 2024-04-26T01:40:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.598+0000 7fa141600640 1 -- 172.21.15.28:0/1559692614 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138035030 con 0x7fa1441085e0 2024-04-26T01:40:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.598+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1441a6620 con 0x7fa1441085e0 2024-04-26T01:40:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.598+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1441a6b60 con 0x7fa1441085e0 2024-04-26T01:40:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.599+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa14410f8a0 con 0x7fa1441085e0 2024-04-26T01:40:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.601+0000 7fa141600640 1 -- 172.21.15.28:0/1559692614 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fa138042070 con 0x7fa1441085e0 2024-04-26T01:40:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.602+0000 7fa141600640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa12c0a3d20 0x7fa12c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.602+0000 7fa143e00640 1 -- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa12c0a3d20 msgr2=0x7fa12c0a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3431203710 2024-04-26T01:40:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.602+0000 7fa141600640 1 -- 172.21.15.28:0/1559692614 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fa1380f0b10 con 0x7fa1441085e0 2024-04-26T01:40:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.602+0000 7fa143e00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa12c0a3d20 0x7fa12c0a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:40:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.607+0000 7fa141600640 1 -- 172.21.15.28:0/1559692614 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1380bce50 con 0x7fa1441085e0 2024-04-26T01:40:48.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:48 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2173138873' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:40:48.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: ignoring --setuser ceph since I am not root 2024-04-26T01:40:48.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: ignoring --setgroup ceph since I am not root 2024-04-26T01:40:48.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:48 smithi169 ceph-mgr[34730]: -- 172.21.15.169:0/3817718915 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5647d438e5a0 con 0x5647d4362800 2024-04-26T01:40:48.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:48.553+0000 7fbbc0ac4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:40:48.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:48 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2173138873' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:40:48.704 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: ignoring --setuser ceph since I am not root 2024-04-26T01:40:48.704 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: ignoring --setgroup ceph since I am not root 2024-04-26T01:40:48.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:48 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2173138873' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:40:48.718 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: ignoring --setuser ceph since I am not root 2024-04-26T01:40:48.719 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: ignoring --setgroup ceph since I am not root 2024-04-26T01:40:48.719 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:48 smithi053 ceph-mgr[33978]: -- 172.21.15.53:0/1325830427 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d095d85a0 con 0x561d095acc00 2024-04-26T01:40:48.719 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:48.562+0000 7f5f33452200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:40:48.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.803+0000 7fa143e00640 1 -- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa12c0a3d20 msgr2=0x7fa12c0a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3431203710 2024-04-26T01:40:48.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.803+0000 7fa143e00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa12c0a3d20 0x7fa12c0a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:40:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.817+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fa14402d5c0 con 0x7fa1441085e0 2024-04-26T01:40:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.823+0000 7fa141600640 1 -- 172.21.15.28:0/1559692614 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7fa1380c1150 con 0x7fa1441085e0 2024-04-26T01:40:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.826+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa12c0a3d20 msgr2=0x7fa12c0a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:40:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.826+0000 7fa14ba00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa12c0a3d20 0x7fa12c0a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.826+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 msgr2=0x7fa14419e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.826+0000 7fa14ba00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 0x7fa14419e770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa138031500 tx=0x7fa138003c10 comp rx=0 tx=0).stop 2024-04-26T01:40:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.827+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 shutdown_connections 2024-04-26T01:40:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.827+0000 7fa14ba00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa12c0a3d20 0x7fa12c0a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.827+0000 7fa14ba00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14410a190 0x7fa1441a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.827+0000 7fa14ba00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1441097d0 0x7fa14419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.827+0000 7fa14ba00640 1 --2- 172.21.15.28:0/1559692614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1441085e0 0x7fa14419e770 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:48.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.827+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 >> 172.21.15.28:0/1559692614 conn(0x7fa1440775a0 msgr2=0x7fa14410ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:48.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.827+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 shutdown_connections 2024-04-26T01:40:48.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:48.827+0000 7fa14ba00640 1 -- 172.21.15.28:0/1559692614 wait complete. 2024-04-26T01:40:48.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-26T01:40:48.962 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:48.705+0000 7f2f8c443200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:40:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 -- 172.21.15.28:0/2406450119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 msgr2=0x7f032c103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 --2- 172.21.15.28:0/2406450119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 0x7f032c103450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0318009900 tx=0x7f031802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 -- 172.21.15.28:0/2406450119 shutdown_connections 2024-04-26T01:40:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 --2- 172.21.15.28:0/2406450119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c103990 0x7f032c10a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 --2- 172.21.15.28:0/2406450119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 0x7f032c103450 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 --2- 172.21.15.28:0/2406450119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f032c101de0 0x7f032c102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 -- 172.21.15.28:0/2406450119 >> 172.21.15.28:0/2406450119 conn(0x7f032c0fd350 msgr2=0x7f032c0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 -- 172.21.15.28:0/2406450119 shutdown_connections 2024-04-26T01:40:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.033+0000 7f0331800640 1 -- 172.21.15.28:0/2406450119 wait complete. 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.034+0000 7f0331800640 1 Processor -- start 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.034+0000 7f0331800640 1 -- start start 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.034+0000 7f0331800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c101de0 0x7f032c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.034+0000 7f0331800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 0x7f032c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.034+0000 7f0331800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f032c103990 0x7f032c1a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.034+0000 7f0331800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f032c10d960 con 0x7f032c101de0 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.034+0000 7f0331800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f032c10d7e0 con 0x7f032c103990 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.034+0000 7f0331800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f032c10dae0 con 0x7f032c102fd0 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c101de0 0x7f032c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032b400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 0x7f032c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c101de0 0x7f032c19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49152/0 (socket says 172.21.15.28:49152) 2024-04-26T01:40:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032be00640 1 -- 172.21.15.28:0/4251242640 learned_addr learned my addr 172.21.15.28:0/4251242640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f0330e00640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f032c103990 0x7f032c1a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032b400640 1 -- 172.21.15.28:0/4251242640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f032c103990 msgr2=0x7f032c1a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032b400640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f032c103990 0x7f032c1a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032b400640 1 -- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c101de0 msgr2=0x7f032c19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032b400640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c101de0 0x7f032c19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032b400640 1 -- 172.21.15.28:0/4251242640 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c1a1e60 con 0x7f032c102fd0 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.035+0000 7f032b400640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 0x7f032c19a820 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f031802a9e0 tx=0x7f0318003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.036+0000 7f0328c00640 1 -- 172.21.15.28:0/4251242640 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031803f070 con 0x7f032c102fd0 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.036+0000 7f0328c00640 1 -- 172.21.15.28:0/4251242640 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0318031c50 con 0x7f032c102fd0 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.036+0000 7f0328c00640 1 -- 172.21.15.28:0/4251242640 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318035030 con 0x7f032c102fd0 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.036+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032c1a20f0 con 0x7f032c102fd0 2024-04-26T01:40:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.036+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f032c1a2560 con 0x7f032c102fd0 2024-04-26T01:40:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.037+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f032c1a27c0 con 0x7f032c102fd0 2024-04-26T01:40:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.039+0000 7f0328c00640 1 -- 172.21.15.28:0/4251242640 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f0318042070 con 0x7f032c102fd0 2024-04-26T01:40:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.040+0000 7f0328c00640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f03100a3c70 0x7f03100a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.040+0000 7f0328c00640 1 -- 172.21.15.28:0/4251242640 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f03180f0aa0 con 0x7f032c102fd0 2024-04-26T01:40:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.040+0000 7f032be00640 1 -- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f03100a3c70 msgr2=0x7f03100a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3431203710 2024-04-26T01:40:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.040+0000 7f032be00640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f03100a3c70 0x7f03100a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:40:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.044+0000 7f0328c00640 1 -- 172.21.15.28:0/4251242640 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03180bcde0 con 0x7f032c102fd0 2024-04-26T01:40:49.148 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:48.739+0000 7fbbc0ac4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:40:49.148 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:48.841+0000 7fbbc0ac4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:40:49.148 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:49.151+0000 7fbbc0ac4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:40:49.181 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:48.754+0000 7f5f33452200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:40:49.181 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:48.862+0000 7f5f33452200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:40:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.241+0000 7f032be00640 1 -- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f03100a3c70 msgr2=0x7f03100a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3431203710 2024-04-26T01:40:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.241+0000 7f032be00640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f03100a3c70 0x7f03100a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:40:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.246+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f032c1a2990 con 0x7f032c102fd0 2024-04-26T01:40:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.252+0000 7f0328c00640 1 -- 172.21.15.28:0/4251242640 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f03180c10e0 con 0x7f032c102fd0 2024-04-26T01:40:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f03100a3c70 msgr2=0x7f03100a6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:40:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f03100a3c70 0x7f03100a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 msgr2=0x7f032c19a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 0x7f032c19a820 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f031802a9e0 tx=0x7f0318003c10 comp rx=0 tx=0).stop 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 shutdown_connections 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f03100a3c70 0x7f03100a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f032c103990 0x7f032c1a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f032c102fd0 0x7f032c19a820 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 --2- 172.21.15.28:0/4251242640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c101de0 0x7f032c19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 >> 172.21.15.28:0/4251242640 conn(0x7f032c0fd350 msgr2=0x7f032c074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 shutdown_connections 2024-04-26T01:40:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:49.255+0000 7f0331800640 1 -- 172.21.15.28:0/4251242640 wait complete. 2024-04-26T01:40:49.268 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:48.964+0000 7f2f8c443200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:40:49.268 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:49.103+0000 7f2f8c443200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:40:49.271 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T01:40:49.297 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T01:40:49.297 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T01:40:49.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:49.297 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-26T01:40:49.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:49.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-26T01:40:49.410 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:49.236+0000 7fbbc0ac4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:40:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:49 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2173138873' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:40:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:49 smithi169 ceph-mon[32851]: mgrmap e19: x(active, since 12m), standbys: y, z 2024-04-26T01:40:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:49 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' 2024-04-26T01:40:49.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:49 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' 2024-04-26T01:40:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-04-26T01:40:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:40:49.437 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v1', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-04-26T01:40:49.437 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T01:40:49.438 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-26T01:40:49.438 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-26T01:40:49.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T01:40:49.470 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:49.184+0000 7f5f33452200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:40:49.470 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:49.274+0000 7f5f33452200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:40:49.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:49 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2173138873' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:40:49.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:49 smithi053 ceph-mon[32593]: mgrmap e19: x(active, since 12m), standbys: y, z 2024-04-26T01:40:49.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:49 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' 2024-04-26T01:40:49.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:49 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' 2024-04-26T01:40:49.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:49 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2173138873' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:40:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:49 smithi028 ceph-mon[28247]: mgrmap e19: x(active, since 12m), standbys: y, z 2024-04-26T01:40:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:49 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' 2024-04-26T01:40:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:49 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' 2024-04-26T01:40:49.532 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:49.517+0000 7f2f8c443200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:40:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:50.029 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:49.636+0000 7f2f8c443200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:40:50.758 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:50.420+0000 7fbbc0ac4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:40:50.758 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:50.589+0000 7fbbc0ac4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:40:50.880 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:50.527+0000 7f5f33452200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:40:50.880 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:50.706+0000 7f5f33452200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:40:51.047 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:50.760+0000 7fbbc0ac4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:40:51.183 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:50.883+0000 7f5f33452200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:40:51.342 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:51.049+0000 7fbbc0ac4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:40:51.343 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:51.142+0000 7fbbc0ac4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:40:51.468 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:51.186+0000 7f5f33452200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:40:51.468 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:51.283+0000 7f5f33452200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:40:51.526 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:51 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:51.253+0000 7f2f8c443200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:40:51.526 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:51 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:51.491+0000 7f2f8c443200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:40:51.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:51.345+0000 7fbbc0ac4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:40:51.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:51.538+0000 7fbbc0ac4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:40:51.781 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:51 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:51.715+0000 7f2f8c443200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:40:51.968 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:51.496+0000 7f5f33452200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:40:51.968 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:51.698+0000 7f5f33452200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:40:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 -- 172.21.15.28:0/2162453645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 msgr2=0x7fa51c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 --2- 172.21.15.28:0/2162453645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 0x7fa51c101dc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa5080098f0 tx=0x7fa508009c00 comp rx=0 tx=0).stop 2024-04-26T01:40:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 -- 172.21.15.28:0/2162453645 shutdown_connections 2024-04-26T01:40:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 --2- 172.21.15.28:0/2162453645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51c102300 0x7fa51c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 --2- 172.21.15.28:0/2162453645 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c107990 0x7fa51c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 --2- 172.21.15.28:0/2162453645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 0x7fa51c101dc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 -- 172.21.15.28:0/2162453645 >> 172.21.15.28:0/2162453645 conn(0x7fa51c0788f0 msgr2=0x7fa51c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:52.380 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:52.123+0000 7f2f8c443200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:40:52.380 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:52.253+0000 7f2f8c443200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:40:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 -- 172.21.15.28:0/2162453645 shutdown_connections 2024-04-26T01:40:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.382+0000 7fa523800700 1 -- 172.21.15.28:0/2162453645 wait complete. 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.384+0000 7fa523800700 1 Processor -- start 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.384+0000 7fa523800700 1 -- start start 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa523800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 0x7fa51c19b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa523800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51c102300 0x7fa51c19b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa523800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c107990 0x7fa51c19fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa523800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c111b30 con 0x7fa51c101940 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa523800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c1119b0 con 0x7fa51c102300 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa523800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c111cb0 con 0x7fa51c107990 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa51be00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51c102300 0x7fa51c19b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa51be00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51c102300 0x7fa51c19b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50608/0 (socket says 172.21.15.28:50608) 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa51be00700 1 -- 172.21.15.28:0/3925533906 learned_addr learned my addr 172.21.15.28:0/3925533906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa520e00700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 0x7fa51c19b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa521800700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c107990 0x7fa51c19fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa51be00700 1 -- 172.21.15.28:0/3925533906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c107990 msgr2=0x7fa51c19fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa51be00700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c107990 0x7fa51c19fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa51be00700 1 -- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 msgr2=0x7fa51c19b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.385+0000 7fa51be00700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 0x7fa51c19b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.386+0000 7fa51be00700 1 -- 172.21.15.28:0/3925533906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c1a0490 con 0x7fa51c102300 2024-04-26T01:40:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.386+0000 7fa520e00700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 0x7fa51c19b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:40:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.386+0000 7fa51be00700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51c102300 0x7fa51c19b9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa508009230 tx=0x7fa50800fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.386+0000 7fa519600700 1 -- 172.21.15.28:0/3925533906 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa508005220 con 0x7fa51c102300 2024-04-26T01:40:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.386+0000 7fa519600700 1 -- 172.21.15.28:0/3925533906 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa50800b560 con 0x7fa51c102300 2024-04-26T01:40:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.386+0000 7fa519600700 1 -- 172.21.15.28:0/3925533906 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa508003a80 con 0x7fa51c102300 2024-04-26T01:40:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.386+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51c1a0720 con 0x7fa51c102300 2024-04-26T01:40:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.386+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa51c1a8000 con 0x7fa51c102300 2024-04-26T01:40:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.388+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa51c103060 con 0x7fa51c102300 2024-04-26T01:40:52.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.390+0000 7fa519600700 1 -- 172.21.15.28:0/3925533906 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fa50800ba10 con 0x7fa51c102300 2024-04-26T01:40:52.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.391+0000 7fa519600700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa5040a3c90 0x7fa5040a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:52.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.391+0000 7fa520e00700 1 -- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa5040a3c90 msgr2=0x7fa5040a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3431203710 2024-04-26T01:40:52.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.391+0000 7fa519600700 1 -- 172.21.15.28:0/3925533906 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7fa5080d1420 con 0x7fa51c102300 2024-04-26T01:40:52.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.391+0000 7fa520e00700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa5040a3c90 0x7fa5040a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:40:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.395+0000 7fa519600700 1 -- 172.21.15.28:0/3925533906 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa508013030 con 0x7fa51c102300 2024-04-26T01:40:52.410 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:52.137+0000 7fbbc0ac4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:40:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.592+0000 7fa520e00700 1 -- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa5040a3c90 msgr2=0x7fa5040a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3431203710 2024-04-26T01:40:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.592+0000 7fa520e00700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa5040a3c90 0x7fa5040a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:40:52.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.673+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa51c1a81d0 con 0x7fa51c102300 2024-04-26T01:40:52.670 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:52.522+0000 7f2f8c443200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:40:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.674+0000 7fa519600700 1 -- 172.21.15.28:0/3925533906 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+73 (secure 0 0 0) 0x7fa50809de40 con 0x7fa51c102300 2024-04-26T01:40:52.670 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T01:40:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.676+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa5040a3c90 msgr2=0x7fa5040a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:40:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.676+0000 7fa523800700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa5040a3c90 0x7fa5040a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.676+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51c102300 msgr2=0x7fa51c19b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.676+0000 7fa523800700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51c102300 0x7fa51c19b9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa508009230 tx=0x7fa50800fb20 comp rx=0 tx=0).stop 2024-04-26T01:40:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.676+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 shutdown_connections 2024-04-26T01:40:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.677+0000 7fa523800700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7fa5040a3c90 0x7fa5040a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.677+0000 7fa523800700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c107990 0x7fa51c19fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.677+0000 7fa523800700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c101940 0x7fa51c19b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.677+0000 7fa523800700 1 --2- 172.21.15.28:0/3925533906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51c102300 0x7fa51c19b9c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.677+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 >> 172.21.15.28:0/3925533906 conn(0x7fa51c0788f0 msgr2=0x7fa51c06a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.677+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 shutdown_connections 2024-04-26T01:40:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:52.677+0000 7fa523800700 1 -- 172.21.15.28:0/3925533906 wait complete. 2024-04-26T01:40:52.718 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:52.329+0000 7f5f33452200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:40:52.853 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:52 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3925533906' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:40:52.853 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:52.557+0000 7fbbc0ac4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:40:52.853 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:52.653+0000 7fbbc0ac4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:40:53.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:52.798+0000 7f2f8c443200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:40:53.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:52 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3925533906' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:40:53.080 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:52 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3925533906' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:40:53.081 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:52.771+0000 7f5f33452200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:40:53.081 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:52.872+0000 7f5f33452200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:40:53.160 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:52.855+0000 7fbbc0ac4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:40:53.468 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:53.084+0000 7f5f33452200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:40:53.551 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T01:40:53.551 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T01:40:53.551 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-26T01:40:53.551 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T01:40:53.552 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-26T01:40:53.552 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T01:40:53.552 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T01:40:53.552 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T01:40:53.552 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-26T01:40:53.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:53.552 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-26T01:40:53.607 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:40:53.608 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41814sec preferred_lft 41814sec 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-26T01:40:53.609 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-26T01:40:53.610 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T01:40:53.611 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:40:53.611 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-26T01:40:53.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-26T01:40:53.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-26T01:40:53.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-26T01:40:53.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T01:40:53.611 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-26T01:40:53.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:53.382+0000 7fbbc0ac4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:40:53.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:53.475+0000 7fbbc0ac4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:40:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:40:53.735 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:53 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:53.641+0000 7f2f8c443200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:40:53.968 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:53.633+0000 7f5f33452200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:40:53.968 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:53.730+0000 7f5f33452200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:40:54.062 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:53.719+0000 7fbbc0ac4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:40:54.062 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:53.852+0000 7fbbc0ac4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:40:54.063 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:53.948+0000 7fbbc0ac4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:40:54.063 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:54.064+0000 7fbbc0ac4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:40:54.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:54.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:54.114 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T01:40:54.178 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-26T01:40:54.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:54.182 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-26T01:40:54.231 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-26T01:40:54.231 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T01:40:54.231 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T01:40:54.231 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T01:40:54.231 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T01:40:54.231 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:40:54.232 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-26T01:40:54.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T01:40:54.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T01:40:54.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T01:40:54.232 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-26T01:40:54.281 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:54.239+0000 7f2f8c443200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:40:54.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:40:54.343 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:53.985+0000 7f5f33452200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:40:54.343 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:54.125+0000 7f5f33452200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:40:54.344 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:54.224+0000 7f5f33452200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:40:54.344 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:54.347+0000 7f5f33452200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:40:54.597 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:54.386+0000 7f2f8c443200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:40:54.660 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:54.372+0000 7fbbc0ac4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:40:54.955 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:54.678+0000 7f2f8c443200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:40:54.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:54.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:54.968 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-26T01:40:54.968 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:54.660+0000 7f5f33452200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:40:55.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:55.017 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-26T01:40:55.062 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:40:55.062 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-26T01:40:55.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-26T01:40:55.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-26T01:40:55.062 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-26T01:40:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:40:55.258 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:54.974+0000 7fbbc0ac4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:40:55.258 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:40:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z[34675]: 2024-04-26T01:40:55.189+0000 7fbbc0ac4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:40:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:55.285 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-26T01:40:55.286 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:40:55.286 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-26T01:40:55.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-26T01:40:55.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T01:40:55.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-26T01:40:55.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-26T01:40:55.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-26T01:40:55.286 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-26T01:40:55.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:40:55.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-mon[28247]: Standby manager daemon z restarted 2024-04-26T01:40:55.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-mon[28247]: Standby manager daemon z started 2024-04-26T01:40:55.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:40:55.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:40:55.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:40:55.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:40:55.498 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:55.371+0000 7f2f8c443200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:40:55.521 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:55.298+0000 7f5f33452200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:40:55.521 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:55 smithi053 ceph-mon[32593]: Standby manager daemon z restarted 2024-04-26T01:40:55.521 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:55 smithi053 ceph-mon[32593]: Standby manager daemon z started 2024-04-26T01:40:55.521 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:55 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:40:55.521 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:55 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:40:55.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:55 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:40:55.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:55 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:40:55.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:55.582 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:40:55.582 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-26T01:40:55.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-26T01:40:55.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T01:40:55.582 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-26T01:40:55.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:55 smithi169 ceph-mon[32851]: Standby manager daemon z restarted 2024-04-26T01:40:55.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:55 smithi169 ceph-mon[32851]: Standby manager daemon z started 2024-04-26T01:40:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:55 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:40:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:55 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:40:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:55 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:40:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:55 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.169:0/1185049567' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:40:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:40:55.754 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:55.518+0000 7f2f8c443200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:40:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:55.779 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T01:40:55.779 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /tmp/cephfs 2024-04-26T01:40:55.822 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-26T01:40:55.822 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T01:40:55.822 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /tmp/cephfs 2024-04-26T01:40:55.913 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T01:40:55.913 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T01:40:55.913 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/57e9c16f-4990-4d2e-88b0-536ce8dec643 2024-04-26T01:40:55.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:55.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/57e9c16f-4990-4d2e-88b0-536ce8dec643 /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context=system_u:object_r:postgresql_db_t:s0 2024-04-26T01:40:55.968 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:40:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y[33974]: 2024-04-26T01:40:55.524+0000 7f5f33452200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:40:56.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:55.885+0000 7f2f8c443200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:40:56.046 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0" 2024-04-26T01:40:56.046 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-26T01:40:56.046 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:56.078+0000 7f2f8c443200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:56.214+0000 7f2f8c443200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-mon[28247]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-mon[28247]: Standby manager daemon y restarted 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-mon[28247]: Standby manager daemon y started 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:40:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-mon[28247]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:40:56.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:56 smithi169 ceph-mon[32851]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:40:56.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:56 smithi169 ceph-mon[32851]: Standby manager daemon y restarted 2024-04-26T01:40:56.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:56 smithi169 ceph-mon[32851]: Standby manager daemon y started 2024-04-26T01:40:56.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:56 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:40:56.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:56 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:40:56.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:56 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:40:56.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:56 smithi169 ceph-mon[32851]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:40:56.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:56 smithi053 ceph-mon[32593]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:40:56.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:56 smithi053 ceph-mon[32593]: Standby manager daemon y restarted 2024-04-26T01:40:56.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:56 smithi053 ceph-mon[32593]: Standby manager daemon y started 2024-04-26T01:40:56.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:56 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:40:56.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:56 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:40:56.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:56 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:40:56.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:56 smithi053 ceph-mon[32593]: from='mgr.? 172.21.15.53:0/2986803954' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:40:56.735 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:56.383+0000 7f2f8c443200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:40:56.759 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0" 2024-04-26T01:40:56.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-26T01:40:56.759 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-04-26T01:40:56.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.53:3300,172.21.15.169:3300" 2024-04-26T01:40:56.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:3300,172.21.15.53:3300,172.21.15.169:3300:/volumes/qa/sv_1/57e9c16f-4990-4d2e-88b0-536ce8dec643 2024-04-26T01:40:56.759 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",key=0,fsid=ead407d4-036b-11ef-bc93-c7b262605968" will pass to kernel 2024-04-26T01:40:56.767 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T01:40:56.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:40:56.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /tmp/cephfs 2024-04-26T01:40:56.841 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:40:56.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:40:56.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:56.957 INFO:teuthology.orchestra.run.smithi028.stdout:client25055 2024-04-26T01:40:56.958 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:40:56.958 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T01:40:57.007 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:40:57.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:56.768+0000 7f2f8c443200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:40:57.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:40:57.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:57.119 INFO:teuthology.orchestra.run.smithi028.stdout:client25055 2024-04-26T01:40:57.120 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:40:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:40:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:client25055 2024-04-26T01:40:57.280 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/ead407d4-036b-11ef-bc93-c7b262605968.client25055/status') 2024-04-26T01:40:57.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:40:57.409 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.25055 (3)172.21.15.28:0/1714588308 2024-04-26T01:40:57.409 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-04-26T01:40:57.409 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-04-26T01:40:57.409 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-04-26T01:40:57.409 INFO:teuthology.orchestra.run.smithi028.stderr:71 bytes copied, 9.1159e-05 s, 779 kB/s 2024-04-26T01:40:57.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:40:57.420 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T01:40:57.431 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:40:57.431 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:40:57.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-26T01:40:57.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:57 smithi028 ceph-mon[28247]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:40:57.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.622+0000 7f748d000640 1 -- 172.21.15.28:0/160952205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488103920 msgr2=0x7f748810a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.622+0000 7f748d000640 1 --2- 172.21.15.28:0/160952205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488103920 0x7f748810a240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7480009be0 tx=0x7f748002a6c0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.622+0000 7f748d000640 1 -- 172.21.15.28:0/160952205 shutdown_connections 2024-04-26T01:40:57.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.622+0000 7f748d000640 1 --2- 172.21.15.28:0/160952205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488103920 0x7f748810a240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.622+0000 7f748d000640 1 --2- 172.21.15.28:0/160952205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7488102f60 0x7f74881033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.622+0000 7f748d000640 1 --2- 172.21.15.28:0/160952205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7488101e10 0x7f7488102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.622+0000 7f748d000640 1 -- 172.21.15.28:0/160952205 >> 172.21.15.28:0/160952205 conn(0x7f74880fd380 msgr2=0x7f74880ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:57.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.623+0000 7f748d000640 1 -- 172.21.15.28:0/160952205 shutdown_connections 2024-04-26T01:40:57.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.623+0000 7f748d000640 1 -- 172.21.15.28:0/160952205 wait complete. 2024-04-26T01:40:57.620 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:57 smithi053 ceph-mon[32593]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:40:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f748d000640 1 Processor -- start 2024-04-26T01:40:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f748d000640 1 -- start start 2024-04-26T01:40:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f748d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7488101e10 0x7f7488075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f748d000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488102f60 0x7f7488074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f748d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7488103920 0x7f7488074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f748d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f748810d970 con 0x7f7488103920 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f748d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f748810d7f0 con 0x7f7488101e10 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f748d000640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f748810daf0 con 0x7f7488102f60 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f747ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488102f60 0x7f7488074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f7487400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7488101e10 0x7f7488075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f7487e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7488103920 0x7f7488074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f747ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488102f60 0x7f7488074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:47560/0 (socket says 172.21.15.28:47560) 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.624+0000 7f747ea00640 1 -- 172.21.15.28:0/2493170148 learned_addr learned my addr 172.21.15.28:0/2493170148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.625+0000 7f747ea00640 1 -- 172.21.15.28:0/2493170148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7488101e10 msgr2=0x7f7488075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.625+0000 7f747ea00640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7488101e10 0x7f7488075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.625+0000 7f747ea00640 1 -- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7488103920 msgr2=0x7f7488074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.625+0000 7f747ea00640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7488103920 0x7f7488074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.625+0000 7f747ea00640 1 -- 172.21.15.28:0/2493170148 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7488075110 con 0x7f7488102f60 2024-04-26T01:40:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.625+0000 7f7487e00640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7488103920 0x7f7488074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.625+0000 7f747ea00640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488102f60 0x7f7488074340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f747000e9c0 tx=0x7f747000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.626+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747000cd80 con 0x7f7488102f60 2024-04-26T01:40:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.626+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74700035f0 con 0x7f7488102f60 2024-04-26T01:40:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.626+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74881a2110 con 0x7f7488102f60 2024-04-26T01:40:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.626+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470010690 con 0x7f7488102f60 2024-04-26T01:40:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.626+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74881a2610 con 0x7f7488102f60 2024-04-26T01:40:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.627+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7488109120 con 0x7f7488102f60 2024-04-26T01:40:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.629+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 21) ==== 138663+0+0 (secure 0 0 0) 0x7f7470004000 con 0x7f7488102f60 2024-04-26T01:40:57.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.631+0000 7f7484c00640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f74680a3bf0 0x7f74680a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.631+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8211+0+0 (secure 0 0 0) 0x7f74700d0a30 con 0x7f7488102f60 2024-04-26T01:40:57.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.631+0000 7f7487400640 1 -- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f74680a3bf0 msgr2=0x7f74680a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3431203710 2024-04-26T01:40:57.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.631+0000 7f7487400640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f74680a3bf0 0x7f74680a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:40:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.635+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f747009d3c0 con 0x7f7488102f60 2024-04-26T01:40:57.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:57 smithi169 ceph-mon[32851]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:40:57.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.832+0000 7f7487400640 1 -- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f74680a3bf0 msgr2=0x7f74680a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3431203710 2024-04-26T01:40:57.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.832+0000 7f7487400640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f74680a3bf0 0x7f74680a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:40:57.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.898+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f748802d5c0 con 0x7f7488102f60 2024-04-26T01:40:57.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:57 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:57.590+0000 7f2f8c443200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.945+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f7470099330 con 0x7f7488102f60 2024-04-26T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.946+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f74680a3bf0 msgr2=0x7f74680a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:57.946+0000 7f7484c00640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f74680a3bf0 0x7f74680a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.244 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:57 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:57.922+0000 7f2f8c443200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:40:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.293+0000 7f7484c00640 1 -- 172.21.15.28:0/2493170148 <== mon.1 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7f747008ddd0 con 0x7f7488102f60 2024-04-26T01:40:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488102f60 msgr2=0x7f7488074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488102f60 0x7f7488074340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f747000e9c0 tx=0x7f747000ee80 comp rx=0 tx=0).stop 2024-04-26T01:40:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 shutdown_connections 2024-04-26T01:40:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:6800/3431203710,v1:172.21.15.28:6801/3431203710] conn(0x7f74680a3bf0 0x7f74680a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7488103920 0x7f7488074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7488102f60 0x7f7488074340 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 --2- 172.21.15.28:0/2493170148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7488101e10 0x7f7488075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 >> 172.21.15.28:0/2493170148 conn(0x7f74880fd380 msgr2=0x7f74881062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.296+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 shutdown_connections 2024-04-26T01:40:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.297+0000 7f748d000640 1 -- 172.21.15.28:0/2493170148 wait complete. 2024-04-26T01:40:58.311 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-26T01:40:58.323 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T01:40:58.337 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:40:58.337 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:40:58.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-26T01:40:58.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2493170148' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: Active manager daemon x restarted 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: Activating manager daemon x 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: mgrmap e22: x(active, starting, since 0.0156734s), standbys: y, z 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:40:58.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:40:58.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:58.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:40:58.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:40:58.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:40:58.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: Manager daemon x is now available 2024-04-26T01:40:58.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:58.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:40:58.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:58 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2493170148' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: Active manager daemon x restarted 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: Activating manager daemon x 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: mgrmap e22: x(active, starting, since 0.0156734s), standbys: y, z 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:40:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:40:58.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:40:58.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:40:58.472 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:40:58.472 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: Manager daemon x is now available 2024-04-26T01:40:58.472 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:58.472 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:40:58.472 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:58 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:40:58.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2493170148' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:40:58.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:40:58.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: Active manager daemon x restarted 2024-04-26T01:40:58.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: Activating manager daemon x 2024-04-26T01:40:58.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:40:58.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: mgrmap e22: x(active, starting, since 0.0156734s), standbys: y, z 2024-04-26T01:40:58.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:40:58.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:40:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.248+0000 7f2ede200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.248+0000 7f2ede200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.248+0000 7f2ede200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.248+0000 7f2ede200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.248+0000 7f2ede200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.250+0000 7f2ee2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.250+0000 7f2ee2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.250+0000 7f2ee2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.250+0000 7f2ee2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:40:58.250+0000 7f2ee2800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: Manager daemon x is now available 2024-04-26T01:40:58.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:58.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:40:58.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:58 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.536+0000 7ff697c00640 1 -- 172.21.15.28:0/2710153128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901053d0 msgr2=0x7ff690105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.536+0000 7ff697c00640 1 --2- 172.21.15.28:0/2710153128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901053d0 0x7ff690105850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009900 tx=0x7ff67c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.537+0000 7ff697c00640 1 -- 172.21.15.28:0/2710153128 shutdown_connections 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.537+0000 7ff697c00640 1 --2- 172.21.15.28:0/2710153128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff690105d90 0x7ff69010c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.537+0000 7ff697c00640 1 --2- 172.21.15.28:0/2710153128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901053d0 0x7ff690105850 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.537+0000 7ff697c00640 1 --2- 172.21.15.28:0/2710153128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901041e0 0x7ff690104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.537+0000 7ff697c00640 1 -- 172.21.15.28:0/2710153128 >> 172.21.15.28:0/2710153128 conn(0x7ff6900ff770 msgr2=0x7ff690101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.537+0000 7ff697c00640 1 -- 172.21.15.28:0/2710153128 shutdown_connections 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.537+0000 7ff697c00640 1 -- 172.21.15.28:0/2710153128 wait complete. 2024-04-26T01:40:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.538+0000 7ff697c00640 1 Processor -- start 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.538+0000 7ff697c00640 1 -- start start 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff697c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901041e0 0x7ff69019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff697c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901053d0 0x7ff69019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff697c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff690105d90 0x7ff6901a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff697c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff690076da0 con 0x7ff6901053d0 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff697c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff690076c20 con 0x7ff690105d90 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff697c00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff690076f20 con 0x7ff6901041e0 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901041e0 0x7ff69019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:58.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff68fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901053d0 0x7ff69019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901041e0 0x7ff69019e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:47708/0 (socket says 172.21.15.28:47708) 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695200640 1 -- 172.21.15.28:0/1936692272 learned_addr learned my addr 172.21.15.28:0/1936692272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695c00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff690105d90 0x7ff6901a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695200640 1 -- 172.21.15.28:0/1936692272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff690105d90 msgr2=0x7ff6901a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695200640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff690105d90 0x7ff6901a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695200640 1 -- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901053d0 msgr2=0x7ff69019ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695200640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901053d0 0x7ff69019ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.539+0000 7ff695200640 1 -- 172.21.15.28:0/1936692272 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6901a6340 con 0x7ff6901041e0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.540+0000 7ff695200640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901041e0 0x7ff69019e720 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff68400e9c0 tx=0x7ff68400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.540+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68400cd80 con 0x7ff6901041e0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.540+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6840035f0 con 0x7ff6901041e0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.540+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684010690 con 0x7ff6901041e0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.540+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6901a6630 con 0x7ff6901041e0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.540+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6901a6b50 con 0x7ff6901041e0 2024-04-26T01:40:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.541+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6901a6e10 con 0x7ff6901041e0 2024-04-26T01:40:58.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.545+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7ff6840037a0 con 0x7ff6901041e0 2024-04-26T01:40:58.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.546+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7ff6840d05e0 con 0x7ff6901041e0 2024-04-26T01:40:58.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.547+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff68409d2d0 con 0x7ff6901041e0 2024-04-26T01:40:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.954+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7ff684098920 con 0x7ff6901041e0 2024-04-26T01:40:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.955+0000 7ff68d600640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6780aa9a0 0x7ff6780acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.955+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff684016ec0 con 0x7ff6780aa9a0 2024-04-26T01:40:58.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.965+0000 7ff68fe00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6780aa9a0 0x7ff6780acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:58.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:58.972+0000 7ff68fe00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6780aa9a0 0x7ff6780acd90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff67c000c00 tx=0x7ff67c0314a0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.058+0000 7ff68d600640 1 -- 172.21.15.28:0/1936692272 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff684016ec0 con 0x7ff6780aa9a0 2024-04-26T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6780aa9a0 msgr2=0x7ff6780acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6780aa9a0 0x7ff6780acd90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff67c000c00 tx=0x7ff67c0314a0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901041e0 msgr2=0x7ff69019e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901041e0 0x7ff69019e720 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff68400e9c0 tx=0x7ff68400ee80 comp rx=0 tx=0).stop 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 shutdown_connections 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6780aa9a0 0x7ff6780acd90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff690105d90 0x7ff6901a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901053d0 0x7ff69019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 --2- 172.21.15.28:0/1936692272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6901041e0 0x7ff69019e720 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.060+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 >> 172.21.15.28:0/1936692272 conn(0x7ff6900ff770 msgr2=0x7ff6900797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.061+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 shutdown_connections 2024-04-26T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.061+0000 7ff697c00640 1 -- 172.21.15.28:0/1936692272 wait complete. 2024-04-26T01:40:59.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-26T01:40:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.261+0000 7fc5e8a00640 1 -- 172.21.15.28:0/2573358405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e41051a0 msgr2=0x7fc5e4107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.261+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/2573358405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e41051a0 0x7fc5e4107590 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009900 tx=0x7fc5cc02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.262+0000 7fc5e8a00640 1 -- 172.21.15.28:0/2573358405 shutdown_connections 2024-04-26T01:40:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.262+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/2573358405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e4107ad0 0x7fc5e4109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.262+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/2573358405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e41051a0 0x7fc5e4107590 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.262+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/2573358405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5e4102870 0x7fc5e4104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.262+0000 7fc5e8a00640 1 -- 172.21.15.28:0/2573358405 >> 172.21.15.28:0/2573358405 conn(0x7fc5e40fc280 msgr2=0x7fc5e40fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.262+0000 7fc5e8a00640 1 -- 172.21.15.28:0/2573358405 shutdown_connections 2024-04-26T01:40:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.262+0000 7fc5e8a00640 1 -- 172.21.15.28:0/2573358405 wait complete. 2024-04-26T01:40:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e8a00640 1 Processor -- start 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e8a00640 1 -- start start 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e8a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e4102870 0x7fc5e4195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e41051a0 0x7fc5e4196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5e4107ad0 0x7fc5e419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e410b790 con 0x7fc5e4107ad0 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e410b610 con 0x7fc5e41051a0 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e8a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e410b910 con 0x7fc5e4102870 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e3400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e4102870 0x7fc5e4195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e3400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e4102870 0x7fc5e4195e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:49906/0 (socket says 172.21.15.28:49906) 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e3400640 1 -- 172.21.15.28:0/510460395 learned_addr learned my addr 172.21.15.28:0/510460395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5dbe00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e41051a0 0x7fc5e4196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.263+0000 7fc5e3e00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5e4107ad0 0x7fc5e419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5dbe00640 1 -- 172.21.15.28:0/510460395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e4102870 msgr2=0x7fc5e4195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5dbe00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e4102870 0x7fc5e4195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5dbe00640 1 -- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5e4107ad0 msgr2=0x7fc5e419d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5dbe00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5e4107ad0 0x7fc5e419d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5dbe00640 1 -- 172.21.15.28:0/510460395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e419da40 con 0x7fc5e41051a0 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5dbe00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e41051a0 0x7fc5e4196360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc02a910 tx=0x7fc5cc003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5e0c00640 1 -- 172.21.15.28:0/510460395 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc03f070 con 0x7fc5e41051a0 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5e0c00640 1 -- 172.21.15.28:0/510460395 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5cc031c50 con 0x7fc5e41051a0 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.264+0000 7fc5e0c00640 1 -- 172.21.15.28:0/510460395 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc035030 con 0x7fc5e41051a0 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.265+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e419dcd0 con 0x7fc5e41051a0 2024-04-26T01:40:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.265+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e419e140 con 0x7fc5e41051a0 2024-04-26T01:40:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.266+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e40fdd70 con 0x7fc5e41051a0 2024-04-26T01:40:59.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.268+0000 7fc5e0c00640 1 -- 172.21.15.28:0/510460395 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7fc5cc042070 con 0x7fc5e41051a0 2024-04-26T01:40:59.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.269+0000 7fc5e0c00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc5c80a7da0 0x7fc5c80aa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:59.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.269+0000 7fc5e0c00640 1 -- 172.21.15.28:0/510460395 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fc5cc0f0740 con 0x7fc5e41051a0 2024-04-26T01:40:59.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.270+0000 7fc5e3400640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc5c80a7da0 0x7fc5c80aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:59.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.270+0000 7fc5e3400640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc5c80a7da0 0x7fc5c80aa260 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4009c90 tx=0x7fc5d4005f90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:59.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.273+0000 7fc5e0c00640 1 -- 172.21.15.28:0/510460395 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5cc0bcc90 con 0x7fc5e41051a0 2024-04-26T01:40:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.519+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fc5e4062430 con 0x7fc5c80a7da0 2024-04-26T01:40:59.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:59 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:40:59.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:59 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:40:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:40:59 smithi028 ceph-mon[28247]: mgrmap e23: x(active, since 1.03041s), standbys: y, z 2024-04-26T01:40:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.635+0000 7fc5e0c00640 1 -- 172.21.15.28:0/510460395 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fc5e4062430 con 0x7fc5c80a7da0 2024-04-26T01:40:59.632 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-04-26T01:40:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc5c80a7da0 msgr2=0x7fc5c80aa260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc5c80a7da0 0x7fc5c80aa260 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4009c90 tx=0x7fc5d4005f90 comp rx=0 tx=0).stop 2024-04-26T01:40:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e41051a0 msgr2=0x7fc5e4196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e41051a0 0x7fc5e4196360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc02a910 tx=0x7fc5cc003c10 comp rx=0 tx=0).stop 2024-04-26T01:40:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 shutdown_connections 2024-04-26T01:40:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc5c80a7da0 0x7fc5c80aa260 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5e4107ad0 0x7fc5e419d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e41051a0 0x7fc5e4196360 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 --2- 172.21.15.28:0/510460395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e4102870 0x7fc5e4195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 >> 172.21.15.28:0/510460395 conn(0x7fc5e40fc280 msgr2=0x7fc5e4103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 shutdown_connections 2024-04-26T01:40:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.638+0000 7fc5e8a00640 1 -- 172.21.15.28:0/510460395 wait complete. 2024-04-26T01:40:59.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T01:40:59.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:59 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:40:59.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:59 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:40:59.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:40:59 smithi169 ceph-mon[32851]: mgrmap e23: x(active, since 1.03041s), standbys: y, z 2024-04-26T01:40:59.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:59 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:40:59.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:59 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:40:59.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:40:59 smithi053 ceph-mon[32593]: mgrmap e23: x(active, since 1.03041s), standbys: y, z 2024-04-26T01:40:59.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 -- 172.21.15.28:0/669492332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b60105200 msgr2=0x7f9b60105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 --2- 172.21.15.28:0/669492332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b60105200 0x7f9b60105680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009900 tx=0x7f9b4c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 -- 172.21.15.28:0/669492332 shutdown_connections 2024-04-26T01:40:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 --2- 172.21.15.28:0/669492332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b60105bc0 0x7f9b6010c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 --2- 172.21.15.28:0/669492332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b60105200 0x7f9b60105680 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 --2- 172.21.15.28:0/669492332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b60104010 0x7f9b60104430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 -- 172.21.15.28:0/669492332 >> 172.21.15.28:0/669492332 conn(0x7f9b600ff5c0 msgr2=0x7f9b601019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 -- 172.21.15.28:0/669492332 shutdown_connections 2024-04-26T01:40:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.843+0000 7f9b66800640 1 -- 172.21.15.28:0/669492332 wait complete. 2024-04-26T01:40:59.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b66800640 1 Processor -- start 2024-04-26T01:40:59.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b66800640 1 -- start start 2024-04-26T01:40:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b66800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b60104010 0x7f9b6019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b66800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b60105200 0x7f9b6019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b66800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b60105bc0 0x7f9b601a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b66800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6010fba0 con 0x7f9b60105200 2024-04-26T01:40:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b66800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6010fa20 con 0x7f9b60105bc0 2024-04-26T01:40:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b66800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6010fd20 con 0x7f9b60104010 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.845+0000 7f9b65400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b60104010 0x7f9b6019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b60104010 0x7f9b6019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:49926/0 (socket says 172.21.15.28:49926) 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65400640 1 -- 172.21.15.28:0/3638550752 learned_addr learned my addr 172.21.15.28:0/3638550752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b64a00640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b60105200 0x7f9b6019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65e00640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b60105bc0 0x7f9b601a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65400640 1 -- 172.21.15.28:0/3638550752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b60105bc0 msgr2=0x7f9b601a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65400640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b60105bc0 0x7f9b601a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65400640 1 -- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b60105200 msgr2=0x7f9b6019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65400640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b60105200 0x7f9b6019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65400640 1 -- 172.21.15.28:0/3638550752 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b601a1e10 con 0x7f9b60104010 2024-04-26T01:40:59.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.846+0000 7f9b65400640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b60104010 0x7f9b6019a1f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400e9c0 tx=0x7f9b5400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:59.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.847+0000 7f9b5e000640 1 -- 172.21.15.28:0/3638550752 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400cde0 con 0x7f9b60104010 2024-04-26T01:40:59.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.847+0000 7f9b5e000640 1 -- 172.21.15.28:0/3638550752 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b540108e0 con 0x7f9b60104010 2024-04-26T01:40:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.847+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b601a2100 con 0x7f9b60104010 2024-04-26T01:40:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.847+0000 7f9b5e000640 1 -- 172.21.15.28:0/3638550752 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54018960 con 0x7f9b60104010 2024-04-26T01:40:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.847+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b601a2620 con 0x7f9b60104010 2024-04-26T01:40:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.848+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6010b330 con 0x7f9b60104010 2024-04-26T01:40:59.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.850+0000 7f9b5e000640 1 -- 172.21.15.28:0/3638550752 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f9b5401f020 con 0x7f9b60104010 2024-04-26T01:40:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.851+0000 7f9b5e000640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9b480a3990 0x7f9b480a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.851+0000 7f9b5e000640 1 -- 172.21.15.28:0/3638550752 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f9b540d0660 con 0x7f9b60104010 2024-04-26T01:40:59.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.852+0000 7f9b64a00640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9b480a3990 0x7f9b480a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:59.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.852+0000 7f9b64a00640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9b480a3990 0x7f9b480a5e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c031500 tx=0x7f9b4c005a20 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:40:59.855+0000 7f9b5e000640 1 -- 172.21.15.28:0/3638550752 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b5409d200 con 0x7f9b60104010 2024-04-26T01:41:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.095+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f9b60062390 con 0x7f9b480a3990 2024-04-26T01:41:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.114+0000 7f9b5e000640 1 -- 172.21.15.28:0/3638550752 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f9b60062390 con 0x7f9b480a3990 2024-04-26T01:41:00.111 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T01:40:58", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-26T01:41:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9b480a3990 msgr2=0x7f9b480a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9b480a3990 0x7f9b480a5e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c031500 tx=0x7f9b4c005a20 comp rx=0 tx=0).stop 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b60104010 msgr2=0x7f9b6019a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b60104010 0x7f9b6019a1f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400e9c0 tx=0x7f9b5400ee80 comp rx=0 tx=0).stop 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 shutdown_connections 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9b480a3990 0x7f9b480a5e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b60105bc0 0x7f9b601a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b60105200 0x7f9b6019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 --2- 172.21.15.28:0/3638550752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b60104010 0x7f9b6019a1f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 >> 172.21.15.28:0/3638550752 conn(0x7f9b600ff5c0 msgr2=0x7f9b60108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 shutdown_connections 2024-04-26T01:41:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.116+0000 7f9b66800640 1 -- 172.21.15.28:0/3638550752 wait complete. 2024-04-26T01:41:00.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.349+0000 7f5d1ee00640 1 -- 172.21.15.28:0/3634047614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d181053d0 msgr2=0x7f5d18105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.349+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/3634047614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d181053d0 0x7f5d18105850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5d04009900 tx=0x7f5d0402a400 comp rx=0 tx=0).stop 2024-04-26T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.349+0000 7f5d1ee00640 1 -- 172.21.15.28:0/3634047614 shutdown_connections 2024-04-26T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.349+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/3634047614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d18105d90 0x7f5d1810c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.349+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/3634047614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d181053d0 0x7f5d18105850 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.349+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/3634047614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d181041e0 0x7f5d18104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.350+0000 7f5d1ee00640 1 -- 172.21.15.28:0/3634047614 >> 172.21.15.28:0/3634047614 conn(0x7f5d180ff770 msgr2=0x7f5d18101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.350+0000 7f5d1ee00640 1 -- 172.21.15.28:0/3634047614 shutdown_connections 2024-04-26T01:41:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.350+0000 7f5d1ee00640 1 -- 172.21.15.28:0/3634047614 wait complete. 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ee00640 1 Processor -- start 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ee00640 1 -- start start 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ee00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d181041e0 0x7f5d1819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d181053d0 0x7f5d1819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d18105d90 0x7f5d181a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1810fd70 con 0x7f5d181053d0 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1810fbf0 con 0x7f5d18105d90 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ee00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1810fef0 con 0x7f5d181041e0 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d17e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d181041e0 0x7f5d1819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d17e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d181041e0 0x7f5d1819a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:49966/0 (socket says 172.21.15.28:49966) 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d17e00640 1 -- 172.21.15.28:0/389334233 learned_addr learned my addr 172.21.15.28:0/389334233 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.351+0000 7f5d1ce00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d18105d90 0x7f5d181a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.352+0000 7f5d17400640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d181053d0 0x7f5d1819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.352+0000 7f5d17e00640 1 -- 172.21.15.28:0/389334233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d18105d90 msgr2=0x7f5d181a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.352+0000 7f5d17e00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d18105d90 0x7f5d181a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.352+0000 7f5d17e00640 1 -- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d181053d0 msgr2=0x7f5d1819a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.352+0000 7f5d17e00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d181053d0 0x7f5d1819a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.352+0000 7f5d17e00640 1 -- 172.21.15.28:0/389334233 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d181a1f40 con 0x7f5d181041e0 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.352+0000 7f5d17e00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d181041e0 0x7f5d1819a320 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e9c0 tx=0x7f5d0800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.353+0000 7f5d14c00640 1 -- 172.21.15.28:0/389334233 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800cde0 con 0x7f5d181041e0 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.353+0000 7f5d14c00640 1 -- 172.21.15.28:0/389334233 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d080108e0 con 0x7f5d181041e0 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.353+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d181a2230 con 0x7f5d181041e0 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.353+0000 7f5d14c00640 1 -- 172.21.15.28:0/389334233 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08018960 con 0x7f5d181041e0 2024-04-26T01:41:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.353+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d181a27a0 con 0x7f5d181041e0 2024-04-26T01:41:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.354+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d1810b4a0 con 0x7f5d181041e0 2024-04-26T01:41:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.356+0000 7f5d14c00640 1 -- 172.21.15.28:0/389334233 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 24) ==== 138766+0+0 (secure 0 0 0) 0x7f5d0801f020 con 0x7f5d181041e0 2024-04-26T01:41:00.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.357+0000 7f5d14c00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5cfc0a3e40 0x7f5cfc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:00.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.357+0000 7f5d14c00640 1 -- 172.21.15.28:0/389334233 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f5d080d0b90 con 0x7f5d181041e0 2024-04-26T01:41:00.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.358+0000 7f5d17400640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5cfc0a3e40 0x7f5cfc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.358+0000 7f5d17400640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5cfc0a3e40 0x7f5cfc0a6300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5d04000c00 tx=0x7f5d040314a0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:00.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.361+0000 7f5d14c00640 1 -- 172.21.15.28:0/389334233 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d08053ff0 con 0x7f5d181041e0 2024-04-26T01:41:00.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: [26/Apr/2024:01:40:59] ENGINE Bus STARTING 2024-04-26T01:41:00.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: [26/Apr/2024:01:40:59] ENGINE Serving on https://172.21.15.28:7150 2024-04-26T01:41:00.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: [26/Apr/2024:01:40:59] ENGINE Serving on http://172.21.15.28:8765 2024-04-26T01:41:00.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: [26/Apr/2024:01:40:59] ENGINE Bus STARTED 2024-04-26T01:41:00.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:00.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.610+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f5d18062390 con 0x7f5cfc0a3e40 2024-04-26T01:41:00.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.620+0000 7f5d14c00640 1 -- 172.21.15.28:0/389334233 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f5d18062390 con 0x7f5cfc0a3e40 2024-04-26T01:41:00.617 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-26T01:41:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5cfc0a3e40 msgr2=0x7f5cfc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5cfc0a3e40 0x7f5cfc0a6300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5d04000c00 tx=0x7f5d040314a0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d181041e0 msgr2=0x7f5d1819a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d181041e0 0x7f5d1819a320 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e9c0 tx=0x7f5d0800ee80 comp rx=0 tx=0).stop 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 shutdown_connections 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5cfc0a3e40 0x7f5cfc0a6300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d18105d90 0x7f5d181a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d181053d0 0x7f5d1819a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 --2- 172.21.15.28:0/389334233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d181041e0 0x7f5d1819a320 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 >> 172.21.15.28:0/389334233 conn(0x7f5d180ff770 msgr2=0x7f5d18108670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 shutdown_connections 2024-04-26T01:41:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:00.622+0000 7f5d1ee00640 1 -- 172.21.15.28:0/389334233 wait complete. 2024-04-26T01:41:00.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-26T01:41:00.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: [26/Apr/2024:01:40:59] ENGINE Bus STARTING 2024-04-26T01:41:00.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: [26/Apr/2024:01:40:59] ENGINE Serving on https://172.21.15.28:7150 2024-04-26T01:41:00.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: [26/Apr/2024:01:40:59] ENGINE Serving on http://172.21.15.28:8765 2024-04-26T01:41:00.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: [26/Apr/2024:01:40:59] ENGINE Bus STARTED 2024-04-26T01:41:00.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:00.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:00 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: [26/Apr/2024:01:40:59] ENGINE Bus STARTING 2024-04-26T01:41:00.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: [26/Apr/2024:01:40:59] ENGINE Serving on https://172.21.15.28:7150 2024-04-26T01:41:00.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: [26/Apr/2024:01:40:59] ENGINE Serving on http://172.21.15.28:8765 2024-04-26T01:41:00.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: [26/Apr/2024:01:40:59] ENGINE Bus STARTED 2024-04-26T01:41:00.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:00.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:00 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:00.720 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-26T01:41:00.731 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-26T01:41:00.809 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:41:00.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-26T01:41:00.815 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-26T01:41:00.829 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:41:00.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:41:00.889 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:41:00.647+0000 7f2eb8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:00.889 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:41:00.647+0000 7f2eb8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:00.889 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:41:00.647+0000 7f2eb8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:00.889 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:41:00.647+0000 7f2eb8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:00.889 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:41:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:41:00.647+0000 7f2eb8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:01.239 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:41:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:01 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:01 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:01.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:41:01.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:01 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:02.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-04-26T01:41:02.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: Unable to set osd_memory_target on smithi169 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T01:41:02.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:02.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:02.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-26T01:41:02.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:02 smithi169 ceph-mon[32851]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T01:41:02.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: Unable to set osd_memory_target on smithi169 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-26T01:41:02.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:02 smithi053 ceph-mon[32593]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T01:41:02.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: from='client.25165 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:02.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-04-26T01:41:02.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: Unable to set osd_memory_target on smithi169 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T01:41:02.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:02.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:02.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:02.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-26T01:41:02.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:02 smithi028 ceph-mon[28247]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T01:41:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.601+0000 7f8d44600700 1 -- 172.21.15.28:0/2053056578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 msgr2=0x7f8d3c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.601+0000 7f8d44600700 1 --2- 172.21.15.28:0/2053056578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 0x7f8d3c075050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009990 tx=0x7f8d30009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.601+0000 7f8d44600700 1 -- 172.21.15.28:0/2053056578 shutdown_connections 2024-04-26T01:41:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.601+0000 7f8d44600700 1 --2- 172.21.15.28:0/2053056578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d3c10f440 0x7f8d3c111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.601+0000 7f8d44600700 1 --2- 172.21.15.28:0/2053056578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 0x7f8d3c075050 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.601+0000 7f8d44600700 1 --2- 172.21.15.28:0/2053056578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c075590 0x7f8d3c10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.601+0000 7f8d44600700 1 -- 172.21.15.28:0/2053056578 >> 172.21.15.28:0/2053056578 conn(0x7f8d3c079ba0 msgr2=0x7f8d3c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.602+0000 7f8d44600700 1 -- 172.21.15.28:0/2053056578 shutdown_connections 2024-04-26T01:41:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.602+0000 7f8d44600700 1 -- 172.21.15.28:0/2053056578 wait complete. 2024-04-26T01:41:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.603+0000 7f8d44600700 1 Processor -- start 2024-04-26T01:41:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.603+0000 7f8d44600700 1 -- start start 2024-04-26T01:41:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d44600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 0x7f8d3c19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d44600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d3c075590 0x7f8d3c19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d44600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c10f440 0x7f8d3c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d44600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c114e30 con 0x7f8d3c074c70 2024-04-26T01:41:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d44600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c114cb0 con 0x7f8d3c10f440 2024-04-26T01:41:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d44600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c114fb0 con 0x7f8d3c075590 2024-04-26T01:41:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d41c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 0x7f8d3c19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d41c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 0x7f8d3c19b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39210/0 (socket says 172.21.15.28:39210) 2024-04-26T01:41:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.604+0000 7f8d41c00700 1 -- 172.21.15.28:0/1177399030 learned_addr learned my addr 172.21.15.28:0/1177399030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.605+0000 7f8d41200700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d3c075590 0x7f8d3c19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.605+0000 7f8d42600700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c10f440 0x7f8d3c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.605+0000 7f8d41c00700 1 -- 172.21.15.28:0/1177399030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d3c075590 msgr2=0x7f8d3c19b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.605+0000 7f8d41c00700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d3c075590 0x7f8d3c19b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.605+0000 7f8d41c00700 1 -- 172.21.15.28:0/1177399030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c10f440 msgr2=0x7f8d3c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.605+0000 7f8d41c00700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c10f440 0x7f8d3c19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.605+0000 7f8d41c00700 1 -- 172.21.15.28:0/1177399030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3c1a0380 con 0x7f8d3c074c70 2024-04-26T01:41:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.605+0000 7f8d41c00700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 0x7f8d3c19b3c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8d30000c00 tx=0x7f8d300057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.606+0000 7f8d3a000700 1 -- 172.21.15.28:0/1177399030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30027070 con 0x7f8d3c074c70 2024-04-26T01:41:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.606+0000 7f8d3a000700 1 -- 172.21.15.28:0/1177399030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d30022470 con 0x7f8d3c074c70 2024-04-26T01:41:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.606+0000 7f8d3a000700 1 -- 172.21.15.28:0/1177399030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30017650 con 0x7f8d3c074c70 2024-04-26T01:41:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.606+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3c1a0610 con 0x7f8d3c074c70 2024-04-26T01:41:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.606+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d3c1a7ef0 con 0x7f8d3c074c70 2024-04-26T01:41:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.608+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d3c069730 con 0x7f8d3c074c70 2024-04-26T01:41:03.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.609+0000 7f8d3a000700 1 -- 172.21.15.28:0/1177399030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f8d3002a070 con 0x7f8d3c074c70 2024-04-26T01:41:03.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:03 smithi028 ceph-mon[28247]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:03 smithi028 ceph-mon[28247]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:41:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.612+0000 7f8d3a000700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8d240a3ed0 0x7f8d240a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.612+0000 7f8d41200700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8d240a3ed0 0x7f8d240a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.612+0000 7f8d3a000700 1 -- 172.21.15.28:0/1177399030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f8d300d16d0 con 0x7f8d3c074c70 2024-04-26T01:41:03.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.613+0000 7f8d41200700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8d240a3ed0 0x7f8d240a6390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009680 tx=0x7f8d28009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.618+0000 7f8d3a000700 1 -- 172.21.15.28:0/1177399030 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d30013070 con 0x7f8d3c074c70 2024-04-26T01:41:03.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:03 smithi169 ceph-mon[32851]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:03 smithi169 ceph-mon[32851]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:41:03.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:03 smithi053 ceph-mon[32593]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:03 smithi053 ceph-mon[32593]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:41:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.865+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8d3c19c6d0 con 0x7f8d240a3ed0 2024-04-26T01:41:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.874+0000 7f8d3a000700 1 -- 172.21.15.28:0/1177399030 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258058 (secure 0 0 0) 0x7f8d3c19c6d0 con 0x7f8d240a3ed0 2024-04-26T01:41:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8d240a3ed0 msgr2=0x7f8d240a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8d240a3ed0 0x7f8d240a6390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009680 tx=0x7f8d28009260 comp rx=0 tx=0).stop 2024-04-26T01:41:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 msgr2=0x7f8d3c19b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 0x7f8d3c19b3c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8d30000c00 tx=0x7f8d300057c0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 shutdown_connections 2024-04-26T01:41:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8d240a3ed0 0x7f8d240a6390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c10f440 0x7f8d3c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d3c074c70 0x7f8d3c19b3c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 --2- 172.21.15.28:0/1177399030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d3c075590 0x7f8d3c19b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.880+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 >> 172.21.15.28:0/1177399030 conn(0x7f8d3c079ba0 msgr2=0x7f8d3c0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.881+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 shutdown_connections 2024-04-26T01:41:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:03.881+0000 7f8d44600700 1 -- 172.21.15.28:0/1177399030 wait complete. 2024-04-26T01:41:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-26T01:41:04.926 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-26T01:41:01.965338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191566,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1589,"num_read_kb":959,"num_write":481,"num_write_kb":4384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181492,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378132,"kb_used_data":12300,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563692,"statfs":{"total":1151940427776,"available":1151553220608,"internally_reserved":0,"allocated":12595200,"data_stored":9433019,"data_compressed":224022,"data_compressed_allocated":294912,"data_compressed_original":3440640,"omap_allocated":19140,"internal_metadata":374453564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.982510"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206116+0000","last_change":"2024-04-26T01:36:54.288460+0000","last_active":"2024-04-26T01:40:58.206116+0000","last_peered":"2024-04-26T01:40:58.206116+0000","last_clean":"2024-04-26T01:40:58.206116+0000","last_became_active":"2024-04-26T01:36:54.288033+0000","last_became_peered":"2024-04-26T01:36:54.288033+0000","last_unstale":"2024-04-26T01:40:58.206116+0000","last_undegraded":"2024-04-26T01:40:58.206116+0000","last_fullsized":"2024-04-26T01:40:58.206116+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:33.737622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938795+0000","last_change":"2024-04-26T01:36:54.287367+0000","last_active":"2024-04-26T01:40:57.938795+0000","last_peered":"2024-04-26T01:40:57.938795+0000","last_clean":"2024-04-26T01:40:57.938795+0000","last_became_active":"2024-04-26T01:36:54.287024+0000","last_became_peered":"2024-04-26T01:36:54.287024+0000","last_unstale":"2024-04-26T01:40:57.938795+0000","last_undegraded":"2024-04-26T01:40:57.938795+0000","last_fullsized":"2024-04-26T01:40:57.938795+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:31.014627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938474+0000","last_change":"2024-04-26T01:36:54.288433+0000","last_active":"2024-04-26T01:40:57.938474+0000","last_peered":"2024-04-26T01:40:57.938474+0000","last_clean":"2024-04-26T01:40:57.938474+0000","last_became_active":"2024-04-26T01:36:54.288002+0000","last_became_peered":"2024-04-26T01:36:54.288002+0000","last_unstale":"2024-04-26T01:40:57.938474+0000","last_undegraded":"2024-04-26T01:40:57.938474+0000","last_fullsized":"2024-04-26T01:40:57.938474+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:37.885179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939129+0000","last_change":"2024-04-26T01:36:54.273857+0000","last_active":"2024-04-26T01:40:57.939129+0000","last_peered":"2024-04-26T01:40:57.939129+0000","last_clean":"2024-04-26T01:40:57.939129+0000","last_became_active":"2024-04-26T01:36:54.272376+0000","last_became_peered":"2024-04-26T01:36:54.272376+0000","last_unstale":"2024-04-26T01:40:57.939129+0000","last_undegraded":"2024-04-26T01:40:57.939129+0000","last_fullsized":"2024-04-26T01:40:57.939129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:30.110044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079196+0000","last_change":"2024-04-26T01:36:54.288606+0000","last_active":"2024-04-26T01:40:58.079196+0000","last_peered":"2024-04-26T01:40:58.079196+0000","last_clean":"2024-04-26T01:40:58.079196+0000","last_became_active":"2024-04-26T01:36:54.288247+0000","last_became_peered":"2024-04-26T01:36:54.288247+0000","last_unstale":"2024-04-26T01:40:58.079196+0000","last_undegraded":"2024-04-26T01:40:58.079196+0000","last_fullsized":"2024-04-26T01:40:58.079196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:47:56.944164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967023+0000","last_change":"2024-04-26T01:36:54.277701+0000","last_active":"2024-04-26T01:40:57.967023+0000","last_peered":"2024-04-26T01:40:57.967023+0000","last_clean":"2024-04-26T01:40:57.967023+0000","last_became_active":"2024-04-26T01:36:54.277306+0000","last_became_peered":"2024-04-26T01:36:54.277306+0000","last_unstale":"2024-04-26T01:40:57.967023+0000","last_undegraded":"2024-04-26T01:40:57.967023+0000","last_fullsized":"2024-04-26T01:40:57.967023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:42:05.058525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206390+0000","last_change":"2024-04-26T01:36:54.274090+0000","last_active":"2024-04-26T01:40:58.206390+0000","last_peered":"2024-04-26T01:40:58.206390+0000","last_clean":"2024-04-26T01:40:58.206390+0000","last_became_active":"2024-04-26T01:36:54.273700+0000","last_became_peered":"2024-04-26T01:36:54.273700+0000","last_unstale":"2024-04-26T01:40:58.206390+0000","last_undegraded":"2024-04-26T01:40:58.206390+0000","last_fullsized":"2024-04-26T01:40:58.206390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:32.207534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076612+0000","last_change":"2024-04-26T01:37:25.499759+0000","last_active":"2024-04-26T01:40:58.076612+0000","last_peered":"2024-04-26T01:40:58.076612+0000","last_clean":"2024-04-26T01:40:58.076612+0000","last_became_active":"2024-04-26T01:37:25.499317+0000","last_became_peered":"2024-04-26T01:37:25.499317+0000","last_unstale":"2024-04-26T01:40:58.076612+0000","last_undegraded":"2024-04-26T01:40:58.076612+0000","last_fullsized":"2024-04-26T01:40:58.076612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:39.972315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206270+0000","last_change":"2024-04-26T01:36:54.277457+0000","last_active":"2024-04-26T01:40:58.206270+0000","last_peered":"2024-04-26T01:40:58.206270+0000","last_clean":"2024-04-26T01:40:58.206270+0000","last_became_active":"2024-04-26T01:36:54.277118+0000","last_became_peered":"2024-04-26T01:36:54.277118+0000","last_unstale":"2024-04-26T01:40:58.206270+0000","last_undegraded":"2024-04-26T01:40:58.206270+0000","last_fullsized":"2024-04-26T01:40:58.206270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:05.628027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967353+0000","last_change":"2024-04-26T01:36:54.278494+0000","last_active":"2024-04-26T01:40:57.967353+0000","last_peered":"2024-04-26T01:40:57.967353+0000","last_clean":"2024-04-26T01:40:57.967353+0000","last_became_active":"2024-04-26T01:36:54.278188+0000","last_became_peered":"2024-04-26T01:36:54.278188+0000","last_unstale":"2024-04-26T01:40:57.967353+0000","last_undegraded":"2024-04-26T01:40:57.967353+0000","last_fullsized":"2024-04-26T01:40:57.967353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:59.150771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.207126+0000","last_change":"2024-04-26T01:36:54.277424+0000","last_active":"2024-04-26T01:40:58.207126+0000","last_peered":"2024-04-26T01:40:58.207126+0000","last_clean":"2024-04-26T01:40:58.207126+0000","last_became_active":"2024-04-26T01:36:54.275752+0000","last_became_peered":"2024-04-26T01:36:54.275752+0000","last_unstale":"2024-04-26T01:40:58.207126+0000","last_undegraded":"2024-04-26T01:40:58.207126+0000","last_fullsized":"2024-04-26T01:40:58.207126+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:22.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076889+0000","last_change":"2024-04-26T01:37:25.502448+0000","last_active":"2024-04-26T01:40:58.076889+0000","last_peered":"2024-04-26T01:40:58.076889+0000","last_clean":"2024-04-26T01:40:58.076889+0000","last_became_active":"2024-04-26T01:37:25.501161+0000","last_became_peered":"2024-04-26T01:37:25.501161+0000","last_unstale":"2024-04-26T01:40:58.076889+0000","last_undegraded":"2024-04-26T01:40:58.076889+0000","last_fullsized":"2024-04-26T01:40:58.076889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:41:40.443780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949697+0000","last_change":"2024-04-26T01:36:54.268721+0000","last_active":"2024-04-26T01:40:57.949697+0000","last_peered":"2024-04-26T01:40:57.949697+0000","last_clean":"2024-04-26T01:40:57.949697+0000","last_became_active":"2024-04-26T01:36:54.268007+0000","last_became_peered":"2024-04-26T01:36:54.268007+0000","last_unstale":"2024-04-26T01:40:57.949697+0000","last_undegraded":"2024-04-26T01:40:57.949697+0000","last_fullsized":"2024-04-26T01:40:57.949697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:55.120130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938165+0000","last_change":"2024-04-26T01:36:54.286595+0000","last_active":"2024-04-26T01:40:57.938165+0000","last_peered":"2024-04-26T01:40:57.938165+0000","last_clean":"2024-04-26T01:40:57.938165+0000","last_became_active":"2024-04-26T01:36:54.286236+0000","last_became_peered":"2024-04-26T01:36:54.286236+0000","last_unstale":"2024-04-26T01:40:57.938165+0000","last_undegraded":"2024-04-26T01:40:57.938165+0000","last_fullsized":"2024-04-26T01:40:57.938165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:59.443467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938105+0000","last_change":"2024-04-26T01:36:54.293254+0000","last_active":"2024-04-26T01:40:57.938105+0000","last_peered":"2024-04-26T01:40:57.938105+0000","last_clean":"2024-04-26T01:40:57.938105+0000","last_became_active":"2024-04-26T01:36:54.292853+0000","last_became_peered":"2024-04-26T01:36:54.292853+0000","last_unstale":"2024-04-26T01:40:57.938105+0000","last_undegraded":"2024-04-26T01:40:57.938105+0000","last_fullsized":"2024-04-26T01:40:57.938105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:19.844006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205747+0000","last_change":"2024-04-26T01:36:54.285346+0000","last_active":"2024-04-26T01:40:58.205747+0000","last_peered":"2024-04-26T01:40:58.205747+0000","last_clean":"2024-04-26T01:40:58.205747+0000","last_became_active":"2024-04-26T01:36:54.284949+0000","last_became_peered":"2024-04-26T01:36:54.284949+0000","last_unstale":"2024-04-26T01:40:58.205747+0000","last_undegraded":"2024-04-26T01:40:58.205747+0000","last_fullsized":"2024-04-26T01:40:58.205747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:33:29.276240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205959+0000","last_change":"2024-04-26T01:36:54.283161+0000","last_active":"2024-04-26T01:40:58.205959+0000","last_peered":"2024-04-26T01:40:58.205959+0000","last_clean":"2024-04-26T01:40:58.205959+0000","last_became_active":"2024-04-26T01:36:54.282908+0000","last_became_peered":"2024-04-26T01:36:54.282908+0000","last_unstale":"2024-04-26T01:40:58.205959+0000","last_undegraded":"2024-04-26T01:40:58.205959+0000","last_fullsized":"2024-04-26T01:40:58.205959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:26.503964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.077086+0000","last_change":"2024-04-26T01:36:54.282516+0000","last_active":"2024-04-26T01:40:58.077086+0000","last_peered":"2024-04-26T01:40:58.077086+0000","last_clean":"2024-04-26T01:40:58.077086+0000","last_became_active":"2024-04-26T01:36:54.278634+0000","last_became_peered":"2024-04-26T01:36:54.278634+0000","last_unstale":"2024-04-26T01:40:58.077086+0000","last_undegraded":"2024-04-26T01:40:58.077086+0000","last_fullsized":"2024-04-26T01:40:58.077086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:10:38.275720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939505+0000","last_change":"2024-04-26T01:36:54.289502+0000","last_active":"2024-04-26T01:40:57.939505+0000","last_peered":"2024-04-26T01:40:57.939505+0000","last_clean":"2024-04-26T01:40:57.939505+0000","last_became_active":"2024-04-26T01:36:54.289079+0000","last_became_peered":"2024-04-26T01:36:54.289079+0000","last_unstale":"2024-04-26T01:40:57.939505+0000","last_undegraded":"2024-04-26T01:40:57.939505+0000","last_fullsized":"2024-04-26T01:40:57.939505+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:51.412032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076838+0000","last_change":"2024-04-26T01:36:50.243164+0000","last_active":"2024-04-26T01:40:58.076838+0000","last_peered":"2024-04-26T01:40:58.076838+0000","last_clean":"2024-04-26T01:40:58.076838+0000","last_became_active":"2024-04-26T01:36:50.242508+0000","last_became_peered":"2024-04-26T01:36:50.242508+0000","last_unstale":"2024-04-26T01:40:58.076838+0000","last_undegraded":"2024-04-26T01:40:58.076838+0000","last_fullsized":"2024-04-26T01:40:58.076838+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:06:01.662698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954129+0000","last_change":"2024-04-26T01:36:54.279809+0000","last_active":"2024-04-26T01:40:57.954129+0000","last_peered":"2024-04-26T01:40:57.954129+0000","last_clean":"2024-04-26T01:40:57.954129+0000","last_became_active":"2024-04-26T01:36:54.279509+0000","last_became_peered":"2024-04-26T01:36:54.279509+0000","last_unstale":"2024-04-26T01:40:57.954129+0000","last_undegraded":"2024-04-26T01:40:57.954129+0000","last_fullsized":"2024-04-26T01:40:57.954129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:54:26.734136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954751+0000","last_change":"2024-04-26T01:36:50.261942+0000","last_active":"2024-04-26T01:40:57.954751+0000","last_peered":"2024-04-26T01:40:57.954751+0000","last_clean":"2024-04-26T01:40:57.954751+0000","last_became_active":"2024-04-26T01:36:50.261564+0000","last_became_peered":"2024-04-26T01:36:50.261564+0000","last_unstale":"2024-04-26T01:40:57.954751+0000","last_undegraded":"2024-04-26T01:40:57.954751+0000","last_fullsized":"2024-04-26T01:40:57.954751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:25.102557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938723+0000","last_change":"2024-04-26T01:36:54.288011+0000","last_active":"2024-04-26T01:40:57.938723+0000","last_peered":"2024-04-26T01:40:57.938723+0000","last_clean":"2024-04-26T01:40:57.938723+0000","last_became_active":"2024-04-26T01:36:54.286853+0000","last_became_peered":"2024-04-26T01:36:54.286853+0000","last_unstale":"2024-04-26T01:40:57.938723+0000","last_undegraded":"2024-04-26T01:40:57.938723+0000","last_fullsized":"2024-04-26T01:40:57.938723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:53.422470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079561+0000","last_change":"2024-04-26T01:36:50.235291+0000","last_active":"2024-04-26T01:40:58.079561+0000","last_peered":"2024-04-26T01:40:58.079561+0000","last_clean":"2024-04-26T01:40:58.079561+0000","last_became_active":"2024-04-26T01:36:50.234939+0000","last_became_peered":"2024-04-26T01:36:50.234939+0000","last_unstale":"2024-04-26T01:40:58.079561+0000","last_undegraded":"2024-04-26T01:40:58.079561+0000","last_fullsized":"2024-04-26T01:40:58.079561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:13.048185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205932+0000","last_change":"2024-04-26T01:36:54.282731+0000","last_active":"2024-04-26T01:40:58.205932+0000","last_peered":"2024-04-26T01:40:58.205932+0000","last_clean":"2024-04-26T01:40:58.205932+0000","last_became_active":"2024-04-26T01:36:54.282384+0000","last_became_peered":"2024-04-26T01:36:54.282384+0000","last_unstale":"2024-04-26T01:40:58.205932+0000","last_undegraded":"2024-04-26T01:40:58.205932+0000","last_fullsized":"2024-04-26T01:40:58.205932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:09.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079407+0000","last_change":"2024-04-26T01:36:50.233581+0000","last_active":"2024-04-26T01:40:58.079407+0000","last_peered":"2024-04-26T01:40:58.079407+0000","last_clean":"2024-04-26T01:40:58.079407+0000","last_became_active":"2024-04-26T01:36:50.233210+0000","last_became_peered":"2024-04-26T01:36:50.233210+0000","last_unstale":"2024-04-26T01:40:58.079407+0000","last_undegraded":"2024-04-26T01:40:58.079407+0000","last_fullsized":"2024-04-26T01:40:58.079407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:01.223954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970312+0000","last_change":"2024-04-26T01:36:54.274657+0000","last_active":"2024-04-26T01:40:57.970312+0000","last_peered":"2024-04-26T01:40:57.970312+0000","last_clean":"2024-04-26T01:40:57.970312+0000","last_became_active":"2024-04-26T01:36:54.274333+0000","last_became_peered":"2024-04-26T01:36:54.274333+0000","last_unstale":"2024-04-26T01:40:57.970312+0000","last_undegraded":"2024-04-26T01:40:57.970312+0000","last_fullsized":"2024-04-26T01:40:57.970312+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:37.243307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":277,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.973266+0000","last_change":"2024-04-26T01:37:25.502311+0000","last_active":"2024-04-26T01:40:58.973266+0000","last_peered":"2024-04-26T01:40:58.973266+0000","last_clean":"2024-04-26T01:40:58.973266+0000","last_became_active":"2024-04-26T01:37:25.493259+0000","last_became_peered":"2024-04-26T01:37:25.493259+0000","last_unstale":"2024-04-26T01:40:58.973266+0000","last_undegraded":"2024-04-26T01:40:58.973266+0000","last_fullsized":"2024-04-26T01:40:58.973266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:46:49.224905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079111+0000","last_change":"2024-04-26T01:36:54.275116+0000","last_active":"2024-04-26T01:40:58.079111+0000","last_peered":"2024-04-26T01:40:58.079111+0000","last_clean":"2024-04-26T01:40:58.079111+0000","last_became_active":"2024-04-26T01:36:54.274815+0000","last_became_peered":"2024-04-26T01:36:54.274815+0000","last_unstale":"2024-04-26T01:40:58.079111+0000","last_undegraded":"2024-04-26T01:40:58.079111+0000","last_fullsized":"2024-04-26T01:40:58.079111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:04.427670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.966940+0000","last_change":"2024-04-26T01:37:25.505327+0000","last_active":"2024-04-26T01:40:57.966940+0000","last_peered":"2024-04-26T01:40:57.966940+0000","last_clean":"2024-04-26T01:40:57.966940+0000","last_became_active":"2024-04-26T01:37:25.497881+0000","last_became_peered":"2024-04-26T01:37:25.497881+0000","last_unstale":"2024-04-26T01:40:57.966940+0000","last_undegraded":"2024-04-26T01:40:57.966940+0000","last_fullsized":"2024-04-26T01:40:57.966940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:09.181372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970138+0000","last_change":"2024-04-26T01:36:54.276415+0000","last_active":"2024-04-26T01:40:57.970138+0000","last_peered":"2024-04-26T01:40:57.970138+0000","last_clean":"2024-04-26T01:40:57.970138+0000","last_became_active":"2024-04-26T01:36:54.276080+0000","last_became_peered":"2024-04-26T01:36:54.276080+0000","last_unstale":"2024-04-26T01:40:57.970138+0000","last_undegraded":"2024-04-26T01:40:57.970138+0000","last_fullsized":"2024-04-26T01:40:57.970138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:08:17.171620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949572+0000","last_change":"2024-04-26T01:36:50.241552+0000","last_active":"2024-04-26T01:40:57.949572+0000","last_peered":"2024-04-26T01:40:57.949572+0000","last_clean":"2024-04-26T01:40:57.949572+0000","last_became_active":"2024-04-26T01:36:50.241022+0000","last_became_peered":"2024-04-26T01:36:50.241022+0000","last_unstale":"2024-04-26T01:40:57.949572+0000","last_undegraded":"2024-04-26T01:40:57.949572+0000","last_fullsized":"2024-04-26T01:40:57.949572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:55.503831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953427+0000","last_change":"2024-04-26T01:36:54.276641+0000","last_active":"2024-04-26T01:40:57.953427+0000","last_peered":"2024-04-26T01:40:57.953427+0000","last_clean":"2024-04-26T01:40:57.953427+0000","last_became_active":"2024-04-26T01:36:54.276309+0000","last_became_peered":"2024-04-26T01:36:54.276309+0000","last_unstale":"2024-04-26T01:40:57.953427+0000","last_undegraded":"2024-04-26T01:40:57.953427+0000","last_fullsized":"2024-04-26T01:40:57.953427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:53.725584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969805+0000","last_change":"2024-04-26T01:36:50.235504+0000","last_active":"2024-04-26T01:40:57.969805+0000","last_peered":"2024-04-26T01:40:57.969805+0000","last_clean":"2024-04-26T01:40:57.969805+0000","last_became_active":"2024-04-26T01:36:50.235161+0000","last_became_peered":"2024-04-26T01:36:50.235161+0000","last_unstale":"2024-04-26T01:40:57.969805+0000","last_undegraded":"2024-04-26T01:40:57.969805+0000","last_fullsized":"2024-04-26T01:40:57.969805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:05.951521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076712+0000","last_change":"2024-04-26T01:36:54.298236+0000","last_active":"2024-04-26T01:40:58.076712+0000","last_peered":"2024-04-26T01:40:58.076712+0000","last_clean":"2024-04-26T01:40:58.076712+0000","last_became_active":"2024-04-26T01:36:54.297850+0000","last_became_peered":"2024-04-26T01:36:54.297850+0000","last_unstale":"2024-04-26T01:40:58.076712+0000","last_undegraded":"2024-04-26T01:40:58.076712+0000","last_fullsized":"2024-04-26T01:40:58.076712+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:35:35.244502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":255,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967591+0000","last_change":"2024-04-26T01:36:50.241418+0000","last_active":"2024-04-26T01:40:57.967591+0000","last_peered":"2024-04-26T01:40:57.967591+0000","last_clean":"2024-04-26T01:40:57.967591+0000","last_became_active":"2024-04-26T01:36:50.241044+0000","last_became_peered":"2024-04-26T01:36:50.241044+0000","last_unstale":"2024-04-26T01:40:57.967591+0000","last_undegraded":"2024-04-26T01:40:57.967591+0000","last_fullsized":"2024-04-26T01:40:57.967591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:13.317953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":221,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206340+0000","last_change":"2024-04-26T01:36:54.284410+0000","last_active":"2024-04-26T01:40:58.206340+0000","last_peered":"2024-04-26T01:40:58.206340+0000","last_clean":"2024-04-26T01:40:58.206340+0000","last_became_active":"2024-04-26T01:36:54.283629+0000","last_became_peered":"2024-04-26T01:36:54.283629+0000","last_unstale":"2024-04-26T01:40:58.206340+0000","last_undegraded":"2024-04-26T01:40:58.206340+0000","last_fullsized":"2024-04-26T01:40:58.206340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:15.380006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205831+0000","last_change":"2024-04-26T01:37:25.499874+0000","last_active":"2024-04-26T01:40:58.205831+0000","last_peered":"2024-04-26T01:40:58.205831+0000","last_clean":"2024-04-26T01:40:58.205831+0000","last_became_active":"2024-04-26T01:37:25.494878+0000","last_became_peered":"2024-04-26T01:37:25.494878+0000","last_unstale":"2024-04-26T01:40:58.205831+0000","last_undegraded":"2024-04-26T01:40:58.205831+0000","last_fullsized":"2024-04-26T01:40:58.205831+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:48:12.678331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938888+0000","last_change":"2024-04-26T01:36:54.285118+0000","last_active":"2024-04-26T01:40:57.938888+0000","last_peered":"2024-04-26T01:40:57.938888+0000","last_clean":"2024-04-26T01:40:57.938888+0000","last_became_active":"2024-04-26T01:36:54.284554+0000","last_became_peered":"2024-04-26T01:36:54.284554+0000","last_unstale":"2024-04-26T01:40:57.938888+0000","last_undegraded":"2024-04-26T01:40:57.938888+0000","last_fullsized":"2024-04-26T01:40:57.938888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:13.721684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970272+0000","last_change":"2024-04-26T01:36:50.262451+0000","last_active":"2024-04-26T01:40:57.970272+0000","last_peered":"2024-04-26T01:40:57.970272+0000","last_clean":"2024-04-26T01:40:57.970272+0000","last_became_active":"2024-04-26T01:36:50.262056+0000","last_became_peered":"2024-04-26T01:36:50.262056+0000","last_unstale":"2024-04-26T01:40:57.970272+0000","last_undegraded":"2024-04-26T01:40:57.970272+0000","last_fullsized":"2024-04-26T01:40:57.970272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:50.813675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206136+0000","last_change":"2024-04-26T01:36:54.283975+0000","last_active":"2024-04-26T01:40:58.206136+0000","last_peered":"2024-04-26T01:40:58.206136+0000","last_clean":"2024-04-26T01:40:58.206136+0000","last_became_active":"2024-04-26T01:36:54.283564+0000","last_became_peered":"2024-04-26T01:36:54.283564+0000","last_unstale":"2024-04-26T01:40:58.206136+0000","last_undegraded":"2024-04-26T01:40:58.206136+0000","last_fullsized":"2024-04-26T01:40:58.206136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:53.881763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'39","reported_seq":524,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.801073+0000","last_change":"2024-04-26T01:36:50.238839+0000","last_active":"2024-04-26T01:40:58.801073+0000","last_peered":"2024-04-26T01:40:58.801073+0000","last_clean":"2024-04-26T01:40:58.801073+0000","last_became_active":"2024-04-26T01:36:50.238452+0000","last_became_peered":"2024-04-26T01:36:50.238452+0000","last_unstale":"2024-04-26T01:40:58.801073+0000","last_undegraded":"2024-04-26T01:40:58.801073+0000","last_fullsized":"2024-04-26T01:40:58.801073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:05:40.426825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70914,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":456,"num_read_kb":297,"num_write":40,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079361+0000","last_change":"2024-04-26T01:36:54.271446+0000","last_active":"2024-04-26T01:40:58.079361+0000","last_peered":"2024-04-26T01:40:58.079361+0000","last_clean":"2024-04-26T01:40:58.079361+0000","last_became_active":"2024-04-26T01:36:54.271122+0000","last_became_peered":"2024-04-26T01:36:54.271122+0000","last_unstale":"2024-04-26T01:40:58.079361+0000","last_undegraded":"2024-04-26T01:40:58.079361+0000","last_fullsized":"2024-04-26T01:40:58.079361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:28:24.275481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939266+0000","last_change":"2024-04-26T01:36:50.237726+0000","last_active":"2024-04-26T01:40:57.939266+0000","last_peered":"2024-04-26T01:40:57.939266+0000","last_clean":"2024-04-26T01:40:57.939266+0000","last_became_active":"2024-04-26T01:36:50.237215+0000","last_became_peered":"2024-04-26T01:36:50.237215+0000","last_unstale":"2024-04-26T01:40:57.939266+0000","last_undegraded":"2024-04-26T01:40:57.939266+0000","last_fullsized":"2024-04-26T01:40:57.939266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:41.781876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949425+0000","last_change":"2024-04-26T01:36:54.290616+0000","last_active":"2024-04-26T01:40:57.949425+0000","last_peered":"2024-04-26T01:40:57.949425+0000","last_clean":"2024-04-26T01:40:57.949425+0000","last_became_active":"2024-04-26T01:36:54.290293+0000","last_became_peered":"2024-04-26T01:36:54.290293+0000","last_unstale":"2024-04-26T01:40:57.949425+0000","last_undegraded":"2024-04-26T01:40:57.949425+0000","last_fullsized":"2024-04-26T01:40:57.949425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:07.439333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969982+0000","last_change":"2024-04-26T01:36:50.237578+0000","last_active":"2024-04-26T01:40:57.969982+0000","last_peered":"2024-04-26T01:40:57.969982+0000","last_clean":"2024-04-26T01:40:57.969982+0000","last_became_active":"2024-04-26T01:36:50.237224+0000","last_became_peered":"2024-04-26T01:36:50.237224+0000","last_unstale":"2024-04-26T01:40:57.969982+0000","last_undegraded":"2024-04-26T01:40:57.969982+0000","last_fullsized":"2024-04-26T01:40:57.969982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:46:38.078631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206709+0000","last_change":"2024-04-26T01:36:54.280818+0000","last_active":"2024-04-26T01:40:58.206709+0000","last_peered":"2024-04-26T01:40:58.206709+0000","last_clean":"2024-04-26T01:40:58.206709+0000","last_became_active":"2024-04-26T01:36:54.280440+0000","last_became_peered":"2024-04-26T01:36:54.280440+0000","last_unstale":"2024-04-26T01:40:58.206709+0000","last_undegraded":"2024-04-26T01:40:58.206709+0000","last_fullsized":"2024-04-26T01:40:58.206709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:47.185219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":258,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.951154+0000","last_change":"2024-04-26T01:36:50.234602+0000","last_active":"2024-04-26T01:40:57.951154+0000","last_peered":"2024-04-26T01:40:57.951154+0000","last_clean":"2024-04-26T01:40:57.951154+0000","last_became_active":"2024-04-26T01:36:50.234175+0000","last_became_peered":"2024-04-26T01:36:50.234175+0000","last_unstale":"2024-04-26T01:40:57.951154+0000","last_undegraded":"2024-04-26T01:40:57.951154+0000","last_fullsized":"2024-04-26T01:40:57.951154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:17:14.236069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":898,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":226,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206181+0000","last_change":"2024-04-26T01:36:54.271466+0000","last_active":"2024-04-26T01:40:58.206181+0000","last_peered":"2024-04-26T01:40:58.206181+0000","last_clean":"2024-04-26T01:40:58.206181+0000","last_became_active":"2024-04-26T01:36:54.271086+0000","last_became_peered":"2024-04-26T01:36:54.271086+0000","last_unstale":"2024-04-26T01:40:58.206181+0000","last_undegraded":"2024-04-26T01:40:58.206181+0000","last_fullsized":"2024-04-26T01:40:58.206181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:41.052895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079039+0000","last_change":"2024-04-26T01:37:25.518730+0000","last_active":"2024-04-26T01:40:58.079039+0000","last_peered":"2024-04-26T01:40:58.079039+0000","last_clean":"2024-04-26T01:40:58.079039+0000","last_became_active":"2024-04-26T01:37:25.504464+0000","last_became_peered":"2024-04-26T01:37:25.504464+0000","last_unstale":"2024-04-26T01:40:58.079039+0000","last_undegraded":"2024-04-26T01:40:58.079039+0000","last_fullsized":"2024-04-26T01:40:58.079039+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:07.578917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969886+0000","last_change":"2024-04-26T01:36:54.273621+0000","last_active":"2024-04-26T01:40:57.969886+0000","last_peered":"2024-04-26T01:40:57.969886+0000","last_clean":"2024-04-26T01:40:57.969886+0000","last_became_active":"2024-04-26T01:36:54.273218+0000","last_became_peered":"2024-04-26T01:36:54.273218+0000","last_unstale":"2024-04-26T01:40:57.969886+0000","last_undegraded":"2024-04-26T01:40:57.969886+0000","last_fullsized":"2024-04-26T01:40:57.969886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:55.543943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205908+0000","last_change":"2024-04-26T01:37:25.508575+0000","last_active":"2024-04-26T01:40:58.205908+0000","last_peered":"2024-04-26T01:40:58.205908+0000","last_clean":"2024-04-26T01:40:58.205908+0000","last_became_active":"2024-04-26T01:37:25.496407+0000","last_became_peered":"2024-04-26T01:37:25.496407+0000","last_unstale":"2024-04-26T01:40:58.205908+0000","last_undegraded":"2024-04-26T01:40:58.205908+0000","last_fullsized":"2024-04-26T01:40:58.205908+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:05.878252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205907+0000","last_change":"2024-04-26T01:36:54.272147+0000","last_active":"2024-04-26T01:40:58.205907+0000","last_peered":"2024-04-26T01:40:58.205907+0000","last_clean":"2024-04-26T01:40:58.205907+0000","last_became_active":"2024-04-26T01:36:54.271737+0000","last_became_peered":"2024-04-26T01:36:54.271737+0000","last_unstale":"2024-04-26T01:40:58.205907+0000","last_undegraded":"2024-04-26T01:40:58.205907+0000","last_fullsized":"2024-04-26T01:40:58.205907+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:41.445073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076929+0000","last_change":"2024-04-26T01:36:50.264444+0000","last_active":"2024-04-26T01:40:58.076929+0000","last_peered":"2024-04-26T01:40:58.076929+0000","last_clean":"2024-04-26T01:40:58.076929+0000","last_became_active":"2024-04-26T01:36:50.264058+0000","last_became_peered":"2024-04-26T01:36:50.264058+0000","last_unstale":"2024-04-26T01:40:58.076929+0000","last_undegraded":"2024-04-26T01:40:58.076929+0000","last_fullsized":"2024-04-26T01:40:58.076929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:56:15.428588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967226+0000","last_change":"2024-04-26T01:36:54.289537+0000","last_active":"2024-04-26T01:40:57.967226+0000","last_peered":"2024-04-26T01:40:57.967226+0000","last_clean":"2024-04-26T01:40:57.967226+0000","last_became_active":"2024-04-26T01:36:54.289149+0000","last_became_peered":"2024-04-26T01:36:54.289149+0000","last_unstale":"2024-04-26T01:40:57.967226+0000","last_undegraded":"2024-04-26T01:40:57.967226+0000","last_fullsized":"2024-04-26T01:40:57.967226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:07.801551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206728+0000","last_change":"2024-04-26T01:36:50.242190+0000","last_active":"2024-04-26T01:40:58.206728+0000","last_peered":"2024-04-26T01:40:58.206728+0000","last_clean":"2024-04-26T01:40:58.206728+0000","last_became_active":"2024-04-26T01:36:50.241890+0000","last_became_peered":"2024-04-26T01:36:50.241890+0000","last_unstale":"2024-04-26T01:40:58.206728+0000","last_undegraded":"2024-04-26T01:40:58.206728+0000","last_fullsized":"2024-04-26T01:40:58.206728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:29.472502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967023+0000","last_change":"2024-04-26T01:36:54.294699+0000","last_active":"2024-04-26T01:40:57.967023+0000","last_peered":"2024-04-26T01:40:57.967023+0000","last_clean":"2024-04-26T01:40:57.967023+0000","last_became_active":"2024-04-26T01:36:54.294313+0000","last_became_peered":"2024-04-26T01:36:54.294313+0000","last_unstale":"2024-04-26T01:40:57.967023+0000","last_undegraded":"2024-04-26T01:40:57.967023+0000","last_fullsized":"2024-04-26T01:40:57.967023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:15.026357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970178+0000","last_change":"2024-04-26T01:36:54.276798+0000","last_active":"2024-04-26T01:40:57.970178+0000","last_peered":"2024-04-26T01:40:57.970178+0000","last_clean":"2024-04-26T01:40:57.970178+0000","last_became_active":"2024-04-26T01:36:54.275955+0000","last_became_peered":"2024-04-26T01:36:54.275955+0000","last_unstale":"2024-04-26T01:40:57.970178+0000","last_undegraded":"2024-04-26T01:40:57.970178+0000","last_fullsized":"2024-04-26T01:40:57.970178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:17.020909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":256,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.950212+0000","last_change":"2024-04-26T01:36:50.263356+0000","last_active":"2024-04-26T01:40:57.950212+0000","last_peered":"2024-04-26T01:40:57.950212+0000","last_clean":"2024-04-26T01:40:57.950212+0000","last_became_active":"2024-04-26T01:36:50.261596+0000","last_became_peered":"2024-04-26T01:36:50.261596+0000","last_unstale":"2024-04-26T01:40:57.950212+0000","last_undegraded":"2024-04-26T01:40:57.950212+0000","last_fullsized":"2024-04-26T01:40:57.950212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:10.773347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":222,"num_read_kb":222,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953968+0000","last_change":"2024-04-26T01:36:54.288135+0000","last_active":"2024-04-26T01:40:57.953968+0000","last_peered":"2024-04-26T01:40:57.953968+0000","last_clean":"2024-04-26T01:40:57.953968+0000","last_became_active":"2024-04-26T01:36:54.287728+0000","last_became_peered":"2024-04-26T01:36:54.287728+0000","last_unstale":"2024-04-26T01:40:57.953968+0000","last_undegraded":"2024-04-26T01:40:57.953968+0000","last_fullsized":"2024-04-26T01:40:57.953968+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:18.428151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938879+0000","last_change":"2024-04-26T01:36:50.243522+0000","last_active":"2024-04-26T01:40:57.938879+0000","last_peered":"2024-04-26T01:40:57.938879+0000","last_clean":"2024-04-26T01:40:57.938879+0000","last_became_active":"2024-04-26T01:36:50.243084+0000","last_became_peered":"2024-04-26T01:36:50.243084+0000","last_unstale":"2024-04-26T01:40:57.938879+0000","last_undegraded":"2024-04-26T01:40:57.938879+0000","last_fullsized":"2024-04-26T01:40:57.938879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:03.159750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206382+0000","last_change":"2024-04-26T01:35:13.920486+0000","last_active":"2024-04-26T01:40:58.206382+0000","last_peered":"2024-04-26T01:40:58.206382+0000","last_clean":"2024-04-26T01:40:58.206382+0000","last_became_active":"2024-04-26T01:35:13.919834+0000","last_became_peered":"2024-04-26T01:35:13.919834+0000","last_unstale":"2024-04-26T01:40:58.206382+0000","last_undegraded":"2024-04-26T01:40:58.206382+0000","last_fullsized":"2024-04-26T01:40:58.206382+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_clean_scrub_stamp":"2024-04-26T01:31:12.563114+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:53.935491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949165+0000","last_change":"2024-04-26T01:36:54.270553+0000","last_active":"2024-04-26T01:40:57.949165+0000","last_peered":"2024-04-26T01:40:57.949165+0000","last_clean":"2024-04-26T01:40:57.949165+0000","last_became_active":"2024-04-26T01:36:54.269986+0000","last_became_peered":"2024-04-26T01:36:54.269986+0000","last_unstale":"2024-04-26T01:40:57.949165+0000","last_undegraded":"2024-04-26T01:40:57.949165+0000","last_fullsized":"2024-04-26T01:40:57.949165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:19.597337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206618+0000","last_change":"2024-04-26T01:36:50.235583+0000","last_active":"2024-04-26T01:40:58.206618+0000","last_peered":"2024-04-26T01:40:58.206618+0000","last_clean":"2024-04-26T01:40:58.206618+0000","last_became_active":"2024-04-26T01:36:50.235214+0000","last_became_peered":"2024-04-26T01:36:50.235214+0000","last_unstale":"2024-04-26T01:40:58.206618+0000","last_undegraded":"2024-04-26T01:40:58.206618+0000","last_fullsized":"2024-04-26T01:40:58.206618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:05.822872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079436+0000","last_change":"2024-04-26T01:36:50.260988+0000","last_active":"2024-04-26T01:40:58.079436+0000","last_peered":"2024-04-26T01:40:58.079436+0000","last_clean":"2024-04-26T01:40:58.079436+0000","last_became_active":"2024-04-26T01:36:50.260615+0000","last_became_peered":"2024-04-26T01:36:50.260615+0000","last_unstale":"2024-04-26T01:40:58.079436+0000","last_undegraded":"2024-04-26T01:40:58.079436+0000","last_fullsized":"2024-04-26T01:40:58.079436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:15.933939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949773+0000","last_change":"2024-04-26T01:36:54.279650+0000","last_active":"2024-04-26T01:40:57.949773+0000","last_peered":"2024-04-26T01:40:57.949773+0000","last_clean":"2024-04-26T01:40:57.949773+0000","last_became_active":"2024-04-26T01:36:54.275814+0000","last_became_peered":"2024-04-26T01:36:54.275814+0000","last_unstale":"2024-04-26T01:40:57.949773+0000","last_undegraded":"2024-04-26T01:40:57.949773+0000","last_fullsized":"2024-04-26T01:40:57.949773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:58:42.369511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'48","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.210220+0000","last_change":"2024-04-26T01:36:50.248179+0000","last_active":"2024-04-26T01:40:58.210220+0000","last_peered":"2024-04-26T01:40:58.210220+0000","last_clean":"2024-04-26T01:40:58.210220+0000","last_became_active":"2024-04-26T01:36:50.247805+0000","last_became_peered":"2024-04-26T01:36:50.247805+0000","last_unstale":"2024-04-26T01:40:58.210220+0000","last_undegraded":"2024-04-26T01:40:58.210220+0000","last_fullsized":"2024-04-26T01:40:58.210220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:19:23.740920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954091+0000","last_change":"2024-04-26T01:36:54.276626+0000","last_active":"2024-04-26T01:40:57.954091+0000","last_peered":"2024-04-26T01:40:57.954091+0000","last_clean":"2024-04-26T01:40:57.954091+0000","last_became_active":"2024-04-26T01:36:54.276238+0000","last_became_peered":"2024-04-26T01:36:54.276238+0000","last_unstale":"2024-04-26T01:40:57.954091+0000","last_undegraded":"2024-04-26T01:40:57.954091+0000","last_fullsized":"2024-04-26T01:40:57.954091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:57.306950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939024+0000","last_change":"2024-04-26T01:36:50.278502+0000","last_active":"2024-04-26T01:40:57.939024+0000","last_peered":"2024-04-26T01:40:57.939024+0000","last_clean":"2024-04-26T01:40:57.939024+0000","last_became_active":"2024-04-26T01:36:50.278215+0000","last_became_peered":"2024-04-26T01:36:50.278215+0000","last_unstale":"2024-04-26T01:40:57.939024+0000","last_undegraded":"2024-04-26T01:40:57.939024+0000","last_fullsized":"2024-04-26T01:40:57.939024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:05:46.142902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939164+0000","last_change":"2024-04-26T01:36:54.292600+0000","last_active":"2024-04-26T01:40:57.939164+0000","last_peered":"2024-04-26T01:40:57.939164+0000","last_clean":"2024-04-26T01:40:57.939164+0000","last_became_active":"2024-04-26T01:36:54.292260+0000","last_became_peered":"2024-04-26T01:36:54.292260+0000","last_unstale":"2024-04-26T01:40:57.939164+0000","last_undegraded":"2024-04-26T01:40:57.939164+0000","last_fullsized":"2024-04-26T01:40:57.939164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:10:12.296180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205780+0000","last_change":"2024-04-26T01:36:50.262479+0000","last_active":"2024-04-26T01:40:58.205780+0000","last_peered":"2024-04-26T01:40:58.205780+0000","last_clean":"2024-04-26T01:40:58.205780+0000","last_became_active":"2024-04-26T01:36:50.262109+0000","last_became_peered":"2024-04-26T01:36:50.262109+0000","last_unstale":"2024-04-26T01:40:58.205780+0000","last_undegraded":"2024-04-26T01:40:58.205780+0000","last_fullsized":"2024-04-26T01:40:58.205780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:19.805321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079067+0000","last_change":"2024-04-26T01:36:54.270051+0000","last_active":"2024-04-26T01:40:58.079067+0000","last_peered":"2024-04-26T01:40:58.079067+0000","last_clean":"2024-04-26T01:40:58.079067+0000","last_became_active":"2024-04-26T01:36:54.269670+0000","last_became_peered":"2024-04-26T01:36:54.269670+0000","last_unstale":"2024-04-26T01:40:58.079067+0000","last_undegraded":"2024-04-26T01:40:58.079067+0000","last_fullsized":"2024-04-26T01:40:58.079067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:44.137454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953570+0000","last_change":"2024-04-26T01:36:50.248107+0000","last_active":"2024-04-26T01:40:57.953570+0000","last_peered":"2024-04-26T01:40:57.953570+0000","last_clean":"2024-04-26T01:40:57.953570+0000","last_became_active":"2024-04-26T01:36:50.247695+0000","last_became_peered":"2024-04-26T01:36:50.247695+0000","last_unstale":"2024-04-26T01:40:57.953570+0000","last_undegraded":"2024-04-26T01:40:57.953570+0000","last_fullsized":"2024-04-26T01:40:57.953570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:52.702512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205622+0000","last_change":"2024-04-26T01:36:54.278686+0000","last_active":"2024-04-26T01:40:58.205622+0000","last_peered":"2024-04-26T01:40:58.205622+0000","last_clean":"2024-04-26T01:40:58.205622+0000","last_became_active":"2024-04-26T01:36:54.278312+0000","last_became_peered":"2024-04-26T01:36:54.278312+0000","last_unstale":"2024-04-26T01:40:58.205622+0000","last_undegraded":"2024-04-26T01:40:58.205622+0000","last_fullsized":"2024-04-26T01:40:58.205622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:46.193032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939223+0000","last_change":"2024-04-26T01:36:50.236715+0000","last_active":"2024-04-26T01:40:57.939223+0000","last_peered":"2024-04-26T01:40:57.939223+0000","last_clean":"2024-04-26T01:40:57.939223+0000","last_became_active":"2024-04-26T01:36:50.236184+0000","last_became_peered":"2024-04-26T01:36:50.236184+0000","last_unstale":"2024-04-26T01:40:57.939223+0000","last_undegraded":"2024-04-26T01:40:57.939223+0000","last_fullsized":"2024-04-26T01:40:57.939223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:24:16.063401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954345+0000","last_change":"2024-04-26T01:36:54.290601+0000","last_active":"2024-04-26T01:40:57.954345+0000","last_peered":"2024-04-26T01:40:57.954345+0000","last_clean":"2024-04-26T01:40:57.954345+0000","last_became_active":"2024-04-26T01:36:54.290168+0000","last_became_peered":"2024-04-26T01:36:54.290168+0000","last_unstale":"2024-04-26T01:40:57.954345+0000","last_undegraded":"2024-04-26T01:40:57.954345+0000","last_fullsized":"2024-04-26T01:40:57.954345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:48.770431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079189+0000","last_change":"2024-04-26T01:36:50.235394+0000","last_active":"2024-04-26T01:40:58.079189+0000","last_peered":"2024-04-26T01:40:58.079189+0000","last_clean":"2024-04-26T01:40:58.079189+0000","last_became_active":"2024-04-26T01:36:50.235012+0000","last_became_peered":"2024-04-26T01:36:50.235012+0000","last_unstale":"2024-04-26T01:40:58.079189+0000","last_undegraded":"2024-04-26T01:40:58.079189+0000","last_fullsized":"2024-04-26T01:40:58.079189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:24:17.218945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206668+0000","last_change":"2024-04-26T01:36:54.280490+0000","last_active":"2024-04-26T01:40:58.206668+0000","last_peered":"2024-04-26T01:40:58.206668+0000","last_clean":"2024-04-26T01:40:58.206668+0000","last_became_active":"2024-04-26T01:36:54.280047+0000","last_became_peered":"2024-04-26T01:36:54.280047+0000","last_unstale":"2024-04-26T01:40:58.206668+0000","last_undegraded":"2024-04-26T01:40:58.206668+0000","last_fullsized":"2024-04-26T01:40:58.206668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:32.610972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939636+0000","last_change":"2024-04-26T01:36:50.278325+0000","last_active":"2024-04-26T01:40:57.939636+0000","last_peered":"2024-04-26T01:40:57.939636+0000","last_clean":"2024-04-26T01:40:57.939636+0000","last_became_active":"2024-04-26T01:36:50.277882+0000","last_became_peered":"2024-04-26T01:36:50.277882+0000","last_unstale":"2024-04-26T01:40:57.939636+0000","last_undegraded":"2024-04-26T01:40:57.939636+0000","last_fullsized":"2024-04-26T01:40:57.939636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:06.082223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206962+0000","last_change":"2024-04-26T01:36:54.278106+0000","last_active":"2024-04-26T01:40:58.206962+0000","last_peered":"2024-04-26T01:40:58.206962+0000","last_clean":"2024-04-26T01:40:58.206962+0000","last_became_active":"2024-04-26T01:36:54.277644+0000","last_became_peered":"2024-04-26T01:36:54.277644+0000","last_unstale":"2024-04-26T01:40:58.206962+0000","last_undegraded":"2024-04-26T01:40:58.206962+0000","last_fullsized":"2024-04-26T01:40:58.206962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:10.619640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.207588+0000","last_change":"2024-04-26T01:36:50.237072+0000","last_active":"2024-04-26T01:40:58.207588+0000","last_peered":"2024-04-26T01:40:58.207588+0000","last_clean":"2024-04-26T01:40:58.207588+0000","last_became_active":"2024-04-26T01:36:50.236682+0000","last_became_peered":"2024-04-26T01:36:50.236682+0000","last_unstale":"2024-04-26T01:40:58.207588+0000","last_undegraded":"2024-04-26T01:40:58.207588+0000","last_fullsized":"2024-04-26T01:40:58.207588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:58:17.553492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206902+0000","last_change":"2024-04-26T01:36:54.289997+0000","last_active":"2024-04-26T01:40:58.206902+0000","last_peered":"2024-04-26T01:40:58.206902+0000","last_clean":"2024-04-26T01:40:58.206902+0000","last_became_active":"2024-04-26T01:36:54.289505+0000","last_became_peered":"2024-04-26T01:36:54.289505+0000","last_unstale":"2024-04-26T01:40:58.206902+0000","last_undegraded":"2024-04-26T01:40:58.206902+0000","last_fullsized":"2024-04-26T01:40:58.206902+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:09.340231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.950038+0000","last_change":"2024-04-26T01:36:50.245231+0000","last_active":"2024-04-26T01:40:57.950038+0000","last_peered":"2024-04-26T01:40:57.950038+0000","last_clean":"2024-04-26T01:40:57.950038+0000","last_became_active":"2024-04-26T01:36:50.244811+0000","last_became_peered":"2024-04-26T01:36:50.244811+0000","last_unstale":"2024-04-26T01:40:57.950038+0000","last_undegraded":"2024-04-26T01:40:57.950038+0000","last_fullsized":"2024-04-26T01:40:57.950038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:12:08.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206908+0000","last_change":"2024-04-26T01:36:54.272197+0000","last_active":"2024-04-26T01:40:58.206908+0000","last_peered":"2024-04-26T01:40:58.206908+0000","last_clean":"2024-04-26T01:40:58.206908+0000","last_became_active":"2024-04-26T01:36:54.271776+0000","last_became_peered":"2024-04-26T01:36:54.271776+0000","last_unstale":"2024-04-26T01:40:58.206908+0000","last_undegraded":"2024-04-26T01:40:58.206908+0000","last_fullsized":"2024-04-26T01:40:58.206908+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:20.024890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205725+0000","last_change":"2024-04-26T01:36:54.271120+0000","last_active":"2024-04-26T01:40:58.205725+0000","last_peered":"2024-04-26T01:40:58.205725+0000","last_clean":"2024-04-26T01:40:58.205725+0000","last_became_active":"2024-04-26T01:36:54.270796+0000","last_became_peered":"2024-04-26T01:36:54.270796+0000","last_unstale":"2024-04-26T01:40:58.205725+0000","last_undegraded":"2024-04-26T01:40:58.205725+0000","last_fullsized":"2024-04-26T01:40:58.205725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:04:35.093847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939378+0000","last_change":"2024-04-26T01:36:54.273013+0000","last_active":"2024-04-26T01:40:57.939378+0000","last_peered":"2024-04-26T01:40:57.939378+0000","last_clean":"2024-04-26T01:40:57.939378+0000","last_became_active":"2024-04-26T01:36:54.272666+0000","last_became_peered":"2024-04-26T01:36:54.272666+0000","last_unstale":"2024-04-26T01:40:57.939378+0000","last_undegraded":"2024-04-26T01:40:57.939378+0000","last_fullsized":"2024-04-26T01:40:57.939378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:45.203185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939069+0000","last_change":"2024-04-26T01:36:54.289427+0000","last_active":"2024-04-26T01:40:57.939069+0000","last_peered":"2024-04-26T01:40:57.939069+0000","last_clean":"2024-04-26T01:40:57.939069+0000","last_became_active":"2024-04-26T01:36:54.289090+0000","last_became_peered":"2024-04-26T01:36:54.289090+0000","last_unstale":"2024-04-26T01:40:57.939069+0000","last_undegraded":"2024-04-26T01:40:57.939069+0000","last_fullsized":"2024-04-26T01:40:57.939069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:18:18.544777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938512+0000","last_change":"2024-04-26T01:36:54.294308+0000","last_active":"2024-04-26T01:40:57.938512+0000","last_peered":"2024-04-26T01:40:57.938512+0000","last_clean":"2024-04-26T01:40:57.938512+0000","last_became_active":"2024-04-26T01:36:54.291910+0000","last_became_peered":"2024-04-26T01:36:54.291910+0000","last_unstale":"2024-04-26T01:40:57.938512+0000","last_undegraded":"2024-04-26T01:40:57.938512+0000","last_fullsized":"2024-04-26T01:40:57.938512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:35:16.683265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949562+0000","last_change":"2024-04-26T01:36:54.285479+0000","last_active":"2024-04-26T01:40:57.949562+0000","last_peered":"2024-04-26T01:40:57.949562+0000","last_clean":"2024-04-26T01:40:57.949562+0000","last_became_active":"2024-04-26T01:36:54.284139+0000","last_became_peered":"2024-04-26T01:36:54.284139+0000","last_unstale":"2024-04-26T01:40:57.949562+0000","last_undegraded":"2024-04-26T01:40:57.949562+0000","last_fullsized":"2024-04-26T01:40:57.949562+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:55.506496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205871+0000","last_change":"2024-04-26T01:36:54.270567+0000","last_active":"2024-04-26T01:40:58.205871+0000","last_peered":"2024-04-26T01:40:58.205871+0000","last_clean":"2024-04-26T01:40:58.205871+0000","last_became_active":"2024-04-26T01:36:54.270218+0000","last_became_peered":"2024-04-26T01:36:54.270218+0000","last_unstale":"2024-04-26T01:40:58.205871+0000","last_undegraded":"2024-04-26T01:40:58.205871+0000","last_fullsized":"2024-04-26T01:40:58.205871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:50:43.685086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939427+0000","last_change":"2024-04-26T01:36:54.269746+0000","last_active":"2024-04-26T01:40:57.939427+0000","last_peered":"2024-04-26T01:40:57.939427+0000","last_clean":"2024-04-26T01:40:57.939427+0000","last_became_active":"2024-04-26T01:36:54.269143+0000","last_became_peered":"2024-04-26T01:36:54.269143+0000","last_unstale":"2024-04-26T01:40:57.939427+0000","last_undegraded":"2024-04-26T01:40:57.939427+0000","last_fullsized":"2024-04-26T01:40:57.939427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:07:02.911645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953933+0000","last_change":"2024-04-26T01:36:54.279194+0000","last_active":"2024-04-26T01:40:57.953933+0000","last_peered":"2024-04-26T01:40:57.953933+0000","last_clean":"2024-04-26T01:40:57.953933+0000","last_became_active":"2024-04-26T01:36:54.278709+0000","last_became_peered":"2024-04-26T01:36:54.278709+0000","last_unstale":"2024-04-26T01:40:57.953933+0000","last_undegraded":"2024-04-26T01:40:57.953933+0000","last_fullsized":"2024-04-26T01:40:57.953933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:19.604884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969830+0000","last_change":"2024-04-26T01:36:54.277540+0000","last_active":"2024-04-26T01:40:57.969830+0000","last_peered":"2024-04-26T01:40:57.969830+0000","last_clean":"2024-04-26T01:40:57.969830+0000","last_became_active":"2024-04-26T01:36:54.277193+0000","last_became_peered":"2024-04-26T01:36:54.277193+0000","last_unstale":"2024-04-26T01:40:57.969830+0000","last_undegraded":"2024-04-26T01:40:57.969830+0000","last_fullsized":"2024-04-26T01:40:57.969830+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:34.079094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953565+0000","last_change":"2024-04-26T01:36:54.294593+0000","last_active":"2024-04-26T01:40:57.953565+0000","last_peered":"2024-04-26T01:40:57.953565+0000","last_clean":"2024-04-26T01:40:57.953565+0000","last_became_active":"2024-04-26T01:36:54.291894+0000","last_became_peered":"2024-04-26T01:36:54.291894+0000","last_unstale":"2024-04-26T01:40:57.953565+0000","last_undegraded":"2024-04-26T01:40:57.953565+0000","last_fullsized":"2024-04-26T01:40:57.953565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:16.715108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206030+0000","last_change":"2024-04-26T01:36:54.292433+0000","last_active":"2024-04-26T01:40:58.206030+0000","last_peered":"2024-04-26T01:40:58.206030+0000","last_clean":"2024-04-26T01:40:58.206030+0000","last_became_active":"2024-04-26T01:36:54.291445+0000","last_became_peered":"2024-04-26T01:36:54.291445+0000","last_unstale":"2024-04-26T01:40:58.206030+0000","last_undegraded":"2024-04-26T01:40:58.206030+0000","last_fullsized":"2024-04-26T01:40:58.206030+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:49.081871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949742+0000","last_change":"2024-04-26T01:36:54.291892+0000","last_active":"2024-04-26T01:40:57.949742+0000","last_peered":"2024-04-26T01:40:57.949742+0000","last_clean":"2024-04-26T01:40:57.949742+0000","last_became_active":"2024-04-26T01:36:54.291549+0000","last_became_peered":"2024-04-26T01:36:54.291549+0000","last_unstale":"2024-04-26T01:40:57.949742+0000","last_undegraded":"2024-04-26T01:40:57.949742+0000","last_fullsized":"2024-04-26T01:40:57.949742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:09:22.954651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205662+0000","last_change":"2024-04-26T01:37:25.498138+0000","last_active":"2024-04-26T01:40:58.205662+0000","last_peered":"2024-04-26T01:40:58.205662+0000","last_clean":"2024-04-26T01:40:58.205662+0000","last_became_active":"2024-04-26T01:37:25.497681+0000","last_became_peered":"2024-04-26T01:37:25.497681+0000","last_unstale":"2024-04-26T01:40:58.205662+0000","last_undegraded":"2024-04-26T01:40:58.205662+0000","last_fullsized":"2024-04-26T01:40:58.205662+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:03:25.016623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107964,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600964,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1453,"num_read_kb":840,"num_write":298,"num_write_kb":1444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1802892,"data_compressed":81390,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383747,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488830,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645383,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36484,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708668,"statfs":{"total":95995035648,"available":95957676032,"internally_reserved":0,"allocated":1171456,"data_stored":1283113,"data_compressed":46861,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808910,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":882260,"data_compressed":22035,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070547,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":882158,"data_compressed":22035,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332185,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1056768,"data_stored":1082127,"data_compressed":38877,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488433,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":561076,"data_compressed":4717,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":489704,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1073152,"data_stored":559365,"data_compressed":4717,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182511,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1013560,"data_compressed":28191,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":489292,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757350,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1213101,"data_compressed":45033,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":72850,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71530,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134390,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":204402,"data_compressed":7984,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:41:04.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:41:04.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:04.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 821 B/s wr, 4 op/s 2024-04-26T01:41:04.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:04.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:04.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:04.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:04.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:04 smithi053 ceph-mon[32593]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:05.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:05.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:05.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:05.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:05.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 821 B/s wr, 4 op/s 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:05.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:05.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:04 smithi028 ceph-mon[28247]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:05.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:05.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:05.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:05.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Updating smithi169:/etc/ceph/ceph.conf 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 821 B/s wr, 4 op/s 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:05.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:05.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:05.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:04 smithi169 ceph-mon[32851]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:41:05.404 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:06.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:06 smithi028 ceph-mon[28247]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:06.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:06 smithi028 ceph-mon[28247]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:06.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:06 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:06 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:06 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:06 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:06 smithi028 ceph-mon[28247]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:06.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:06 smithi169 ceph-mon[32851]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:06.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:06 smithi169 ceph-mon[32851]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:06.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:06 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:06 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:06 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:06 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:06 smithi169 ceph-mon[32851]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:06.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:06 smithi053 ceph-mon[32593]: Updating smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:06.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:06 smithi053 ceph-mon[32593]: Updating smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:06.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:06 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:06 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:06 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:06 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:06.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:06 smithi053 ceph-mon[32593]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:41:07.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:07.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 169 KiB/s wr, 101 op/s 2024-04-26T01:41:07.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:07.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:07.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:07.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:07 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:07.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 169 KiB/s wr, 101 op/s 2024-04-26T01:41:07.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:07.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:07.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:07.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:07 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 -- 172.21.15.28:0/3390343597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c105870 msgr2=0x7f653c1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 --2- 172.21.15.28:0/3390343597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c105870 0x7f653c1098c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6528009920 tx=0x7f6528009c30 comp rx=0 tx=0).stop 2024-04-26T01:41:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 -- 172.21.15.28:0/3390343597 shutdown_connections 2024-04-26T01:41:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 --2- 172.21.15.28:0/3390343597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c109fa0 0x7f653c111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 --2- 172.21.15.28:0/3390343597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f653c104ec0 0x7f653c1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 --2- 172.21.15.28:0/3390343597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c105870 0x7f653c1098c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 -- 172.21.15.28:0/3390343597 >> 172.21.15.28:0/3390343597 conn(0x7f653c100750 msgr2=0x7f653c102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 -- 172.21.15.28:0/3390343597 shutdown_connections 2024-04-26T01:41:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.709+0000 7f6542800700 1 -- 172.21.15.28:0/3390343597 wait complete. 2024-04-26T01:41:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.711+0000 7f6542800700 1 Processor -- start 2024-04-26T01:41:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.711+0000 7f6542800700 1 -- start start 2024-04-26T01:41:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f6542800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c104ec0 0x7f653c19b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f6542800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c105870 0x7f653c19b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f6542800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f653c109fa0 0x7f653c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f6542800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f653c114e60 con 0x7f653c105870 2024-04-26T01:41:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f6542800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f653c114ce0 con 0x7f653c109fa0 2024-04-26T01:41:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f6542800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f653c114fe0 con 0x7f653c104ec0 2024-04-26T01:41:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f653aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c105870 0x7f653c19b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f653b400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c104ec0 0x7f653c19b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f653aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c105870 0x7f653c19b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39238/0 (socket says 172.21.15.28:39238) 2024-04-26T01:41:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f653aa00700 1 -- 172.21.15.28:0/3546342361 learned_addr learned my addr 172.21.15.28:0/3546342361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.712+0000 7f653be00700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f653c109fa0 0x7f653c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f653aa00700 1 -- 172.21.15.28:0/3546342361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c104ec0 msgr2=0x7f653c19b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f653aa00700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c104ec0 0x7f653c19b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f653aa00700 1 -- 172.21.15.28:0/3546342361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f653c109fa0 msgr2=0x7f653c19fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f653aa00700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f653c109fa0 0x7f653c19fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f653aa00700 1 -- 172.21.15.28:0/3546342361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c1a03f0 con 0x7f653c105870 2024-04-26T01:41:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f653b400700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c104ec0 0x7f653c19b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f653aa00700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c105870 0x7f653c19b970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6528000c00 tx=0x7f652800f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f6533e00700 1 -- 172.21.15.28:0/3546342361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528005040 con 0x7f653c105870 2024-04-26T01:41:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f6533e00700 1 -- 172.21.15.28:0/3546342361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6528005690 con 0x7f653c105870 2024-04-26T01:41:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653c1a0680 con 0x7f653c105870 2024-04-26T01:41:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.713+0000 7f6533e00700 1 -- 172.21.15.28:0/3546342361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65280038e0 con 0x7f653c105870 2024-04-26T01:41:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.714+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f653c1a7f60 con 0x7f653c105870 2024-04-26T01:41:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.715+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f653c1063a0 con 0x7f653c105870 2024-04-26T01:41:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.717+0000 7f6533e00700 1 -- 172.21.15.28:0/3546342361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f652800b760 con 0x7f653c105870 2024-04-26T01:41:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.718+0000 7f6533e00700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f65200a3f50 0x7f65200a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.718+0000 7f6533e00700 1 -- 172.21.15.28:0/3546342361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f65280d1aa0 con 0x7f653c105870 2024-04-26T01:41:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.719+0000 7f653b400700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f65200a3f50 0x7f65200a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.719+0000 7f653b400700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f65200a3f50 0x7f65200a6410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f652c009680 tx=0x7f652c009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.721+0000 7f6533e00700 1 -- 172.21.15.28:0/3546342361 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f652805eff0 con 0x7f653c105870 2024-04-26T01:41:07.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: Updating smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:41:07.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 169 KiB/s wr, 101 op/s 2024-04-26T01:41:07.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:07.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:07.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:07.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:07 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:41:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.953+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f653c062e80 con 0x7f65200a3f50 2024-04-26T01:41:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.976+0000 7f6533e00700 1 -- 172.21.15.28:0/3546342361 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258065 (secure 0 0 0) 0x7f653c062e80 con 0x7f65200a3f50 2024-04-26T01:41:07.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:07.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f65200a3f50 msgr2=0x7f65200a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f65200a3f50 0x7f65200a6410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f652c009680 tx=0x7f652c009260 comp rx=0 tx=0).stop 2024-04-26T01:41:07.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c105870 msgr2=0x7f653c19b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c105870 0x7f653c19b970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6528000c00 tx=0x7f652800f950 comp rx=0 tx=0).stop 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 shutdown_connections 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f65200a3f50 0x7f65200a6410 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f653c109fa0 0x7f653c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f653c104ec0 0x7f653c19b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 --2- 172.21.15.28:0/3546342361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c105870 0x7f653c19b970 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.982+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 >> 172.21.15.28:0/3546342361 conn(0x7f653c100750 msgr2=0x7f653c101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.983+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 shutdown_connections 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:07.983+0000 7f6542800700 1 -- 172.21.15.28:0/3546342361 wait complete. 2024-04-26T01:41:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-26T01:41:08.923 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-04-26T01:41:05.966425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1390111,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":2095,"num_read_kb":1529,"num_write":686,"num_write_kb":5565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181492,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":999,"ondisk_log_size":999,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378196,"kb_used_data":12372,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563628,"statfs":{"total":1151940427776,"available":1151553155072,"internally_reserved":0,"allocated":12668928,"data_stored":10029045,"data_compressed":251610,"data_compressed_allocated":331776,"data_compressed_original":3993600,"omap_allocated":19140,"internal_metadata":374453564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":198545,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":506,"num_read_kb":570,"num_write":205,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.983597"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206116+0000","last_change":"2024-04-26T01:36:54.288460+0000","last_active":"2024-04-26T01:40:58.206116+0000","last_peered":"2024-04-26T01:40:58.206116+0000","last_clean":"2024-04-26T01:40:58.206116+0000","last_became_active":"2024-04-26T01:36:54.288033+0000","last_became_peered":"2024-04-26T01:36:54.288033+0000","last_unstale":"2024-04-26T01:40:58.206116+0000","last_undegraded":"2024-04-26T01:40:58.206116+0000","last_fullsized":"2024-04-26T01:40:58.206116+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:33.737622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938949+0000","last_change":"2024-04-26T01:36:54.287367+0000","last_active":"2024-04-26T01:40:57.938949+0000","last_peered":"2024-04-26T01:40:57.938949+0000","last_clean":"2024-04-26T01:40:57.938949+0000","last_became_active":"2024-04-26T01:36:54.287024+0000","last_became_peered":"2024-04-26T01:36:54.287024+0000","last_unstale":"2024-04-26T01:40:57.938949+0000","last_undegraded":"2024-04-26T01:40:57.938949+0000","last_fullsized":"2024-04-26T01:40:57.938949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:31.014627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938539+0000","last_change":"2024-04-26T01:36:54.288433+0000","last_active":"2024-04-26T01:40:57.938539+0000","last_peered":"2024-04-26T01:40:57.938539+0000","last_clean":"2024-04-26T01:40:57.938539+0000","last_became_active":"2024-04-26T01:36:54.288002+0000","last_became_peered":"2024-04-26T01:36:54.288002+0000","last_unstale":"2024-04-26T01:40:57.938539+0000","last_undegraded":"2024-04-26T01:40:57.938539+0000","last_fullsized":"2024-04-26T01:40:57.938539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:37.885179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939129+0000","last_change":"2024-04-26T01:36:54.273857+0000","last_active":"2024-04-26T01:40:57.939129+0000","last_peered":"2024-04-26T01:40:57.939129+0000","last_clean":"2024-04-26T01:40:57.939129+0000","last_became_active":"2024-04-26T01:36:54.272376+0000","last_became_peered":"2024-04-26T01:36:54.272376+0000","last_unstale":"2024-04-26T01:40:57.939129+0000","last_undegraded":"2024-04-26T01:40:57.939129+0000","last_fullsized":"2024-04-26T01:40:57.939129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:30.110044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079196+0000","last_change":"2024-04-26T01:36:54.288606+0000","last_active":"2024-04-26T01:40:58.079196+0000","last_peered":"2024-04-26T01:40:58.079196+0000","last_clean":"2024-04-26T01:40:58.079196+0000","last_became_active":"2024-04-26T01:36:54.288247+0000","last_became_peered":"2024-04-26T01:36:54.288247+0000","last_unstale":"2024-04-26T01:40:58.079196+0000","last_undegraded":"2024-04-26T01:40:58.079196+0000","last_fullsized":"2024-04-26T01:40:58.079196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:47:56.944164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967023+0000","last_change":"2024-04-26T01:36:54.277701+0000","last_active":"2024-04-26T01:40:57.967023+0000","last_peered":"2024-04-26T01:40:57.967023+0000","last_clean":"2024-04-26T01:40:57.967023+0000","last_became_active":"2024-04-26T01:36:54.277306+0000","last_became_peered":"2024-04-26T01:36:54.277306+0000","last_unstale":"2024-04-26T01:40:57.967023+0000","last_undegraded":"2024-04-26T01:40:57.967023+0000","last_fullsized":"2024-04-26T01:40:57.967023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:42:05.058525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206390+0000","last_change":"2024-04-26T01:36:54.274090+0000","last_active":"2024-04-26T01:40:58.206390+0000","last_peered":"2024-04-26T01:40:58.206390+0000","last_clean":"2024-04-26T01:40:58.206390+0000","last_became_active":"2024-04-26T01:36:54.273700+0000","last_became_peered":"2024-04-26T01:36:54.273700+0000","last_unstale":"2024-04-26T01:40:58.206390+0000","last_undegraded":"2024-04-26T01:40:58.206390+0000","last_fullsized":"2024-04-26T01:40:58.206390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:32.207534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076612+0000","last_change":"2024-04-26T01:37:25.499759+0000","last_active":"2024-04-26T01:40:58.076612+0000","last_peered":"2024-04-26T01:40:58.076612+0000","last_clean":"2024-04-26T01:40:58.076612+0000","last_became_active":"2024-04-26T01:37:25.499317+0000","last_became_peered":"2024-04-26T01:37:25.499317+0000","last_unstale":"2024-04-26T01:40:58.076612+0000","last_undegraded":"2024-04-26T01:40:58.076612+0000","last_fullsized":"2024-04-26T01:40:58.076612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:39.972315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206270+0000","last_change":"2024-04-26T01:36:54.277457+0000","last_active":"2024-04-26T01:40:58.206270+0000","last_peered":"2024-04-26T01:40:58.206270+0000","last_clean":"2024-04-26T01:40:58.206270+0000","last_became_active":"2024-04-26T01:36:54.277118+0000","last_became_peered":"2024-04-26T01:36:54.277118+0000","last_unstale":"2024-04-26T01:40:58.206270+0000","last_undegraded":"2024-04-26T01:40:58.206270+0000","last_fullsized":"2024-04-26T01:40:58.206270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:05.628027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967353+0000","last_change":"2024-04-26T01:36:54.278494+0000","last_active":"2024-04-26T01:40:57.967353+0000","last_peered":"2024-04-26T01:40:57.967353+0000","last_clean":"2024-04-26T01:40:57.967353+0000","last_became_active":"2024-04-26T01:36:54.278188+0000","last_became_peered":"2024-04-26T01:36:54.278188+0000","last_unstale":"2024-04-26T01:40:57.967353+0000","last_undegraded":"2024-04-26T01:40:57.967353+0000","last_fullsized":"2024-04-26T01:40:57.967353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:59.150771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.207126+0000","last_change":"2024-04-26T01:36:54.277424+0000","last_active":"2024-04-26T01:40:58.207126+0000","last_peered":"2024-04-26T01:40:58.207126+0000","last_clean":"2024-04-26T01:40:58.207126+0000","last_became_active":"2024-04-26T01:36:54.275752+0000","last_became_peered":"2024-04-26T01:36:54.275752+0000","last_unstale":"2024-04-26T01:40:58.207126+0000","last_undegraded":"2024-04-26T01:40:58.207126+0000","last_fullsized":"2024-04-26T01:40:58.207126+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:22.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076889+0000","last_change":"2024-04-26T01:37:25.502448+0000","last_active":"2024-04-26T01:40:58.076889+0000","last_peered":"2024-04-26T01:40:58.076889+0000","last_clean":"2024-04-26T01:40:58.076889+0000","last_became_active":"2024-04-26T01:37:25.501161+0000","last_became_peered":"2024-04-26T01:37:25.501161+0000","last_unstale":"2024-04-26T01:40:58.076889+0000","last_undegraded":"2024-04-26T01:40:58.076889+0000","last_fullsized":"2024-04-26T01:40:58.076889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:41:40.443780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949697+0000","last_change":"2024-04-26T01:36:54.268721+0000","last_active":"2024-04-26T01:40:57.949697+0000","last_peered":"2024-04-26T01:40:57.949697+0000","last_clean":"2024-04-26T01:40:57.949697+0000","last_became_active":"2024-04-26T01:36:54.268007+0000","last_became_peered":"2024-04-26T01:36:54.268007+0000","last_unstale":"2024-04-26T01:40:57.949697+0000","last_undegraded":"2024-04-26T01:40:57.949697+0000","last_fullsized":"2024-04-26T01:40:57.949697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:55.120130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938165+0000","last_change":"2024-04-26T01:36:54.286595+0000","last_active":"2024-04-26T01:40:57.938165+0000","last_peered":"2024-04-26T01:40:57.938165+0000","last_clean":"2024-04-26T01:40:57.938165+0000","last_became_active":"2024-04-26T01:36:54.286236+0000","last_became_peered":"2024-04-26T01:36:54.286236+0000","last_unstale":"2024-04-26T01:40:57.938165+0000","last_undegraded":"2024-04-26T01:40:57.938165+0000","last_fullsized":"2024-04-26T01:40:57.938165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:59.443467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938105+0000","last_change":"2024-04-26T01:36:54.293254+0000","last_active":"2024-04-26T01:40:57.938105+0000","last_peered":"2024-04-26T01:40:57.938105+0000","last_clean":"2024-04-26T01:40:57.938105+0000","last_became_active":"2024-04-26T01:36:54.292853+0000","last_became_peered":"2024-04-26T01:36:54.292853+0000","last_unstale":"2024-04-26T01:40:57.938105+0000","last_undegraded":"2024-04-26T01:40:57.938105+0000","last_fullsized":"2024-04-26T01:40:57.938105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:19.844006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205747+0000","last_change":"2024-04-26T01:36:54.285346+0000","last_active":"2024-04-26T01:40:58.205747+0000","last_peered":"2024-04-26T01:40:58.205747+0000","last_clean":"2024-04-26T01:40:58.205747+0000","last_became_active":"2024-04-26T01:36:54.284949+0000","last_became_peered":"2024-04-26T01:36:54.284949+0000","last_unstale":"2024-04-26T01:40:58.205747+0000","last_undegraded":"2024-04-26T01:40:58.205747+0000","last_fullsized":"2024-04-26T01:40:58.205747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:33:29.276240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205959+0000","last_change":"2024-04-26T01:36:54.283161+0000","last_active":"2024-04-26T01:40:58.205959+0000","last_peered":"2024-04-26T01:40:58.205959+0000","last_clean":"2024-04-26T01:40:58.205959+0000","last_became_active":"2024-04-26T01:36:54.282908+0000","last_became_peered":"2024-04-26T01:36:54.282908+0000","last_unstale":"2024-04-26T01:40:58.205959+0000","last_undegraded":"2024-04-26T01:40:58.205959+0000","last_fullsized":"2024-04-26T01:40:58.205959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:26.503964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.077086+0000","last_change":"2024-04-26T01:36:54.282516+0000","last_active":"2024-04-26T01:40:58.077086+0000","last_peered":"2024-04-26T01:40:58.077086+0000","last_clean":"2024-04-26T01:40:58.077086+0000","last_became_active":"2024-04-26T01:36:54.278634+0000","last_became_peered":"2024-04-26T01:36:54.278634+0000","last_unstale":"2024-04-26T01:40:58.077086+0000","last_undegraded":"2024-04-26T01:40:58.077086+0000","last_fullsized":"2024-04-26T01:40:58.077086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:10:38.275720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939505+0000","last_change":"2024-04-26T01:36:54.289502+0000","last_active":"2024-04-26T01:40:57.939505+0000","last_peered":"2024-04-26T01:40:57.939505+0000","last_clean":"2024-04-26T01:40:57.939505+0000","last_became_active":"2024-04-26T01:36:54.289079+0000","last_became_peered":"2024-04-26T01:36:54.289079+0000","last_unstale":"2024-04-26T01:40:57.939505+0000","last_undegraded":"2024-04-26T01:40:57.939505+0000","last_fullsized":"2024-04-26T01:40:57.939505+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:51.412032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076838+0000","last_change":"2024-04-26T01:36:50.243164+0000","last_active":"2024-04-26T01:40:58.076838+0000","last_peered":"2024-04-26T01:40:58.076838+0000","last_clean":"2024-04-26T01:40:58.076838+0000","last_became_active":"2024-04-26T01:36:50.242508+0000","last_became_peered":"2024-04-26T01:36:50.242508+0000","last_unstale":"2024-04-26T01:40:58.076838+0000","last_undegraded":"2024-04-26T01:40:58.076838+0000","last_fullsized":"2024-04-26T01:40:58.076838+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:06:01.662698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954129+0000","last_change":"2024-04-26T01:36:54.279809+0000","last_active":"2024-04-26T01:40:57.954129+0000","last_peered":"2024-04-26T01:40:57.954129+0000","last_clean":"2024-04-26T01:40:57.954129+0000","last_became_active":"2024-04-26T01:36:54.279509+0000","last_became_peered":"2024-04-26T01:36:54.279509+0000","last_unstale":"2024-04-26T01:40:57.954129+0000","last_undegraded":"2024-04-26T01:40:57.954129+0000","last_fullsized":"2024-04-26T01:40:57.954129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:54:26.734136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954751+0000","last_change":"2024-04-26T01:36:50.261942+0000","last_active":"2024-04-26T01:40:57.954751+0000","last_peered":"2024-04-26T01:40:57.954751+0000","last_clean":"2024-04-26T01:40:57.954751+0000","last_became_active":"2024-04-26T01:36:50.261564+0000","last_became_peered":"2024-04-26T01:36:50.261564+0000","last_unstale":"2024-04-26T01:40:57.954751+0000","last_undegraded":"2024-04-26T01:40:57.954751+0000","last_fullsized":"2024-04-26T01:40:57.954751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:25.102557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938783+0000","last_change":"2024-04-26T01:36:54.288011+0000","last_active":"2024-04-26T01:40:57.938783+0000","last_peered":"2024-04-26T01:40:57.938783+0000","last_clean":"2024-04-26T01:40:57.938783+0000","last_became_active":"2024-04-26T01:36:54.286853+0000","last_became_peered":"2024-04-26T01:36:54.286853+0000","last_unstale":"2024-04-26T01:40:57.938783+0000","last_undegraded":"2024-04-26T01:40:57.938783+0000","last_fullsized":"2024-04-26T01:40:57.938783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:53.422470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079561+0000","last_change":"2024-04-26T01:36:50.235291+0000","last_active":"2024-04-26T01:40:58.079561+0000","last_peered":"2024-04-26T01:40:58.079561+0000","last_clean":"2024-04-26T01:40:58.079561+0000","last_became_active":"2024-04-26T01:36:50.234939+0000","last_became_peered":"2024-04-26T01:36:50.234939+0000","last_unstale":"2024-04-26T01:40:58.079561+0000","last_undegraded":"2024-04-26T01:40:58.079561+0000","last_fullsized":"2024-04-26T01:40:58.079561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:13.048185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205932+0000","last_change":"2024-04-26T01:36:54.282731+0000","last_active":"2024-04-26T01:40:58.205932+0000","last_peered":"2024-04-26T01:40:58.205932+0000","last_clean":"2024-04-26T01:40:58.205932+0000","last_became_active":"2024-04-26T01:36:54.282384+0000","last_became_peered":"2024-04-26T01:36:54.282384+0000","last_unstale":"2024-04-26T01:40:58.205932+0000","last_undegraded":"2024-04-26T01:40:58.205932+0000","last_fullsized":"2024-04-26T01:40:58.205932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:09.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079407+0000","last_change":"2024-04-26T01:36:50.233581+0000","last_active":"2024-04-26T01:40:58.079407+0000","last_peered":"2024-04-26T01:40:58.079407+0000","last_clean":"2024-04-26T01:40:58.079407+0000","last_became_active":"2024-04-26T01:36:50.233210+0000","last_became_peered":"2024-04-26T01:36:50.233210+0000","last_unstale":"2024-04-26T01:40:58.079407+0000","last_undegraded":"2024-04-26T01:40:58.079407+0000","last_fullsized":"2024-04-26T01:40:58.079407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:01.223954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970312+0000","last_change":"2024-04-26T01:36:54.274657+0000","last_active":"2024-04-26T01:40:57.970312+0000","last_peered":"2024-04-26T01:40:57.970312+0000","last_clean":"2024-04-26T01:40:57.970312+0000","last_became_active":"2024-04-26T01:36:54.274333+0000","last_became_peered":"2024-04-26T01:36:54.274333+0000","last_unstale":"2024-04-26T01:40:57.970312+0000","last_undegraded":"2024-04-26T01:40:57.970312+0000","last_fullsized":"2024-04-26T01:40:57.970312+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:37.243307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":281,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:01.979917+0000","last_change":"2024-04-26T01:37:25.502311+0000","last_active":"2024-04-26T01:41:01.979917+0000","last_peered":"2024-04-26T01:41:01.979917+0000","last_clean":"2024-04-26T01:41:01.979917+0000","last_became_active":"2024-04-26T01:37:25.493259+0000","last_became_peered":"2024-04-26T01:37:25.493259+0000","last_unstale":"2024-04-26T01:41:01.979917+0000","last_undegraded":"2024-04-26T01:41:01.979917+0000","last_fullsized":"2024-04-26T01:41:01.979917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:46:49.224905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":228,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079111+0000","last_change":"2024-04-26T01:36:54.275116+0000","last_active":"2024-04-26T01:40:58.079111+0000","last_peered":"2024-04-26T01:40:58.079111+0000","last_clean":"2024-04-26T01:40:58.079111+0000","last_became_active":"2024-04-26T01:36:54.274815+0000","last_became_peered":"2024-04-26T01:36:54.274815+0000","last_unstale":"2024-04-26T01:40:58.079111+0000","last_undegraded":"2024-04-26T01:40:58.079111+0000","last_fullsized":"2024-04-26T01:40:58.079111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:04.427670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.966940+0000","last_change":"2024-04-26T01:37:25.505327+0000","last_active":"2024-04-26T01:40:57.966940+0000","last_peered":"2024-04-26T01:40:57.966940+0000","last_clean":"2024-04-26T01:40:57.966940+0000","last_became_active":"2024-04-26T01:37:25.497881+0000","last_became_peered":"2024-04-26T01:37:25.497881+0000","last_unstale":"2024-04-26T01:40:57.966940+0000","last_undegraded":"2024-04-26T01:40:57.966940+0000","last_fullsized":"2024-04-26T01:40:57.966940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:09.181372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970138+0000","last_change":"2024-04-26T01:36:54.276415+0000","last_active":"2024-04-26T01:40:57.970138+0000","last_peered":"2024-04-26T01:40:57.970138+0000","last_clean":"2024-04-26T01:40:57.970138+0000","last_became_active":"2024-04-26T01:36:54.276080+0000","last_became_peered":"2024-04-26T01:36:54.276080+0000","last_unstale":"2024-04-26T01:40:57.970138+0000","last_undegraded":"2024-04-26T01:40:57.970138+0000","last_fullsized":"2024-04-26T01:40:57.970138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:08:17.171620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949572+0000","last_change":"2024-04-26T01:36:50.241552+0000","last_active":"2024-04-26T01:40:57.949572+0000","last_peered":"2024-04-26T01:40:57.949572+0000","last_clean":"2024-04-26T01:40:57.949572+0000","last_became_active":"2024-04-26T01:36:50.241022+0000","last_became_peered":"2024-04-26T01:36:50.241022+0000","last_unstale":"2024-04-26T01:40:57.949572+0000","last_undegraded":"2024-04-26T01:40:57.949572+0000","last_fullsized":"2024-04-26T01:40:57.949572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:55.503831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953427+0000","last_change":"2024-04-26T01:36:54.276641+0000","last_active":"2024-04-26T01:40:57.953427+0000","last_peered":"2024-04-26T01:40:57.953427+0000","last_clean":"2024-04-26T01:40:57.953427+0000","last_became_active":"2024-04-26T01:36:54.276309+0000","last_became_peered":"2024-04-26T01:36:54.276309+0000","last_unstale":"2024-04-26T01:40:57.953427+0000","last_undegraded":"2024-04-26T01:40:57.953427+0000","last_fullsized":"2024-04-26T01:40:57.953427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:53.725584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969805+0000","last_change":"2024-04-26T01:36:50.235504+0000","last_active":"2024-04-26T01:40:57.969805+0000","last_peered":"2024-04-26T01:40:57.969805+0000","last_clean":"2024-04-26T01:40:57.969805+0000","last_became_active":"2024-04-26T01:36:50.235161+0000","last_became_peered":"2024-04-26T01:36:50.235161+0000","last_unstale":"2024-04-26T01:40:57.969805+0000","last_undegraded":"2024-04-26T01:40:57.969805+0000","last_fullsized":"2024-04-26T01:40:57.969805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:05.951521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076712+0000","last_change":"2024-04-26T01:36:54.298236+0000","last_active":"2024-04-26T01:40:58.076712+0000","last_peered":"2024-04-26T01:40:58.076712+0000","last_clean":"2024-04-26T01:40:58.076712+0000","last_became_active":"2024-04-26T01:36:54.297850+0000","last_became_peered":"2024-04-26T01:36:54.297850+0000","last_unstale":"2024-04-26T01:40:58.076712+0000","last_undegraded":"2024-04-26T01:40:58.076712+0000","last_fullsized":"2024-04-26T01:40:58.076712+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:35:35.244502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":260,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:01.978966+0000","last_change":"2024-04-26T01:36:50.241418+0000","last_active":"2024-04-26T01:41:01.978966+0000","last_peered":"2024-04-26T01:41:01.978966+0000","last_clean":"2024-04-26T01:41:01.978966+0000","last_became_active":"2024-04-26T01:36:50.241044+0000","last_became_peered":"2024-04-26T01:36:50.241044+0000","last_unstale":"2024-04-26T01:41:01.978966+0000","last_undegraded":"2024-04-26T01:41:01.978966+0000","last_fullsized":"2024-04-26T01:41:01.978966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:13.317953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206340+0000","last_change":"2024-04-26T01:36:54.284410+0000","last_active":"2024-04-26T01:40:58.206340+0000","last_peered":"2024-04-26T01:40:58.206340+0000","last_clean":"2024-04-26T01:40:58.206340+0000","last_became_active":"2024-04-26T01:36:54.283629+0000","last_became_peered":"2024-04-26T01:36:54.283629+0000","last_unstale":"2024-04-26T01:40:58.206340+0000","last_undegraded":"2024-04-26T01:40:58.206340+0000","last_fullsized":"2024-04-26T01:40:58.206340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:15.380006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205831+0000","last_change":"2024-04-26T01:37:25.499874+0000","last_active":"2024-04-26T01:40:58.205831+0000","last_peered":"2024-04-26T01:40:58.205831+0000","last_clean":"2024-04-26T01:40:58.205831+0000","last_became_active":"2024-04-26T01:37:25.494878+0000","last_became_peered":"2024-04-26T01:37:25.494878+0000","last_unstale":"2024-04-26T01:40:58.205831+0000","last_undegraded":"2024-04-26T01:40:58.205831+0000","last_fullsized":"2024-04-26T01:40:58.205831+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:48:12.678331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938888+0000","last_change":"2024-04-26T01:36:54.285118+0000","last_active":"2024-04-26T01:40:57.938888+0000","last_peered":"2024-04-26T01:40:57.938888+0000","last_clean":"2024-04-26T01:40:57.938888+0000","last_became_active":"2024-04-26T01:36:54.284554+0000","last_became_peered":"2024-04-26T01:36:54.284554+0000","last_unstale":"2024-04-26T01:40:57.938888+0000","last_undegraded":"2024-04-26T01:40:57.938888+0000","last_fullsized":"2024-04-26T01:40:57.938888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:13.721684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970272+0000","last_change":"2024-04-26T01:36:50.262451+0000","last_active":"2024-04-26T01:40:57.970272+0000","last_peered":"2024-04-26T01:40:57.970272+0000","last_clean":"2024-04-26T01:40:57.970272+0000","last_became_active":"2024-04-26T01:36:50.262056+0000","last_became_peered":"2024-04-26T01:36:50.262056+0000","last_unstale":"2024-04-26T01:40:57.970272+0000","last_undegraded":"2024-04-26T01:40:57.970272+0000","last_fullsized":"2024-04-26T01:40:57.970272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:50.813675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206136+0000","last_change":"2024-04-26T01:36:54.283975+0000","last_active":"2024-04-26T01:40:58.206136+0000","last_peered":"2024-04-26T01:40:58.206136+0000","last_clean":"2024-04-26T01:40:58.206136+0000","last_became_active":"2024-04-26T01:36:54.283564+0000","last_became_peered":"2024-04-26T01:36:54.283564+0000","last_unstale":"2024-04-26T01:40:58.206136+0000","last_undegraded":"2024-04-26T01:40:58.206136+0000","last_fullsized":"2024-04-26T01:40:58.206136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:53.881763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'43","reported_seq":540,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:03.809192+0000","last_change":"2024-04-26T01:36:50.238839+0000","last_active":"2024-04-26T01:41:03.809192+0000","last_peered":"2024-04-26T01:41:03.809192+0000","last_clean":"2024-04-26T01:41:03.809192+0000","last_became_active":"2024-04-26T01:36:50.238452+0000","last_became_peered":"2024-04-26T01:36:50.238452+0000","last_unstale":"2024-04-26T01:41:03.809192+0000","last_undegraded":"2024-04-26T01:41:03.809192+0000","last_fullsized":"2024-04-26T01:41:03.809192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:05:40.426825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72436,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":468,"num_read_kb":305,"num_write":44,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079361+0000","last_change":"2024-04-26T01:36:54.271446+0000","last_active":"2024-04-26T01:40:58.079361+0000","last_peered":"2024-04-26T01:40:58.079361+0000","last_clean":"2024-04-26T01:40:58.079361+0000","last_became_active":"2024-04-26T01:36:54.271122+0000","last_became_peered":"2024-04-26T01:36:54.271122+0000","last_unstale":"2024-04-26T01:40:58.079361+0000","last_undegraded":"2024-04-26T01:40:58.079361+0000","last_fullsized":"2024-04-26T01:40:58.079361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:28:24.275481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939266+0000","last_change":"2024-04-26T01:36:50.237726+0000","last_active":"2024-04-26T01:40:57.939266+0000","last_peered":"2024-04-26T01:40:57.939266+0000","last_clean":"2024-04-26T01:40:57.939266+0000","last_became_active":"2024-04-26T01:36:50.237215+0000","last_became_peered":"2024-04-26T01:36:50.237215+0000","last_unstale":"2024-04-26T01:40:57.939266+0000","last_undegraded":"2024-04-26T01:40:57.939266+0000","last_fullsized":"2024-04-26T01:40:57.939266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:41.781876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949425+0000","last_change":"2024-04-26T01:36:54.290616+0000","last_active":"2024-04-26T01:40:57.949425+0000","last_peered":"2024-04-26T01:40:57.949425+0000","last_clean":"2024-04-26T01:40:57.949425+0000","last_became_active":"2024-04-26T01:36:54.290293+0000","last_became_peered":"2024-04-26T01:36:54.290293+0000","last_unstale":"2024-04-26T01:40:57.949425+0000","last_undegraded":"2024-04-26T01:40:57.949425+0000","last_fullsized":"2024-04-26T01:40:57.949425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:07.439333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969982+0000","last_change":"2024-04-26T01:36:50.237578+0000","last_active":"2024-04-26T01:40:57.969982+0000","last_peered":"2024-04-26T01:40:57.969982+0000","last_clean":"2024-04-26T01:40:57.969982+0000","last_became_active":"2024-04-26T01:36:50.237224+0000","last_became_peered":"2024-04-26T01:36:50.237224+0000","last_unstale":"2024-04-26T01:40:57.969982+0000","last_undegraded":"2024-04-26T01:40:57.969982+0000","last_fullsized":"2024-04-26T01:40:57.969982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:46:38.078631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206709+0000","last_change":"2024-04-26T01:36:54.280818+0000","last_active":"2024-04-26T01:40:58.206709+0000","last_peered":"2024-04-26T01:40:58.206709+0000","last_clean":"2024-04-26T01:40:58.206709+0000","last_became_active":"2024-04-26T01:36:54.280440+0000","last_became_peered":"2024-04-26T01:36:54.280440+0000","last_unstale":"2024-04-26T01:40:58.206709+0000","last_undegraded":"2024-04-26T01:40:58.206709+0000","last_fullsized":"2024-04-26T01:40:58.206709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:47.185219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":264,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:01.970955+0000","last_change":"2024-04-26T01:36:50.234602+0000","last_active":"2024-04-26T01:41:01.970955+0000","last_peered":"2024-04-26T01:41:01.970955+0000","last_clean":"2024-04-26T01:41:01.970955+0000","last_became_active":"2024-04-26T01:36:50.234175+0000","last_became_peered":"2024-04-26T01:36:50.234175+0000","last_unstale":"2024-04-26T01:41:01.970955+0000","last_undegraded":"2024-04-26T01:41:01.970955+0000","last_fullsized":"2024-04-26T01:41:01.970955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:17:14.236069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1289,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":231,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206181+0000","last_change":"2024-04-26T01:36:54.271466+0000","last_active":"2024-04-26T01:40:58.206181+0000","last_peered":"2024-04-26T01:40:58.206181+0000","last_clean":"2024-04-26T01:40:58.206181+0000","last_became_active":"2024-04-26T01:36:54.271086+0000","last_became_peered":"2024-04-26T01:36:54.271086+0000","last_unstale":"2024-04-26T01:40:58.206181+0000","last_undegraded":"2024-04-26T01:40:58.206181+0000","last_fullsized":"2024-04-26T01:40:58.206181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:41.052895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079039+0000","last_change":"2024-04-26T01:37:25.518730+0000","last_active":"2024-04-26T01:40:58.079039+0000","last_peered":"2024-04-26T01:40:58.079039+0000","last_clean":"2024-04-26T01:40:58.079039+0000","last_became_active":"2024-04-26T01:37:25.504464+0000","last_became_peered":"2024-04-26T01:37:25.504464+0000","last_unstale":"2024-04-26T01:40:58.079039+0000","last_undegraded":"2024-04-26T01:40:58.079039+0000","last_fullsized":"2024-04-26T01:40:58.079039+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:07.578917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969886+0000","last_change":"2024-04-26T01:36:54.273621+0000","last_active":"2024-04-26T01:40:57.969886+0000","last_peered":"2024-04-26T01:40:57.969886+0000","last_clean":"2024-04-26T01:40:57.969886+0000","last_became_active":"2024-04-26T01:36:54.273218+0000","last_became_peered":"2024-04-26T01:36:54.273218+0000","last_unstale":"2024-04-26T01:40:57.969886+0000","last_undegraded":"2024-04-26T01:40:57.969886+0000","last_fullsized":"2024-04-26T01:40:57.969886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:55.543943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205908+0000","last_change":"2024-04-26T01:37:25.508575+0000","last_active":"2024-04-26T01:40:58.205908+0000","last_peered":"2024-04-26T01:40:58.205908+0000","last_clean":"2024-04-26T01:40:58.205908+0000","last_became_active":"2024-04-26T01:37:25.496407+0000","last_became_peered":"2024-04-26T01:37:25.496407+0000","last_unstale":"2024-04-26T01:40:58.205908+0000","last_undegraded":"2024-04-26T01:40:58.205908+0000","last_fullsized":"2024-04-26T01:40:58.205908+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:05.878252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205907+0000","last_change":"2024-04-26T01:36:54.272147+0000","last_active":"2024-04-26T01:40:58.205907+0000","last_peered":"2024-04-26T01:40:58.205907+0000","last_clean":"2024-04-26T01:40:58.205907+0000","last_became_active":"2024-04-26T01:36:54.271737+0000","last_became_peered":"2024-04-26T01:36:54.271737+0000","last_unstale":"2024-04-26T01:40:58.205907+0000","last_undegraded":"2024-04-26T01:40:58.205907+0000","last_fullsized":"2024-04-26T01:40:58.205907+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:41.445073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076978+0000","last_change":"2024-04-26T01:36:50.264444+0000","last_active":"2024-04-26T01:40:58.076978+0000","last_peered":"2024-04-26T01:40:58.076978+0000","last_clean":"2024-04-26T01:40:58.076978+0000","last_became_active":"2024-04-26T01:36:50.264058+0000","last_became_peered":"2024-04-26T01:36:50.264058+0000","last_unstale":"2024-04-26T01:40:58.076978+0000","last_undegraded":"2024-04-26T01:40:58.076978+0000","last_fullsized":"2024-04-26T01:40:58.076978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:56:15.428588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967226+0000","last_change":"2024-04-26T01:36:54.289537+0000","last_active":"2024-04-26T01:40:57.967226+0000","last_peered":"2024-04-26T01:40:57.967226+0000","last_clean":"2024-04-26T01:40:57.967226+0000","last_became_active":"2024-04-26T01:36:54.289149+0000","last_became_peered":"2024-04-26T01:36:54.289149+0000","last_unstale":"2024-04-26T01:40:57.967226+0000","last_undegraded":"2024-04-26T01:40:57.967226+0000","last_fullsized":"2024-04-26T01:40:57.967226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:07.801551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206728+0000","last_change":"2024-04-26T01:36:50.242190+0000","last_active":"2024-04-26T01:40:58.206728+0000","last_peered":"2024-04-26T01:40:58.206728+0000","last_clean":"2024-04-26T01:40:58.206728+0000","last_became_active":"2024-04-26T01:36:50.241890+0000","last_became_peered":"2024-04-26T01:36:50.241890+0000","last_unstale":"2024-04-26T01:40:58.206728+0000","last_undegraded":"2024-04-26T01:40:58.206728+0000","last_fullsized":"2024-04-26T01:40:58.206728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:29.472502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967093+0000","last_change":"2024-04-26T01:36:54.294699+0000","last_active":"2024-04-26T01:40:57.967093+0000","last_peered":"2024-04-26T01:40:57.967093+0000","last_clean":"2024-04-26T01:40:57.967093+0000","last_became_active":"2024-04-26T01:36:54.294313+0000","last_became_peered":"2024-04-26T01:36:54.294313+0000","last_unstale":"2024-04-26T01:40:57.967093+0000","last_undegraded":"2024-04-26T01:40:57.967093+0000","last_fullsized":"2024-04-26T01:40:57.967093+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:15.026357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970178+0000","last_change":"2024-04-26T01:36:54.276798+0000","last_active":"2024-04-26T01:40:57.970178+0000","last_peered":"2024-04-26T01:40:57.970178+0000","last_clean":"2024-04-26T01:40:57.970178+0000","last_became_active":"2024-04-26T01:36:54.275955+0000","last_became_peered":"2024-04-26T01:36:54.275955+0000","last_unstale":"2024-04-26T01:40:57.970178+0000","last_undegraded":"2024-04-26T01:40:57.970178+0000","last_fullsized":"2024-04-26T01:40:57.970178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:17.020909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":262,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:03.973630+0000","last_change":"2024-04-26T01:36:50.263356+0000","last_active":"2024-04-26T01:41:03.973630+0000","last_peered":"2024-04-26T01:41:03.973630+0000","last_clean":"2024-04-26T01:41:03.973630+0000","last_became_active":"2024-04-26T01:36:50.261596+0000","last_became_peered":"2024-04-26T01:36:50.261596+0000","last_unstale":"2024-04-26T01:41:03.973630+0000","last_undegraded":"2024-04-26T01:41:03.973630+0000","last_fullsized":"2024-04-26T01:41:03.973630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:10.773347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":227,"num_read_kb":227,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954042+0000","last_change":"2024-04-26T01:36:54.288135+0000","last_active":"2024-04-26T01:40:57.954042+0000","last_peered":"2024-04-26T01:40:57.954042+0000","last_clean":"2024-04-26T01:40:57.954042+0000","last_became_active":"2024-04-26T01:36:54.287728+0000","last_became_peered":"2024-04-26T01:36:54.287728+0000","last_unstale":"2024-04-26T01:40:57.954042+0000","last_undegraded":"2024-04-26T01:40:57.954042+0000","last_fullsized":"2024-04-26T01:40:57.954042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:18.428151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938946+0000","last_change":"2024-04-26T01:36:50.243522+0000","last_active":"2024-04-26T01:40:57.938946+0000","last_peered":"2024-04-26T01:40:57.938946+0000","last_clean":"2024-04-26T01:40:57.938946+0000","last_became_active":"2024-04-26T01:36:50.243084+0000","last_became_peered":"2024-04-26T01:36:50.243084+0000","last_unstale":"2024-04-26T01:40:57.938946+0000","last_undegraded":"2024-04-26T01:40:57.938946+0000","last_fullsized":"2024-04-26T01:40:57.938946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:03.159750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'148","reported_seq":406,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:00.093944+0000","last_change":"2024-04-26T01:35:13.920486+0000","last_active":"2024-04-26T01:41:00.093944+0000","last_peered":"2024-04-26T01:41:00.093944+0000","last_clean":"2024-04-26T01:41:00.093944+0000","last_became_active":"2024-04-26T01:35:13.919834+0000","last_became_peered":"2024-04-26T01:35:13.919834+0000","last_unstale":"2024-04-26T01:41:00.093944+0000","last_undegraded":"2024-04-26T01:41:00.093944+0000","last_fullsized":"2024-04-26T01:41:00.093944+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_clean_scrub_stamp":"2024-04-26T01:31:12.563114+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:53.935491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949264+0000","last_change":"2024-04-26T01:36:54.270553+0000","last_active":"2024-04-26T01:40:57.949264+0000","last_peered":"2024-04-26T01:40:57.949264+0000","last_clean":"2024-04-26T01:40:57.949264+0000","last_became_active":"2024-04-26T01:36:54.269986+0000","last_became_peered":"2024-04-26T01:36:54.269986+0000","last_unstale":"2024-04-26T01:40:57.949264+0000","last_undegraded":"2024-04-26T01:40:57.949264+0000","last_fullsized":"2024-04-26T01:40:57.949264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:19.597337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206618+0000","last_change":"2024-04-26T01:36:50.235583+0000","last_active":"2024-04-26T01:40:58.206618+0000","last_peered":"2024-04-26T01:40:58.206618+0000","last_clean":"2024-04-26T01:40:58.206618+0000","last_became_active":"2024-04-26T01:36:50.235214+0000","last_became_peered":"2024-04-26T01:36:50.235214+0000","last_unstale":"2024-04-26T01:40:58.206618+0000","last_undegraded":"2024-04-26T01:40:58.206618+0000","last_fullsized":"2024-04-26T01:40:58.206618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:05.822872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079498+0000","last_change":"2024-04-26T01:36:50.260988+0000","last_active":"2024-04-26T01:40:58.079498+0000","last_peered":"2024-04-26T01:40:58.079498+0000","last_clean":"2024-04-26T01:40:58.079498+0000","last_became_active":"2024-04-26T01:36:50.260615+0000","last_became_peered":"2024-04-26T01:36:50.260615+0000","last_unstale":"2024-04-26T01:40:58.079498+0000","last_undegraded":"2024-04-26T01:40:58.079498+0000","last_fullsized":"2024-04-26T01:40:58.079498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:15.933939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949847+0000","last_change":"2024-04-26T01:36:54.279650+0000","last_active":"2024-04-26T01:40:57.949847+0000","last_peered":"2024-04-26T01:40:57.949847+0000","last_clean":"2024-04-26T01:40:57.949847+0000","last_became_active":"2024-04-26T01:36:54.275814+0000","last_became_peered":"2024-04-26T01:36:54.275814+0000","last_unstale":"2024-04-26T01:40:57.949847+0000","last_undegraded":"2024-04-26T01:40:57.949847+0000","last_fullsized":"2024-04-26T01:40:57.949847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:58:42.369511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'125","reported_seq":279,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:00.730699+0000","last_change":"2024-04-26T01:36:50.248179+0000","last_active":"2024-04-26T01:41:00.730699+0000","last_peered":"2024-04-26T01:41:00.730699+0000","last_clean":"2024-04-26T01:41:00.730699+0000","last_became_active":"2024-04-26T01:36:50.247805+0000","last_became_peered":"2024-04-26T01:36:50.247805+0000","last_unstale":"2024-04-26T01:41:00.730699+0000","last_undegraded":"2024-04-26T01:41:00.730699+0000","last_fullsized":"2024-04-26T01:41:00.730699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:19:23.740920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":422,"num_read_kb":376,"num_write":231,"num_write_kb":1414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954091+0000","last_change":"2024-04-26T01:36:54.276626+0000","last_active":"2024-04-26T01:40:57.954091+0000","last_peered":"2024-04-26T01:40:57.954091+0000","last_clean":"2024-04-26T01:40:57.954091+0000","last_became_active":"2024-04-26T01:36:54.276238+0000","last_became_peered":"2024-04-26T01:36:54.276238+0000","last_unstale":"2024-04-26T01:40:57.954091+0000","last_undegraded":"2024-04-26T01:40:57.954091+0000","last_fullsized":"2024-04-26T01:40:57.954091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:57.306950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939024+0000","last_change":"2024-04-26T01:36:50.278502+0000","last_active":"2024-04-26T01:40:57.939024+0000","last_peered":"2024-04-26T01:40:57.939024+0000","last_clean":"2024-04-26T01:40:57.939024+0000","last_became_active":"2024-04-26T01:36:50.278215+0000","last_became_peered":"2024-04-26T01:36:50.278215+0000","last_unstale":"2024-04-26T01:40:57.939024+0000","last_undegraded":"2024-04-26T01:40:57.939024+0000","last_fullsized":"2024-04-26T01:40:57.939024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:05:46.142902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939164+0000","last_change":"2024-04-26T01:36:54.292600+0000","last_active":"2024-04-26T01:40:57.939164+0000","last_peered":"2024-04-26T01:40:57.939164+0000","last_clean":"2024-04-26T01:40:57.939164+0000","last_became_active":"2024-04-26T01:36:54.292260+0000","last_became_peered":"2024-04-26T01:36:54.292260+0000","last_unstale":"2024-04-26T01:40:57.939164+0000","last_undegraded":"2024-04-26T01:40:57.939164+0000","last_fullsized":"2024-04-26T01:40:57.939164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:10:12.296180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205780+0000","last_change":"2024-04-26T01:36:50.262479+0000","last_active":"2024-04-26T01:40:58.205780+0000","last_peered":"2024-04-26T01:40:58.205780+0000","last_clean":"2024-04-26T01:40:58.205780+0000","last_became_active":"2024-04-26T01:36:50.262109+0000","last_became_peered":"2024-04-26T01:36:50.262109+0000","last_unstale":"2024-04-26T01:40:58.205780+0000","last_undegraded":"2024-04-26T01:40:58.205780+0000","last_fullsized":"2024-04-26T01:40:58.205780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:19.805321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079067+0000","last_change":"2024-04-26T01:36:54.270051+0000","last_active":"2024-04-26T01:40:58.079067+0000","last_peered":"2024-04-26T01:40:58.079067+0000","last_clean":"2024-04-26T01:40:58.079067+0000","last_became_active":"2024-04-26T01:36:54.269670+0000","last_became_peered":"2024-04-26T01:36:54.269670+0000","last_unstale":"2024-04-26T01:40:58.079067+0000","last_undegraded":"2024-04-26T01:40:58.079067+0000","last_fullsized":"2024-04-26T01:40:58.079067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:44.137454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953570+0000","last_change":"2024-04-26T01:36:50.248107+0000","last_active":"2024-04-26T01:40:57.953570+0000","last_peered":"2024-04-26T01:40:57.953570+0000","last_clean":"2024-04-26T01:40:57.953570+0000","last_became_active":"2024-04-26T01:36:50.247695+0000","last_became_peered":"2024-04-26T01:36:50.247695+0000","last_unstale":"2024-04-26T01:40:57.953570+0000","last_undegraded":"2024-04-26T01:40:57.953570+0000","last_fullsized":"2024-04-26T01:40:57.953570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:52.702512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205622+0000","last_change":"2024-04-26T01:36:54.278686+0000","last_active":"2024-04-26T01:40:58.205622+0000","last_peered":"2024-04-26T01:40:58.205622+0000","last_clean":"2024-04-26T01:40:58.205622+0000","last_became_active":"2024-04-26T01:36:54.278312+0000","last_became_peered":"2024-04-26T01:36:54.278312+0000","last_unstale":"2024-04-26T01:40:58.205622+0000","last_undegraded":"2024-04-26T01:40:58.205622+0000","last_fullsized":"2024-04-26T01:40:58.205622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:46.193032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939223+0000","last_change":"2024-04-26T01:36:50.236715+0000","last_active":"2024-04-26T01:40:57.939223+0000","last_peered":"2024-04-26T01:40:57.939223+0000","last_clean":"2024-04-26T01:40:57.939223+0000","last_became_active":"2024-04-26T01:36:50.236184+0000","last_became_peered":"2024-04-26T01:36:50.236184+0000","last_unstale":"2024-04-26T01:40:57.939223+0000","last_undegraded":"2024-04-26T01:40:57.939223+0000","last_fullsized":"2024-04-26T01:40:57.939223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:24:16.063401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954345+0000","last_change":"2024-04-26T01:36:54.290601+0000","last_active":"2024-04-26T01:40:57.954345+0000","last_peered":"2024-04-26T01:40:57.954345+0000","last_clean":"2024-04-26T01:40:57.954345+0000","last_became_active":"2024-04-26T01:36:54.290168+0000","last_became_peered":"2024-04-26T01:36:54.290168+0000","last_unstale":"2024-04-26T01:40:57.954345+0000","last_undegraded":"2024-04-26T01:40:57.954345+0000","last_fullsized":"2024-04-26T01:40:57.954345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:48.770431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079189+0000","last_change":"2024-04-26T01:36:50.235394+0000","last_active":"2024-04-26T01:40:58.079189+0000","last_peered":"2024-04-26T01:40:58.079189+0000","last_clean":"2024-04-26T01:40:58.079189+0000","last_became_active":"2024-04-26T01:36:50.235012+0000","last_became_peered":"2024-04-26T01:36:50.235012+0000","last_unstale":"2024-04-26T01:40:58.079189+0000","last_undegraded":"2024-04-26T01:40:58.079189+0000","last_fullsized":"2024-04-26T01:40:58.079189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:24:17.218945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206668+0000","last_change":"2024-04-26T01:36:54.280490+0000","last_active":"2024-04-26T01:40:58.206668+0000","last_peered":"2024-04-26T01:40:58.206668+0000","last_clean":"2024-04-26T01:40:58.206668+0000","last_became_active":"2024-04-26T01:36:54.280047+0000","last_became_peered":"2024-04-26T01:36:54.280047+0000","last_unstale":"2024-04-26T01:40:58.206668+0000","last_undegraded":"2024-04-26T01:40:58.206668+0000","last_fullsized":"2024-04-26T01:40:58.206668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:32.610972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939636+0000","last_change":"2024-04-26T01:36:50.278325+0000","last_active":"2024-04-26T01:40:57.939636+0000","last_peered":"2024-04-26T01:40:57.939636+0000","last_clean":"2024-04-26T01:40:57.939636+0000","last_became_active":"2024-04-26T01:36:50.277882+0000","last_became_peered":"2024-04-26T01:36:50.277882+0000","last_unstale":"2024-04-26T01:40:57.939636+0000","last_undegraded":"2024-04-26T01:40:57.939636+0000","last_fullsized":"2024-04-26T01:40:57.939636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:06.082223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206962+0000","last_change":"2024-04-26T01:36:54.278106+0000","last_active":"2024-04-26T01:40:58.206962+0000","last_peered":"2024-04-26T01:40:58.206962+0000","last_clean":"2024-04-26T01:40:58.206962+0000","last_became_active":"2024-04-26T01:36:54.277644+0000","last_became_peered":"2024-04-26T01:36:54.277644+0000","last_unstale":"2024-04-26T01:40:58.206962+0000","last_undegraded":"2024-04-26T01:40:58.206962+0000","last_fullsized":"2024-04-26T01:40:58.206962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:10.619640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'73","reported_seq":144,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:00.725122+0000","last_change":"2024-04-26T01:36:50.237072+0000","last_active":"2024-04-26T01:41:00.725122+0000","last_peered":"2024-04-26T01:41:00.725122+0000","last_clean":"2024-04-26T01:41:00.725122+0000","last_became_active":"2024-04-26T01:36:50.236682+0000","last_became_peered":"2024-04-26T01:36:50.236682+0000","last_unstale":"2024-04-26T01:41:00.725122+0000","last_undegraded":"2024-04-26T01:41:00.725122+0000","last_fullsized":"2024-04-26T01:41:00.725122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:58:17.553492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":44,"num_write":84,"num_write_kb":1075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206902+0000","last_change":"2024-04-26T01:36:54.289997+0000","last_active":"2024-04-26T01:40:58.206902+0000","last_peered":"2024-04-26T01:40:58.206902+0000","last_clean":"2024-04-26T01:40:58.206902+0000","last_became_active":"2024-04-26T01:36:54.289505+0000","last_became_peered":"2024-04-26T01:36:54.289505+0000","last_unstale":"2024-04-26T01:40:58.206902+0000","last_undegraded":"2024-04-26T01:40:58.206902+0000","last_fullsized":"2024-04-26T01:40:58.206902+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:09.340231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.950038+0000","last_change":"2024-04-26T01:36:50.245231+0000","last_active":"2024-04-26T01:40:57.950038+0000","last_peered":"2024-04-26T01:40:57.950038+0000","last_clean":"2024-04-26T01:40:57.950038+0000","last_became_active":"2024-04-26T01:36:50.244811+0000","last_became_peered":"2024-04-26T01:36:50.244811+0000","last_unstale":"2024-04-26T01:40:57.950038+0000","last_undegraded":"2024-04-26T01:40:57.950038+0000","last_fullsized":"2024-04-26T01:40:57.950038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:12:08.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206908+0000","last_change":"2024-04-26T01:36:54.272197+0000","last_active":"2024-04-26T01:40:58.206908+0000","last_peered":"2024-04-26T01:40:58.206908+0000","last_clean":"2024-04-26T01:40:58.206908+0000","last_became_active":"2024-04-26T01:36:54.271776+0000","last_became_peered":"2024-04-26T01:36:54.271776+0000","last_unstale":"2024-04-26T01:40:58.206908+0000","last_undegraded":"2024-04-26T01:40:58.206908+0000","last_fullsized":"2024-04-26T01:40:58.206908+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:20.024890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205725+0000","last_change":"2024-04-26T01:36:54.271120+0000","last_active":"2024-04-26T01:40:58.205725+0000","last_peered":"2024-04-26T01:40:58.205725+0000","last_clean":"2024-04-26T01:40:58.205725+0000","last_became_active":"2024-04-26T01:36:54.270796+0000","last_became_peered":"2024-04-26T01:36:54.270796+0000","last_unstale":"2024-04-26T01:40:58.205725+0000","last_undegraded":"2024-04-26T01:40:58.205725+0000","last_fullsized":"2024-04-26T01:40:58.205725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:04:35.093847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939378+0000","last_change":"2024-04-26T01:36:54.273013+0000","last_active":"2024-04-26T01:40:57.939378+0000","last_peered":"2024-04-26T01:40:57.939378+0000","last_clean":"2024-04-26T01:40:57.939378+0000","last_became_active":"2024-04-26T01:36:54.272666+0000","last_became_peered":"2024-04-26T01:36:54.272666+0000","last_unstale":"2024-04-26T01:40:57.939378+0000","last_undegraded":"2024-04-26T01:40:57.939378+0000","last_fullsized":"2024-04-26T01:40:57.939378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:45.203185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939069+0000","last_change":"2024-04-26T01:36:54.289427+0000","last_active":"2024-04-26T01:40:57.939069+0000","last_peered":"2024-04-26T01:40:57.939069+0000","last_clean":"2024-04-26T01:40:57.939069+0000","last_became_active":"2024-04-26T01:36:54.289090+0000","last_became_peered":"2024-04-26T01:36:54.289090+0000","last_unstale":"2024-04-26T01:40:57.939069+0000","last_undegraded":"2024-04-26T01:40:57.939069+0000","last_fullsized":"2024-04-26T01:40:57.939069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:18:18.544777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938512+0000","last_change":"2024-04-26T01:36:54.294308+0000","last_active":"2024-04-26T01:40:57.938512+0000","last_peered":"2024-04-26T01:40:57.938512+0000","last_clean":"2024-04-26T01:40:57.938512+0000","last_became_active":"2024-04-26T01:36:54.291910+0000","last_became_peered":"2024-04-26T01:36:54.291910+0000","last_unstale":"2024-04-26T01:40:57.938512+0000","last_undegraded":"2024-04-26T01:40:57.938512+0000","last_fullsized":"2024-04-26T01:40:57.938512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:35:16.683265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949562+0000","last_change":"2024-04-26T01:36:54.285479+0000","last_active":"2024-04-26T01:40:57.949562+0000","last_peered":"2024-04-26T01:40:57.949562+0000","last_clean":"2024-04-26T01:40:57.949562+0000","last_became_active":"2024-04-26T01:36:54.284139+0000","last_became_peered":"2024-04-26T01:36:54.284139+0000","last_unstale":"2024-04-26T01:40:57.949562+0000","last_undegraded":"2024-04-26T01:40:57.949562+0000","last_fullsized":"2024-04-26T01:40:57.949562+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:55.506496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205871+0000","last_change":"2024-04-26T01:36:54.270567+0000","last_active":"2024-04-26T01:40:58.205871+0000","last_peered":"2024-04-26T01:40:58.205871+0000","last_clean":"2024-04-26T01:40:58.205871+0000","last_became_active":"2024-04-26T01:36:54.270218+0000","last_became_peered":"2024-04-26T01:36:54.270218+0000","last_unstale":"2024-04-26T01:40:58.205871+0000","last_undegraded":"2024-04-26T01:40:58.205871+0000","last_fullsized":"2024-04-26T01:40:58.205871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:50:43.685086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939427+0000","last_change":"2024-04-26T01:36:54.269746+0000","last_active":"2024-04-26T01:40:57.939427+0000","last_peered":"2024-04-26T01:40:57.939427+0000","last_clean":"2024-04-26T01:40:57.939427+0000","last_became_active":"2024-04-26T01:36:54.269143+0000","last_became_peered":"2024-04-26T01:36:54.269143+0000","last_unstale":"2024-04-26T01:40:57.939427+0000","last_undegraded":"2024-04-26T01:40:57.939427+0000","last_fullsized":"2024-04-26T01:40:57.939427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:07:02.911645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953933+0000","last_change":"2024-04-26T01:36:54.279194+0000","last_active":"2024-04-26T01:40:57.953933+0000","last_peered":"2024-04-26T01:40:57.953933+0000","last_clean":"2024-04-26T01:40:57.953933+0000","last_became_active":"2024-04-26T01:36:54.278709+0000","last_became_peered":"2024-04-26T01:36:54.278709+0000","last_unstale":"2024-04-26T01:40:57.953933+0000","last_undegraded":"2024-04-26T01:40:57.953933+0000","last_fullsized":"2024-04-26T01:40:57.953933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:19.604884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969830+0000","last_change":"2024-04-26T01:36:54.277540+0000","last_active":"2024-04-26T01:40:57.969830+0000","last_peered":"2024-04-26T01:40:57.969830+0000","last_clean":"2024-04-26T01:40:57.969830+0000","last_became_active":"2024-04-26T01:36:54.277193+0000","last_became_peered":"2024-04-26T01:36:54.277193+0000","last_unstale":"2024-04-26T01:40:57.969830+0000","last_undegraded":"2024-04-26T01:40:57.969830+0000","last_fullsized":"2024-04-26T01:40:57.969830+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:34.079094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953565+0000","last_change":"2024-04-26T01:36:54.294593+0000","last_active":"2024-04-26T01:40:57.953565+0000","last_peered":"2024-04-26T01:40:57.953565+0000","last_clean":"2024-04-26T01:40:57.953565+0000","last_became_active":"2024-04-26T01:36:54.291894+0000","last_became_peered":"2024-04-26T01:36:54.291894+0000","last_unstale":"2024-04-26T01:40:57.953565+0000","last_undegraded":"2024-04-26T01:40:57.953565+0000","last_fullsized":"2024-04-26T01:40:57.953565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:16.715108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206030+0000","last_change":"2024-04-26T01:36:54.292433+0000","last_active":"2024-04-26T01:40:58.206030+0000","last_peered":"2024-04-26T01:40:58.206030+0000","last_clean":"2024-04-26T01:40:58.206030+0000","last_became_active":"2024-04-26T01:36:54.291445+0000","last_became_peered":"2024-04-26T01:36:54.291445+0000","last_unstale":"2024-04-26T01:40:58.206030+0000","last_undegraded":"2024-04-26T01:40:58.206030+0000","last_fullsized":"2024-04-26T01:40:58.206030+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:49.081871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949742+0000","last_change":"2024-04-26T01:36:54.291892+0000","last_active":"2024-04-26T01:40:57.949742+0000","last_peered":"2024-04-26T01:40:57.949742+0000","last_clean":"2024-04-26T01:40:57.949742+0000","last_became_active":"2024-04-26T01:36:54.291549+0000","last_became_peered":"2024-04-26T01:36:54.291549+0000","last_unstale":"2024-04-26T01:40:57.949742+0000","last_undegraded":"2024-04-26T01:40:57.949742+0000","last_fullsized":"2024-04-26T01:40:57.949742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:09:22.954651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205662+0000","last_change":"2024-04-26T01:37:25.498138+0000","last_active":"2024-04-26T01:40:58.205662+0000","last_peered":"2024-04-26T01:40:58.205662+0000","last_clean":"2024-04-26T01:40:58.205662+0000","last_became_active":"2024-04-26T01:37:25.497681+0000","last_became_peered":"2024-04-26T01:37:25.497681+0000","last_unstale":"2024-04-26T01:40:58.205662+0000","last_undegraded":"2024-04-26T01:40:58.205662+0000","last_fullsized":"2024-04-26T01:40:58.205662+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:03:25.016623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107964,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799509,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1858,"num_read_kb":1199,"num_write":483,"num_write_kb":2625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2398527,"data_compressed":108978,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383748,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488830,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645384,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36512,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708640,"statfs":{"total":95995035648,"available":95957647360,"internally_reserved":0,"allocated":1196032,"data_stored":1481658,"data_compressed":55907,"data_compressed_allocated":73728,"data_compressed_original":913408,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808911,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":882260,"data_compressed":22185,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070548,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":882158,"data_compressed":22185,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332186,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":1082518,"data_compressed":38877,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495711,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488433,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":562598,"data_compressed":4717,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":489704,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411449,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":561278,"data_compressed":4717,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182512,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":1210192,"data_compressed":37387,"data_compressed_allocated":49152,"data_compressed_original":708608,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":489683,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757351,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":1409733,"data_compressed":54079,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":74372,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":73052,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34498,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":331022,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19296,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19296,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":402947,"data_compressed":17030,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:41:08.927 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:41:08.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:41:09.444 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:09.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:09 smithi028 ceph-mon[28247]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:09.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:09 smithi028 ceph-mon[28247]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 131 KiB/s wr, 79 op/s 2024-04-26T01:41:09.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:09 smithi169 ceph-mon[32851]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:09.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:09 smithi169 ceph-mon[32851]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 131 KiB/s wr, 79 op/s 2024-04-26T01:41:09.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:09 smithi053 ceph-mon[32593]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:09.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:09 smithi053 ceph-mon[32593]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 131 KiB/s wr, 79 op/s 2024-04-26T01:41:11.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 -- 172.21.15.28:0/3470469274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c06b0f0 msgr2=0x7f671c10f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:11.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 --2- 172.21.15.28:0/3470469274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c06b0f0 0x7f671c10f900 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f671400b0a0 tx=0x7f671400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 -- 172.21.15.28:0/3470469274 shutdown_connections 2024-04-26T01:41:11.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 --2- 172.21.15.28:0/3470469274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c06b0f0 0x7f671c10f900 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 --2- 172.21.15.28:0/3470469274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f671c108a30 0x7f671c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 --2- 172.21.15.28:0/3470469274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671c06a750 0x7f671c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 -- 172.21.15.28:0/3470469274 >> 172.21.15.28:0/3470469274 conn(0x7f671c0788f0 msgr2=0x7f671c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:11.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:11 smithi028 ceph-mon[28247]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 108 KiB/s wr, 67 op/s 2024-04-26T01:41:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 -- 172.21.15.28:0/3470469274 shutdown_connections 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.659+0000 7f6722800700 1 -- 172.21.15.28:0/3470469274 wait complete. 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.660+0000 7f6722800700 1 Processor -- start 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f6722800700 1 -- start start 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f6722800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f671c06a750 0x7f671c1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f6722800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671c06b0f0 0x7f671c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f6722800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c108a30 0x7f671c19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f6722800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f671c112c20 con 0x7f671c06b0f0 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f6722800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f671c112aa0 con 0x7f671c06a750 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f6722800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f671c112da0 con 0x7f671c108a30 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f671be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c108a30 0x7f671c19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f6712a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671c06b0f0 0x7f671c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f671be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c108a30 0x7f671c19d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54850/0 (socket says 172.21.15.28:54850) 2024-04-26T01:41:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f671be00700 1 -- 172.21.15.28:0/1844791313 learned_addr learned my addr 172.21.15.28:0/1844791313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.661+0000 7f671b400700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f671c06a750 0x7f671c1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.662+0000 7f671be00700 1 -- 172.21.15.28:0/1844791313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f671c06a750 msgr2=0x7f671c1990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.662+0000 7f671be00700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f671c06a750 0x7f671c1990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.662+0000 7f671be00700 1 -- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671c06b0f0 msgr2=0x7f671c199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.662+0000 7f671be00700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671c06b0f0 0x7f671c199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.662+0000 7f671be00700 1 -- 172.21.15.28:0/1844791313 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671c19e100 con 0x7f671c108a30 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.662+0000 7f671be00700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c108a30 0x7f671c19d9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f671400b070 tx=0x7f6714004cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.663+0000 7f6718c00700 1 -- 172.21.15.28:0/1844791313 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6714021070 con 0x7f671c108a30 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.663+0000 7f6718c00700 1 -- 172.21.15.28:0/1844791313 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6714007e30 con 0x7f671c108a30 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.663+0000 7f6718c00700 1 -- 172.21.15.28:0/1844791313 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67140046d0 con 0x7f671c108a30 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.663+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671c19e390 con 0x7f671c108a30 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.663+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f671c1a5c70 con 0x7f671c108a30 2024-04-26T01:41:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.664+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f671c069730 con 0x7f671c108a30 2024-04-26T01:41:11.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:11 smithi169 ceph-mon[32851]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 108 KiB/s wr, 67 op/s 2024-04-26T01:41:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.666+0000 7f6718c00700 1 -- 172.21.15.28:0/1844791313 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f6714052080 con 0x7f671c108a30 2024-04-26T01:41:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.667+0000 7f6718c00700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67000a3f50 0x7f67000a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.667+0000 7f6718c00700 1 -- 172.21.15.28:0/1844791313 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f671400f020 con 0x7f671c108a30 2024-04-26T01:41:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.668+0000 7f671b400700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67000a3f50 0x7f67000a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.668+0000 7f671b400700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67000a3f50 0x7f67000a6410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f670c0045b0 tx=0x7f670c00b3c0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.670+0000 7f6718c00700 1 -- 172.21.15.28:0/1844791313 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f671402a020 con 0x7f671c108a30 2024-04-26T01:41:11.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:11 smithi053 ceph-mon[32593]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 108 KiB/s wr, 67 op/s 2024-04-26T01:41:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.897+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f671c19a450 con 0x7f67000a3f50 2024-04-26T01:41:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.920+0000 7f6718c00700 1 -- 172.21.15.28:0/1844791313 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258068 (secure 0 0 0) 0x7f671c19a450 con 0x7f67000a3f50 2024-04-26T01:41:11.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67000a3f50 msgr2=0x7f67000a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67000a3f50 0x7f67000a6410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f670c0045b0 tx=0x7f670c00b3c0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c108a30 msgr2=0x7f671c19d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c108a30 0x7f671c19d9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f671400b070 tx=0x7f6714004cf0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 shutdown_connections 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67000a3f50 0x7f67000a6410 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c108a30 0x7f671c19d9c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f671c06a750 0x7f671c1990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.926+0000 7f6722800700 1 --2- 172.21.15.28:0/1844791313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671c06b0f0 0x7f671c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.927+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 >> 172.21.15.28:0/1844791313 conn(0x7f671c0788f0 msgr2=0x7f671c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.927+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 shutdown_connections 2024-04-26T01:41:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:11.927+0000 7f6722800700 1 -- 172.21.15.28:0/1844791313 wait complete. 2024-04-26T01:41:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-26T01:41:12.888 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-26T01:41:09.977562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1390171,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":2125,"num_read_kb":1544,"num_write":687,"num_write_kb":5566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181492,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1000,"ondisk_log_size":1000,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378196,"kb_used_data":12372,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563628,"statfs":{"total":1151940427776,"available":1151553155072,"internally_reserved":0,"allocated":12668928,"data_stored":10029225,"data_compressed":251610,"data_compressed_allocated":331776,"data_compressed_original":3993600,"omap_allocated":19140,"internal_metadata":374453564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":198605,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":536,"num_read_kb":585,"num_write":206,"num_write_kb":1182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.994734"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206141+0000","last_change":"2024-04-26T01:36:54.288460+0000","last_active":"2024-04-26T01:40:58.206141+0000","last_peered":"2024-04-26T01:40:58.206141+0000","last_clean":"2024-04-26T01:40:58.206141+0000","last_became_active":"2024-04-26T01:36:54.288033+0000","last_became_peered":"2024-04-26T01:36:54.288033+0000","last_unstale":"2024-04-26T01:40:58.206141+0000","last_undegraded":"2024-04-26T01:40:58.206141+0000","last_fullsized":"2024-04-26T01:40:58.206141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:33.737622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938949+0000","last_change":"2024-04-26T01:36:54.287367+0000","last_active":"2024-04-26T01:40:57.938949+0000","last_peered":"2024-04-26T01:40:57.938949+0000","last_clean":"2024-04-26T01:40:57.938949+0000","last_became_active":"2024-04-26T01:36:54.287024+0000","last_became_peered":"2024-04-26T01:36:54.287024+0000","last_unstale":"2024-04-26T01:40:57.938949+0000","last_undegraded":"2024-04-26T01:40:57.938949+0000","last_fullsized":"2024-04-26T01:40:57.938949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:31.014627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938539+0000","last_change":"2024-04-26T01:36:54.288433+0000","last_active":"2024-04-26T01:40:57.938539+0000","last_peered":"2024-04-26T01:40:57.938539+0000","last_clean":"2024-04-26T01:40:57.938539+0000","last_became_active":"2024-04-26T01:36:54.288002+0000","last_became_peered":"2024-04-26T01:36:54.288002+0000","last_unstale":"2024-04-26T01:40:57.938539+0000","last_undegraded":"2024-04-26T01:40:57.938539+0000","last_fullsized":"2024-04-26T01:40:57.938539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:58:37.885179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939129+0000","last_change":"2024-04-26T01:36:54.273857+0000","last_active":"2024-04-26T01:40:57.939129+0000","last_peered":"2024-04-26T01:40:57.939129+0000","last_clean":"2024-04-26T01:40:57.939129+0000","last_became_active":"2024-04-26T01:36:54.272376+0000","last_became_peered":"2024-04-26T01:36:54.272376+0000","last_unstale":"2024-04-26T01:40:57.939129+0000","last_undegraded":"2024-04-26T01:40:57.939129+0000","last_fullsized":"2024-04-26T01:40:57.939129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:30.110044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079196+0000","last_change":"2024-04-26T01:36:54.288606+0000","last_active":"2024-04-26T01:40:58.079196+0000","last_peered":"2024-04-26T01:40:58.079196+0000","last_clean":"2024-04-26T01:40:58.079196+0000","last_became_active":"2024-04-26T01:36:54.288247+0000","last_became_peered":"2024-04-26T01:36:54.288247+0000","last_unstale":"2024-04-26T01:40:58.079196+0000","last_undegraded":"2024-04-26T01:40:58.079196+0000","last_fullsized":"2024-04-26T01:40:58.079196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:47:56.944164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967023+0000","last_change":"2024-04-26T01:36:54.277701+0000","last_active":"2024-04-26T01:40:57.967023+0000","last_peered":"2024-04-26T01:40:57.967023+0000","last_clean":"2024-04-26T01:40:57.967023+0000","last_became_active":"2024-04-26T01:36:54.277306+0000","last_became_peered":"2024-04-26T01:36:54.277306+0000","last_unstale":"2024-04-26T01:40:57.967023+0000","last_undegraded":"2024-04-26T01:40:57.967023+0000","last_fullsized":"2024-04-26T01:40:57.967023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:42:05.058525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206390+0000","last_change":"2024-04-26T01:36:54.274090+0000","last_active":"2024-04-26T01:40:58.206390+0000","last_peered":"2024-04-26T01:40:58.206390+0000","last_clean":"2024-04-26T01:40:58.206390+0000","last_became_active":"2024-04-26T01:36:54.273700+0000","last_became_peered":"2024-04-26T01:36:54.273700+0000","last_unstale":"2024-04-26T01:40:58.206390+0000","last_undegraded":"2024-04-26T01:40:58.206390+0000","last_fullsized":"2024-04-26T01:40:58.206390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:32.207534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076612+0000","last_change":"2024-04-26T01:37:25.499759+0000","last_active":"2024-04-26T01:40:58.076612+0000","last_peered":"2024-04-26T01:40:58.076612+0000","last_clean":"2024-04-26T01:40:58.076612+0000","last_became_active":"2024-04-26T01:37:25.499317+0000","last_became_peered":"2024-04-26T01:37:25.499317+0000","last_unstale":"2024-04-26T01:40:58.076612+0000","last_undegraded":"2024-04-26T01:40:58.076612+0000","last_fullsized":"2024-04-26T01:40:58.076612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:39.972315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206270+0000","last_change":"2024-04-26T01:36:54.277457+0000","last_active":"2024-04-26T01:40:58.206270+0000","last_peered":"2024-04-26T01:40:58.206270+0000","last_clean":"2024-04-26T01:40:58.206270+0000","last_became_active":"2024-04-26T01:36:54.277118+0000","last_became_peered":"2024-04-26T01:36:54.277118+0000","last_unstale":"2024-04-26T01:40:58.206270+0000","last_undegraded":"2024-04-26T01:40:58.206270+0000","last_fullsized":"2024-04-26T01:40:58.206270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:48:05.628027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967353+0000","last_change":"2024-04-26T01:36:54.278494+0000","last_active":"2024-04-26T01:40:57.967353+0000","last_peered":"2024-04-26T01:40:57.967353+0000","last_clean":"2024-04-26T01:40:57.967353+0000","last_became_active":"2024-04-26T01:36:54.278188+0000","last_became_peered":"2024-04-26T01:36:54.278188+0000","last_unstale":"2024-04-26T01:40:57.967353+0000","last_undegraded":"2024-04-26T01:40:57.967353+0000","last_fullsized":"2024-04-26T01:40:57.967353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:59.150771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.207126+0000","last_change":"2024-04-26T01:36:54.277424+0000","last_active":"2024-04-26T01:40:58.207126+0000","last_peered":"2024-04-26T01:40:58.207126+0000","last_clean":"2024-04-26T01:40:58.207126+0000","last_became_active":"2024-04-26T01:36:54.275752+0000","last_became_peered":"2024-04-26T01:36:54.275752+0000","last_unstale":"2024-04-26T01:40:58.207126+0000","last_undegraded":"2024-04-26T01:40:58.207126+0000","last_fullsized":"2024-04-26T01:40:58.207126+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:22.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076889+0000","last_change":"2024-04-26T01:37:25.502448+0000","last_active":"2024-04-26T01:40:58.076889+0000","last_peered":"2024-04-26T01:40:58.076889+0000","last_clean":"2024-04-26T01:40:58.076889+0000","last_became_active":"2024-04-26T01:37:25.501161+0000","last_became_peered":"2024-04-26T01:37:25.501161+0000","last_unstale":"2024-04-26T01:40:58.076889+0000","last_undegraded":"2024-04-26T01:40:58.076889+0000","last_fullsized":"2024-04-26T01:40:58.076889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:41:40.443780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949697+0000","last_change":"2024-04-26T01:36:54.268721+0000","last_active":"2024-04-26T01:40:57.949697+0000","last_peered":"2024-04-26T01:40:57.949697+0000","last_clean":"2024-04-26T01:40:57.949697+0000","last_became_active":"2024-04-26T01:36:54.268007+0000","last_became_peered":"2024-04-26T01:36:54.268007+0000","last_unstale":"2024-04-26T01:40:57.949697+0000","last_undegraded":"2024-04-26T01:40:57.949697+0000","last_fullsized":"2024-04-26T01:40:57.949697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:55.120130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938165+0000","last_change":"2024-04-26T01:36:54.286595+0000","last_active":"2024-04-26T01:40:57.938165+0000","last_peered":"2024-04-26T01:40:57.938165+0000","last_clean":"2024-04-26T01:40:57.938165+0000","last_became_active":"2024-04-26T01:36:54.286236+0000","last_became_peered":"2024-04-26T01:36:54.286236+0000","last_unstale":"2024-04-26T01:40:57.938165+0000","last_undegraded":"2024-04-26T01:40:57.938165+0000","last_fullsized":"2024-04-26T01:40:57.938165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:59.443467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938105+0000","last_change":"2024-04-26T01:36:54.293254+0000","last_active":"2024-04-26T01:40:57.938105+0000","last_peered":"2024-04-26T01:40:57.938105+0000","last_clean":"2024-04-26T01:40:57.938105+0000","last_became_active":"2024-04-26T01:36:54.292853+0000","last_became_peered":"2024-04-26T01:36:54.292853+0000","last_unstale":"2024-04-26T01:40:57.938105+0000","last_undegraded":"2024-04-26T01:40:57.938105+0000","last_fullsized":"2024-04-26T01:40:57.938105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:19.844006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205747+0000","last_change":"2024-04-26T01:36:54.285346+0000","last_active":"2024-04-26T01:40:58.205747+0000","last_peered":"2024-04-26T01:40:58.205747+0000","last_clean":"2024-04-26T01:40:58.205747+0000","last_became_active":"2024-04-26T01:36:54.284949+0000","last_became_peered":"2024-04-26T01:36:54.284949+0000","last_unstale":"2024-04-26T01:40:58.205747+0000","last_undegraded":"2024-04-26T01:40:58.205747+0000","last_fullsized":"2024-04-26T01:40:58.205747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:33:29.276240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205959+0000","last_change":"2024-04-26T01:36:54.283161+0000","last_active":"2024-04-26T01:40:58.205959+0000","last_peered":"2024-04-26T01:40:58.205959+0000","last_clean":"2024-04-26T01:40:58.205959+0000","last_became_active":"2024-04-26T01:36:54.282908+0000","last_became_peered":"2024-04-26T01:36:54.282908+0000","last_unstale":"2024-04-26T01:40:58.205959+0000","last_undegraded":"2024-04-26T01:40:58.205959+0000","last_fullsized":"2024-04-26T01:40:58.205959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:26.503964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.077086+0000","last_change":"2024-04-26T01:36:54.282516+0000","last_active":"2024-04-26T01:40:58.077086+0000","last_peered":"2024-04-26T01:40:58.077086+0000","last_clean":"2024-04-26T01:40:58.077086+0000","last_became_active":"2024-04-26T01:36:54.278634+0000","last_became_peered":"2024-04-26T01:36:54.278634+0000","last_unstale":"2024-04-26T01:40:58.077086+0000","last_undegraded":"2024-04-26T01:40:58.077086+0000","last_fullsized":"2024-04-26T01:40:58.077086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:10:38.275720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939505+0000","last_change":"2024-04-26T01:36:54.289502+0000","last_active":"2024-04-26T01:40:57.939505+0000","last_peered":"2024-04-26T01:40:57.939505+0000","last_clean":"2024-04-26T01:40:57.939505+0000","last_became_active":"2024-04-26T01:36:54.289079+0000","last_became_peered":"2024-04-26T01:36:54.289079+0000","last_unstale":"2024-04-26T01:40:57.939505+0000","last_undegraded":"2024-04-26T01:40:57.939505+0000","last_fullsized":"2024-04-26T01:40:57.939505+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:51.412032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076838+0000","last_change":"2024-04-26T01:36:50.243164+0000","last_active":"2024-04-26T01:40:58.076838+0000","last_peered":"2024-04-26T01:40:58.076838+0000","last_clean":"2024-04-26T01:40:58.076838+0000","last_became_active":"2024-04-26T01:36:50.242508+0000","last_became_peered":"2024-04-26T01:36:50.242508+0000","last_unstale":"2024-04-26T01:40:58.076838+0000","last_undegraded":"2024-04-26T01:40:58.076838+0000","last_fullsized":"2024-04-26T01:40:58.076838+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:06:01.662698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954129+0000","last_change":"2024-04-26T01:36:54.279809+0000","last_active":"2024-04-26T01:40:57.954129+0000","last_peered":"2024-04-26T01:40:57.954129+0000","last_clean":"2024-04-26T01:40:57.954129+0000","last_became_active":"2024-04-26T01:36:54.279509+0000","last_became_peered":"2024-04-26T01:36:54.279509+0000","last_unstale":"2024-04-26T01:40:57.954129+0000","last_undegraded":"2024-04-26T01:40:57.954129+0000","last_fullsized":"2024-04-26T01:40:57.954129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:54:26.734136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954751+0000","last_change":"2024-04-26T01:36:50.261942+0000","last_active":"2024-04-26T01:40:57.954751+0000","last_peered":"2024-04-26T01:40:57.954751+0000","last_clean":"2024-04-26T01:40:57.954751+0000","last_became_active":"2024-04-26T01:36:50.261564+0000","last_became_peered":"2024-04-26T01:36:50.261564+0000","last_unstale":"2024-04-26T01:40:57.954751+0000","last_undegraded":"2024-04-26T01:40:57.954751+0000","last_fullsized":"2024-04-26T01:40:57.954751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:25.102557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938783+0000","last_change":"2024-04-26T01:36:54.288011+0000","last_active":"2024-04-26T01:40:57.938783+0000","last_peered":"2024-04-26T01:40:57.938783+0000","last_clean":"2024-04-26T01:40:57.938783+0000","last_became_active":"2024-04-26T01:36:54.286853+0000","last_became_peered":"2024-04-26T01:36:54.286853+0000","last_unstale":"2024-04-26T01:40:57.938783+0000","last_undegraded":"2024-04-26T01:40:57.938783+0000","last_fullsized":"2024-04-26T01:40:57.938783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:53.422470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079561+0000","last_change":"2024-04-26T01:36:50.235291+0000","last_active":"2024-04-26T01:40:58.079561+0000","last_peered":"2024-04-26T01:40:58.079561+0000","last_clean":"2024-04-26T01:40:58.079561+0000","last_became_active":"2024-04-26T01:36:50.234939+0000","last_became_peered":"2024-04-26T01:36:50.234939+0000","last_unstale":"2024-04-26T01:40:58.079561+0000","last_undegraded":"2024-04-26T01:40:58.079561+0000","last_fullsized":"2024-04-26T01:40:58.079561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:13.048185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205932+0000","last_change":"2024-04-26T01:36:54.282731+0000","last_active":"2024-04-26T01:40:58.205932+0000","last_peered":"2024-04-26T01:40:58.205932+0000","last_clean":"2024-04-26T01:40:58.205932+0000","last_became_active":"2024-04-26T01:36:54.282384+0000","last_became_peered":"2024-04-26T01:36:54.282384+0000","last_unstale":"2024-04-26T01:40:58.205932+0000","last_undegraded":"2024-04-26T01:40:58.205932+0000","last_fullsized":"2024-04-26T01:40:58.205932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:09.493841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079407+0000","last_change":"2024-04-26T01:36:50.233581+0000","last_active":"2024-04-26T01:40:58.079407+0000","last_peered":"2024-04-26T01:40:58.079407+0000","last_clean":"2024-04-26T01:40:58.079407+0000","last_became_active":"2024-04-26T01:36:50.233210+0000","last_became_peered":"2024-04-26T01:36:50.233210+0000","last_unstale":"2024-04-26T01:40:58.079407+0000","last_undegraded":"2024-04-26T01:40:58.079407+0000","last_fullsized":"2024-04-26T01:40:58.079407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:01.223954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970312+0000","last_change":"2024-04-26T01:36:54.274657+0000","last_active":"2024-04-26T01:40:57.970312+0000","last_peered":"2024-04-26T01:40:57.970312+0000","last_clean":"2024-04-26T01:40:57.970312+0000","last_became_active":"2024-04-26T01:36:54.274333+0000","last_became_peered":"2024-04-26T01:36:54.274333+0000","last_unstale":"2024-04-26T01:40:57.970312+0000","last_undegraded":"2024-04-26T01:40:57.970312+0000","last_fullsized":"2024-04-26T01:40:57.970312+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:37.243307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":286,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:06.989136+0000","last_change":"2024-04-26T01:37:25.502311+0000","last_active":"2024-04-26T01:41:06.989136+0000","last_peered":"2024-04-26T01:41:06.989136+0000","last_clean":"2024-04-26T01:41:06.989136+0000","last_became_active":"2024-04-26T01:37:25.493259+0000","last_became_peered":"2024-04-26T01:37:25.493259+0000","last_unstale":"2024-04-26T01:41:06.989136+0000","last_undegraded":"2024-04-26T01:41:06.989136+0000","last_fullsized":"2024-04-26T01:41:06.989136+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:46:49.224905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":233,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079111+0000","last_change":"2024-04-26T01:36:54.275116+0000","last_active":"2024-04-26T01:40:58.079111+0000","last_peered":"2024-04-26T01:40:58.079111+0000","last_clean":"2024-04-26T01:40:58.079111+0000","last_became_active":"2024-04-26T01:36:54.274815+0000","last_became_peered":"2024-04-26T01:36:54.274815+0000","last_unstale":"2024-04-26T01:40:58.079111+0000","last_undegraded":"2024-04-26T01:40:58.079111+0000","last_fullsized":"2024-04-26T01:40:58.079111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:04.427670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.966940+0000","last_change":"2024-04-26T01:37:25.505327+0000","last_active":"2024-04-26T01:40:57.966940+0000","last_peered":"2024-04-26T01:40:57.966940+0000","last_clean":"2024-04-26T01:40:57.966940+0000","last_became_active":"2024-04-26T01:37:25.497881+0000","last_became_peered":"2024-04-26T01:37:25.497881+0000","last_unstale":"2024-04-26T01:40:57.966940+0000","last_undegraded":"2024-04-26T01:40:57.966940+0000","last_fullsized":"2024-04-26T01:40:57.966940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:09.181372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970138+0000","last_change":"2024-04-26T01:36:54.276415+0000","last_active":"2024-04-26T01:40:57.970138+0000","last_peered":"2024-04-26T01:40:57.970138+0000","last_clean":"2024-04-26T01:40:57.970138+0000","last_became_active":"2024-04-26T01:36:54.276080+0000","last_became_peered":"2024-04-26T01:36:54.276080+0000","last_unstale":"2024-04-26T01:40:57.970138+0000","last_undegraded":"2024-04-26T01:40:57.970138+0000","last_fullsized":"2024-04-26T01:40:57.970138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:08:17.171620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949572+0000","last_change":"2024-04-26T01:36:50.241552+0000","last_active":"2024-04-26T01:40:57.949572+0000","last_peered":"2024-04-26T01:40:57.949572+0000","last_clean":"2024-04-26T01:40:57.949572+0000","last_became_active":"2024-04-26T01:36:50.241022+0000","last_became_peered":"2024-04-26T01:36:50.241022+0000","last_unstale":"2024-04-26T01:40:57.949572+0000","last_undegraded":"2024-04-26T01:40:57.949572+0000","last_fullsized":"2024-04-26T01:40:57.949572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:55.503831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953427+0000","last_change":"2024-04-26T01:36:54.276641+0000","last_active":"2024-04-26T01:40:57.953427+0000","last_peered":"2024-04-26T01:40:57.953427+0000","last_clean":"2024-04-26T01:40:57.953427+0000","last_became_active":"2024-04-26T01:36:54.276309+0000","last_became_peered":"2024-04-26T01:36:54.276309+0000","last_unstale":"2024-04-26T01:40:57.953427+0000","last_undegraded":"2024-04-26T01:40:57.953427+0000","last_fullsized":"2024-04-26T01:40:57.953427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:53.725584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969805+0000","last_change":"2024-04-26T01:36:50.235504+0000","last_active":"2024-04-26T01:40:57.969805+0000","last_peered":"2024-04-26T01:40:57.969805+0000","last_clean":"2024-04-26T01:40:57.969805+0000","last_became_active":"2024-04-26T01:36:50.235161+0000","last_became_peered":"2024-04-26T01:36:50.235161+0000","last_unstale":"2024-04-26T01:40:57.969805+0000","last_undegraded":"2024-04-26T01:40:57.969805+0000","last_fullsized":"2024-04-26T01:40:57.969805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:05.951521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076712+0000","last_change":"2024-04-26T01:36:54.298236+0000","last_active":"2024-04-26T01:40:58.076712+0000","last_peered":"2024-04-26T01:40:58.076712+0000","last_clean":"2024-04-26T01:40:58.076712+0000","last_became_active":"2024-04-26T01:36:54.297850+0000","last_became_peered":"2024-04-26T01:36:54.297850+0000","last_unstale":"2024-04-26T01:40:58.076712+0000","last_undegraded":"2024-04-26T01:40:58.076712+0000","last_fullsized":"2024-04-26T01:40:58.076712+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:35:35.244502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":265,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:06.988031+0000","last_change":"2024-04-26T01:36:50.241418+0000","last_active":"2024-04-26T01:41:06.988031+0000","last_peered":"2024-04-26T01:41:06.988031+0000","last_clean":"2024-04-26T01:41:06.988031+0000","last_became_active":"2024-04-26T01:36:50.241044+0000","last_became_peered":"2024-04-26T01:36:50.241044+0000","last_unstale":"2024-04-26T01:41:06.988031+0000","last_undegraded":"2024-04-26T01:41:06.988031+0000","last_fullsized":"2024-04-26T01:41:06.988031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:13.317953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":231,"num_read_kb":231,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206340+0000","last_change":"2024-04-26T01:36:54.284410+0000","last_active":"2024-04-26T01:40:58.206340+0000","last_peered":"2024-04-26T01:40:58.206340+0000","last_clean":"2024-04-26T01:40:58.206340+0000","last_became_active":"2024-04-26T01:36:54.283629+0000","last_became_peered":"2024-04-26T01:36:54.283629+0000","last_unstale":"2024-04-26T01:40:58.206340+0000","last_undegraded":"2024-04-26T01:40:58.206340+0000","last_fullsized":"2024-04-26T01:40:58.206340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:48:15.380006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205831+0000","last_change":"2024-04-26T01:37:25.499874+0000","last_active":"2024-04-26T01:40:58.205831+0000","last_peered":"2024-04-26T01:40:58.205831+0000","last_clean":"2024-04-26T01:40:58.205831+0000","last_became_active":"2024-04-26T01:37:25.494878+0000","last_became_peered":"2024-04-26T01:37:25.494878+0000","last_unstale":"2024-04-26T01:40:58.205831+0000","last_undegraded":"2024-04-26T01:40:58.205831+0000","last_fullsized":"2024-04-26T01:40:58.205831+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:48:12.678331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938888+0000","last_change":"2024-04-26T01:36:54.285118+0000","last_active":"2024-04-26T01:40:57.938888+0000","last_peered":"2024-04-26T01:40:57.938888+0000","last_clean":"2024-04-26T01:40:57.938888+0000","last_became_active":"2024-04-26T01:36:54.284554+0000","last_became_peered":"2024-04-26T01:36:54.284554+0000","last_unstale":"2024-04-26T01:40:57.938888+0000","last_undegraded":"2024-04-26T01:40:57.938888+0000","last_fullsized":"2024-04-26T01:40:57.938888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:13.721684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970272+0000","last_change":"2024-04-26T01:36:50.262451+0000","last_active":"2024-04-26T01:40:57.970272+0000","last_peered":"2024-04-26T01:40:57.970272+0000","last_clean":"2024-04-26T01:40:57.970272+0000","last_became_active":"2024-04-26T01:36:50.262056+0000","last_became_peered":"2024-04-26T01:36:50.262056+0000","last_unstale":"2024-04-26T01:40:57.970272+0000","last_undegraded":"2024-04-26T01:40:57.970272+0000","last_fullsized":"2024-04-26T01:40:57.970272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:50.813675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206136+0000","last_change":"2024-04-26T01:36:54.283975+0000","last_active":"2024-04-26T01:40:58.206136+0000","last_peered":"2024-04-26T01:40:58.206136+0000","last_clean":"2024-04-26T01:40:58.206136+0000","last_became_active":"2024-04-26T01:36:54.283564+0000","last_became_peered":"2024-04-26T01:36:54.283564+0000","last_unstale":"2024-04-26T01:40:58.206136+0000","last_undegraded":"2024-04-26T01:40:58.206136+0000","last_fullsized":"2024-04-26T01:40:58.206136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:53.881763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'44","reported_seq":552,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:08.817494+0000","last_change":"2024-04-26T01:36:50.238839+0000","last_active":"2024-04-26T01:41:08.817494+0000","last_peered":"2024-04-26T01:41:08.817494+0000","last_clean":"2024-04-26T01:41:08.817494+0000","last_became_active":"2024-04-26T01:36:50.238452+0000","last_became_peered":"2024-04-26T01:36:50.238452+0000","last_unstale":"2024-04-26T01:41:08.817494+0000","last_undegraded":"2024-04-26T01:41:08.817494+0000","last_fullsized":"2024-04-26T01:41:08.817494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:05:40.426825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72496,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":479,"num_read_kb":311,"num_write":45,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079361+0000","last_change":"2024-04-26T01:36:54.271446+0000","last_active":"2024-04-26T01:40:58.079361+0000","last_peered":"2024-04-26T01:40:58.079361+0000","last_clean":"2024-04-26T01:40:58.079361+0000","last_became_active":"2024-04-26T01:36:54.271122+0000","last_became_peered":"2024-04-26T01:36:54.271122+0000","last_unstale":"2024-04-26T01:40:58.079361+0000","last_undegraded":"2024-04-26T01:40:58.079361+0000","last_fullsized":"2024-04-26T01:40:58.079361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:28:24.275481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939266+0000","last_change":"2024-04-26T01:36:50.237726+0000","last_active":"2024-04-26T01:40:57.939266+0000","last_peered":"2024-04-26T01:40:57.939266+0000","last_clean":"2024-04-26T01:40:57.939266+0000","last_became_active":"2024-04-26T01:36:50.237215+0000","last_became_peered":"2024-04-26T01:36:50.237215+0000","last_unstale":"2024-04-26T01:40:57.939266+0000","last_undegraded":"2024-04-26T01:40:57.939266+0000","last_fullsized":"2024-04-26T01:40:57.939266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:41.781876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949425+0000","last_change":"2024-04-26T01:36:54.290616+0000","last_active":"2024-04-26T01:40:57.949425+0000","last_peered":"2024-04-26T01:40:57.949425+0000","last_clean":"2024-04-26T01:40:57.949425+0000","last_became_active":"2024-04-26T01:36:54.290293+0000","last_became_peered":"2024-04-26T01:36:54.290293+0000","last_unstale":"2024-04-26T01:40:57.949425+0000","last_undegraded":"2024-04-26T01:40:57.949425+0000","last_fullsized":"2024-04-26T01:40:57.949425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:07.439333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969982+0000","last_change":"2024-04-26T01:36:50.237578+0000","last_active":"2024-04-26T01:40:57.969982+0000","last_peered":"2024-04-26T01:40:57.969982+0000","last_clean":"2024-04-26T01:40:57.969982+0000","last_became_active":"2024-04-26T01:36:50.237224+0000","last_became_peered":"2024-04-26T01:36:50.237224+0000","last_unstale":"2024-04-26T01:40:57.969982+0000","last_undegraded":"2024-04-26T01:40:57.969982+0000","last_fullsized":"2024-04-26T01:40:57.969982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:46:38.078631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206709+0000","last_change":"2024-04-26T01:36:54.280818+0000","last_active":"2024-04-26T01:40:58.206709+0000","last_peered":"2024-04-26T01:40:58.206709+0000","last_clean":"2024-04-26T01:40:58.206709+0000","last_became_active":"2024-04-26T01:36:54.280440+0000","last_became_peered":"2024-04-26T01:36:54.280440+0000","last_unstale":"2024-04-26T01:40:58.206709+0000","last_undegraded":"2024-04-26T01:40:58.206709+0000","last_fullsized":"2024-04-26T01:40:58.206709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:47.185219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":269,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:06.980034+0000","last_change":"2024-04-26T01:36:50.234602+0000","last_active":"2024-04-26T01:41:06.980034+0000","last_peered":"2024-04-26T01:41:06.980034+0000","last_clean":"2024-04-26T01:41:06.980034+0000","last_became_active":"2024-04-26T01:36:50.234175+0000","last_became_peered":"2024-04-26T01:36:50.234175+0000","last_unstale":"2024-04-26T01:41:06.980034+0000","last_undegraded":"2024-04-26T01:41:06.980034+0000","last_fullsized":"2024-04-26T01:41:06.980034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:17:14.236069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1289,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":236,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206181+0000","last_change":"2024-04-26T01:36:54.271466+0000","last_active":"2024-04-26T01:40:58.206181+0000","last_peered":"2024-04-26T01:40:58.206181+0000","last_clean":"2024-04-26T01:40:58.206181+0000","last_became_active":"2024-04-26T01:36:54.271086+0000","last_became_peered":"2024-04-26T01:36:54.271086+0000","last_unstale":"2024-04-26T01:40:58.206181+0000","last_undegraded":"2024-04-26T01:40:58.206181+0000","last_fullsized":"2024-04-26T01:40:58.206181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:30:41.052895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079039+0000","last_change":"2024-04-26T01:37:25.518730+0000","last_active":"2024-04-26T01:40:58.079039+0000","last_peered":"2024-04-26T01:40:58.079039+0000","last_clean":"2024-04-26T01:40:58.079039+0000","last_became_active":"2024-04-26T01:37:25.504464+0000","last_became_peered":"2024-04-26T01:37:25.504464+0000","last_unstale":"2024-04-26T01:40:58.079039+0000","last_undegraded":"2024-04-26T01:40:58.079039+0000","last_fullsized":"2024-04-26T01:40:58.079039+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:07.578917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969886+0000","last_change":"2024-04-26T01:36:54.273621+0000","last_active":"2024-04-26T01:40:57.969886+0000","last_peered":"2024-04-26T01:40:57.969886+0000","last_clean":"2024-04-26T01:40:57.969886+0000","last_became_active":"2024-04-26T01:36:54.273218+0000","last_became_peered":"2024-04-26T01:36:54.273218+0000","last_unstale":"2024-04-26T01:40:57.969886+0000","last_undegraded":"2024-04-26T01:40:57.969886+0000","last_fullsized":"2024-04-26T01:40:57.969886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:55.543943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205908+0000","last_change":"2024-04-26T01:37:25.508575+0000","last_active":"2024-04-26T01:40:58.205908+0000","last_peered":"2024-04-26T01:40:58.205908+0000","last_clean":"2024-04-26T01:40:58.205908+0000","last_became_active":"2024-04-26T01:37:25.496407+0000","last_became_peered":"2024-04-26T01:37:25.496407+0000","last_unstale":"2024-04-26T01:40:58.205908+0000","last_undegraded":"2024-04-26T01:40:58.205908+0000","last_fullsized":"2024-04-26T01:40:58.205908+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:05.878252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205907+0000","last_change":"2024-04-26T01:36:54.272147+0000","last_active":"2024-04-26T01:40:58.205907+0000","last_peered":"2024-04-26T01:40:58.205907+0000","last_clean":"2024-04-26T01:40:58.205907+0000","last_became_active":"2024-04-26T01:36:54.271737+0000","last_became_peered":"2024-04-26T01:36:54.271737+0000","last_unstale":"2024-04-26T01:40:58.205907+0000","last_undegraded":"2024-04-26T01:40:58.205907+0000","last_fullsized":"2024-04-26T01:40:58.205907+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:41.445073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.076978+0000","last_change":"2024-04-26T01:36:50.264444+0000","last_active":"2024-04-26T01:40:58.076978+0000","last_peered":"2024-04-26T01:40:58.076978+0000","last_clean":"2024-04-26T01:40:58.076978+0000","last_became_active":"2024-04-26T01:36:50.264058+0000","last_became_peered":"2024-04-26T01:36:50.264058+0000","last_unstale":"2024-04-26T01:40:58.076978+0000","last_undegraded":"2024-04-26T01:40:58.076978+0000","last_fullsized":"2024-04-26T01:40:58.076978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:56:15.428588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967286+0000","last_change":"2024-04-26T01:36:54.289537+0000","last_active":"2024-04-26T01:40:57.967286+0000","last_peered":"2024-04-26T01:40:57.967286+0000","last_clean":"2024-04-26T01:40:57.967286+0000","last_became_active":"2024-04-26T01:36:54.289149+0000","last_became_peered":"2024-04-26T01:36:54.289149+0000","last_unstale":"2024-04-26T01:40:57.967286+0000","last_undegraded":"2024-04-26T01:40:57.967286+0000","last_fullsized":"2024-04-26T01:40:57.967286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:07.801551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206728+0000","last_change":"2024-04-26T01:36:50.242190+0000","last_active":"2024-04-26T01:40:58.206728+0000","last_peered":"2024-04-26T01:40:58.206728+0000","last_clean":"2024-04-26T01:40:58.206728+0000","last_became_active":"2024-04-26T01:36:50.241890+0000","last_became_peered":"2024-04-26T01:36:50.241890+0000","last_unstale":"2024-04-26T01:40:58.206728+0000","last_undegraded":"2024-04-26T01:40:58.206728+0000","last_fullsized":"2024-04-26T01:40:58.206728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:29.472502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.967093+0000","last_change":"2024-04-26T01:36:54.294699+0000","last_active":"2024-04-26T01:40:57.967093+0000","last_peered":"2024-04-26T01:40:57.967093+0000","last_clean":"2024-04-26T01:40:57.967093+0000","last_became_active":"2024-04-26T01:36:54.294313+0000","last_became_peered":"2024-04-26T01:36:54.294313+0000","last_unstale":"2024-04-26T01:40:57.967093+0000","last_undegraded":"2024-04-26T01:40:57.967093+0000","last_fullsized":"2024-04-26T01:40:57.967093+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:15.026357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.970178+0000","last_change":"2024-04-26T01:36:54.276798+0000","last_active":"2024-04-26T01:40:57.970178+0000","last_peered":"2024-04-26T01:40:57.970178+0000","last_clean":"2024-04-26T01:40:57.970178+0000","last_became_active":"2024-04-26T01:36:54.275955+0000","last_became_peered":"2024-04-26T01:36:54.275955+0000","last_unstale":"2024-04-26T01:40:57.970178+0000","last_undegraded":"2024-04-26T01:40:57.970178+0000","last_fullsized":"2024-04-26T01:40:57.970178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:17.020909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:06.978807+0000","last_change":"2024-04-26T01:36:50.263356+0000","last_active":"2024-04-26T01:41:06.978807+0000","last_peered":"2024-04-26T01:41:06.978807+0000","last_clean":"2024-04-26T01:41:06.978807+0000","last_became_active":"2024-04-26T01:36:50.261596+0000","last_became_peered":"2024-04-26T01:36:50.261596+0000","last_unstale":"2024-04-26T01:41:06.978807+0000","last_undegraded":"2024-04-26T01:41:06.978807+0000","last_fullsized":"2024-04-26T01:41:06.978807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:10.773347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":231,"num_read_kb":231,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954042+0000","last_change":"2024-04-26T01:36:54.288135+0000","last_active":"2024-04-26T01:40:57.954042+0000","last_peered":"2024-04-26T01:40:57.954042+0000","last_clean":"2024-04-26T01:40:57.954042+0000","last_became_active":"2024-04-26T01:36:54.287728+0000","last_became_peered":"2024-04-26T01:36:54.287728+0000","last_unstale":"2024-04-26T01:40:57.954042+0000","last_undegraded":"2024-04-26T01:40:57.954042+0000","last_fullsized":"2024-04-26T01:40:57.954042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:18.428151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938946+0000","last_change":"2024-04-26T01:36:50.243522+0000","last_active":"2024-04-26T01:40:57.938946+0000","last_peered":"2024-04-26T01:40:57.938946+0000","last_clean":"2024-04-26T01:40:57.938946+0000","last_became_active":"2024-04-26T01:36:50.243084+0000","last_became_peered":"2024-04-26T01:36:50.243084+0000","last_unstale":"2024-04-26T01:40:57.938946+0000","last_undegraded":"2024-04-26T01:40:57.938946+0000","last_fullsized":"2024-04-26T01:40:57.938946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:03.159750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'148","reported_seq":406,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:00.093944+0000","last_change":"2024-04-26T01:35:13.920486+0000","last_active":"2024-04-26T01:41:00.093944+0000","last_peered":"2024-04-26T01:41:00.093944+0000","last_clean":"2024-04-26T01:41:00.093944+0000","last_became_active":"2024-04-26T01:35:13.919834+0000","last_became_peered":"2024-04-26T01:35:13.919834+0000","last_unstale":"2024-04-26T01:41:00.093944+0000","last_undegraded":"2024-04-26T01:41:00.093944+0000","last_fullsized":"2024-04-26T01:41:00.093944+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:31:12.563114+0000","last_clean_scrub_stamp":"2024-04-26T01:31:12.563114+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:16:53.935491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949264+0000","last_change":"2024-04-26T01:36:54.270553+0000","last_active":"2024-04-26T01:40:57.949264+0000","last_peered":"2024-04-26T01:40:57.949264+0000","last_clean":"2024-04-26T01:40:57.949264+0000","last_became_active":"2024-04-26T01:36:54.269986+0000","last_became_peered":"2024-04-26T01:36:54.269986+0000","last_unstale":"2024-04-26T01:40:57.949264+0000","last_undegraded":"2024-04-26T01:40:57.949264+0000","last_fullsized":"2024-04-26T01:40:57.949264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:19.597337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206618+0000","last_change":"2024-04-26T01:36:50.235583+0000","last_active":"2024-04-26T01:40:58.206618+0000","last_peered":"2024-04-26T01:40:58.206618+0000","last_clean":"2024-04-26T01:40:58.206618+0000","last_became_active":"2024-04-26T01:36:50.235214+0000","last_became_peered":"2024-04-26T01:36:50.235214+0000","last_unstale":"2024-04-26T01:40:58.206618+0000","last_undegraded":"2024-04-26T01:40:58.206618+0000","last_fullsized":"2024-04-26T01:40:58.206618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:05.822872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079498+0000","last_change":"2024-04-26T01:36:50.260988+0000","last_active":"2024-04-26T01:40:58.079498+0000","last_peered":"2024-04-26T01:40:58.079498+0000","last_clean":"2024-04-26T01:40:58.079498+0000","last_became_active":"2024-04-26T01:36:50.260615+0000","last_became_peered":"2024-04-26T01:36:50.260615+0000","last_unstale":"2024-04-26T01:40:58.079498+0000","last_undegraded":"2024-04-26T01:40:58.079498+0000","last_fullsized":"2024-04-26T01:40:58.079498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:15.933939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949847+0000","last_change":"2024-04-26T01:36:54.279650+0000","last_active":"2024-04-26T01:40:57.949847+0000","last_peered":"2024-04-26T01:40:57.949847+0000","last_clean":"2024-04-26T01:40:57.949847+0000","last_became_active":"2024-04-26T01:36:54.275814+0000","last_became_peered":"2024-04-26T01:36:54.275814+0000","last_unstale":"2024-04-26T01:40:57.949847+0000","last_undegraded":"2024-04-26T01:40:57.949847+0000","last_fullsized":"2024-04-26T01:40:57.949847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:58:42.369511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'125","reported_seq":280,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:00.732638+0000","last_change":"2024-04-26T01:36:50.248179+0000","last_active":"2024-04-26T01:41:00.732638+0000","last_peered":"2024-04-26T01:41:00.732638+0000","last_clean":"2024-04-26T01:41:00.732638+0000","last_became_active":"2024-04-26T01:36:50.247805+0000","last_became_peered":"2024-04-26T01:36:50.247805+0000","last_unstale":"2024-04-26T01:41:00.732638+0000","last_undegraded":"2024-04-26T01:41:00.732638+0000","last_fullsized":"2024-04-26T01:41:00.732638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:19:23.740920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":422,"num_read_kb":376,"num_write":231,"num_write_kb":1414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954091+0000","last_change":"2024-04-26T01:36:54.276626+0000","last_active":"2024-04-26T01:40:57.954091+0000","last_peered":"2024-04-26T01:40:57.954091+0000","last_clean":"2024-04-26T01:40:57.954091+0000","last_became_active":"2024-04-26T01:36:54.276238+0000","last_became_peered":"2024-04-26T01:36:54.276238+0000","last_unstale":"2024-04-26T01:40:57.954091+0000","last_undegraded":"2024-04-26T01:40:57.954091+0000","last_fullsized":"2024-04-26T01:40:57.954091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:57.306950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939024+0000","last_change":"2024-04-26T01:36:50.278502+0000","last_active":"2024-04-26T01:40:57.939024+0000","last_peered":"2024-04-26T01:40:57.939024+0000","last_clean":"2024-04-26T01:40:57.939024+0000","last_became_active":"2024-04-26T01:36:50.278215+0000","last_became_peered":"2024-04-26T01:36:50.278215+0000","last_unstale":"2024-04-26T01:40:57.939024+0000","last_undegraded":"2024-04-26T01:40:57.939024+0000","last_fullsized":"2024-04-26T01:40:57.939024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:05:46.142902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939164+0000","last_change":"2024-04-26T01:36:54.292600+0000","last_active":"2024-04-26T01:40:57.939164+0000","last_peered":"2024-04-26T01:40:57.939164+0000","last_clean":"2024-04-26T01:40:57.939164+0000","last_became_active":"2024-04-26T01:36:54.292260+0000","last_became_peered":"2024-04-26T01:36:54.292260+0000","last_unstale":"2024-04-26T01:40:57.939164+0000","last_undegraded":"2024-04-26T01:40:57.939164+0000","last_fullsized":"2024-04-26T01:40:57.939164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:10:12.296180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205780+0000","last_change":"2024-04-26T01:36:50.262479+0000","last_active":"2024-04-26T01:40:58.205780+0000","last_peered":"2024-04-26T01:40:58.205780+0000","last_clean":"2024-04-26T01:40:58.205780+0000","last_became_active":"2024-04-26T01:36:50.262109+0000","last_became_peered":"2024-04-26T01:36:50.262109+0000","last_unstale":"2024-04-26T01:40:58.205780+0000","last_undegraded":"2024-04-26T01:40:58.205780+0000","last_fullsized":"2024-04-26T01:40:58.205780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:19.805321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079067+0000","last_change":"2024-04-26T01:36:54.270051+0000","last_active":"2024-04-26T01:40:58.079067+0000","last_peered":"2024-04-26T01:40:58.079067+0000","last_clean":"2024-04-26T01:40:58.079067+0000","last_became_active":"2024-04-26T01:36:54.269670+0000","last_became_peered":"2024-04-26T01:36:54.269670+0000","last_unstale":"2024-04-26T01:40:58.079067+0000","last_undegraded":"2024-04-26T01:40:58.079067+0000","last_fullsized":"2024-04-26T01:40:58.079067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:06:44.137454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953570+0000","last_change":"2024-04-26T01:36:50.248107+0000","last_active":"2024-04-26T01:40:57.953570+0000","last_peered":"2024-04-26T01:40:57.953570+0000","last_clean":"2024-04-26T01:40:57.953570+0000","last_became_active":"2024-04-26T01:36:50.247695+0000","last_became_peered":"2024-04-26T01:36:50.247695+0000","last_unstale":"2024-04-26T01:40:57.953570+0000","last_undegraded":"2024-04-26T01:40:57.953570+0000","last_fullsized":"2024-04-26T01:40:57.953570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:17:52.702512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205622+0000","last_change":"2024-04-26T01:36:54.278686+0000","last_active":"2024-04-26T01:40:58.205622+0000","last_peered":"2024-04-26T01:40:58.205622+0000","last_clean":"2024-04-26T01:40:58.205622+0000","last_became_active":"2024-04-26T01:36:54.278312+0000","last_became_peered":"2024-04-26T01:36:54.278312+0000","last_unstale":"2024-04-26T01:40:58.205622+0000","last_undegraded":"2024-04-26T01:40:58.205622+0000","last_fullsized":"2024-04-26T01:40:58.205622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:46.193032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939223+0000","last_change":"2024-04-26T01:36:50.236715+0000","last_active":"2024-04-26T01:40:57.939223+0000","last_peered":"2024-04-26T01:40:57.939223+0000","last_clean":"2024-04-26T01:40:57.939223+0000","last_became_active":"2024-04-26T01:36:50.236184+0000","last_became_peered":"2024-04-26T01:36:50.236184+0000","last_unstale":"2024-04-26T01:40:57.939223+0000","last_undegraded":"2024-04-26T01:40:57.939223+0000","last_fullsized":"2024-04-26T01:40:57.939223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:24:16.063401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.954345+0000","last_change":"2024-04-26T01:36:54.290601+0000","last_active":"2024-04-26T01:40:57.954345+0000","last_peered":"2024-04-26T01:40:57.954345+0000","last_clean":"2024-04-26T01:40:57.954345+0000","last_became_active":"2024-04-26T01:36:54.290168+0000","last_became_peered":"2024-04-26T01:36:54.290168+0000","last_unstale":"2024-04-26T01:40:57.954345+0000","last_undegraded":"2024-04-26T01:40:57.954345+0000","last_fullsized":"2024-04-26T01:40:57.954345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:36:48.770431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.079189+0000","last_change":"2024-04-26T01:36:50.235394+0000","last_active":"2024-04-26T01:40:58.079189+0000","last_peered":"2024-04-26T01:40:58.079189+0000","last_clean":"2024-04-26T01:40:58.079189+0000","last_became_active":"2024-04-26T01:36:50.235012+0000","last_became_peered":"2024-04-26T01:36:50.235012+0000","last_unstale":"2024-04-26T01:40:58.079189+0000","last_undegraded":"2024-04-26T01:40:58.079189+0000","last_fullsized":"2024-04-26T01:40:58.079189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:24:17.218945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206668+0000","last_change":"2024-04-26T01:36:54.280490+0000","last_active":"2024-04-26T01:40:58.206668+0000","last_peered":"2024-04-26T01:40:58.206668+0000","last_clean":"2024-04-26T01:40:58.206668+0000","last_became_active":"2024-04-26T01:36:54.280047+0000","last_became_peered":"2024-04-26T01:36:54.280047+0000","last_unstale":"2024-04-26T01:40:58.206668+0000","last_undegraded":"2024-04-26T01:40:58.206668+0000","last_fullsized":"2024-04-26T01:40:58.206668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:32.610972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939636+0000","last_change":"2024-04-26T01:36:50.278325+0000","last_active":"2024-04-26T01:40:57.939636+0000","last_peered":"2024-04-26T01:40:57.939636+0000","last_clean":"2024-04-26T01:40:57.939636+0000","last_became_active":"2024-04-26T01:36:50.277882+0000","last_became_peered":"2024-04-26T01:36:50.277882+0000","last_unstale":"2024-04-26T01:40:57.939636+0000","last_undegraded":"2024-04-26T01:40:57.939636+0000","last_fullsized":"2024-04-26T01:40:57.939636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:06.082223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206962+0000","last_change":"2024-04-26T01:36:54.278106+0000","last_active":"2024-04-26T01:40:58.206962+0000","last_peered":"2024-04-26T01:40:58.206962+0000","last_clean":"2024-04-26T01:40:58.206962+0000","last_became_active":"2024-04-26T01:36:54.277644+0000","last_became_peered":"2024-04-26T01:36:54.277644+0000","last_unstale":"2024-04-26T01:40:58.206962+0000","last_undegraded":"2024-04-26T01:40:58.206962+0000","last_fullsized":"2024-04-26T01:40:58.206962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:10.619640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'73","reported_seq":144,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:41:00.725122+0000","last_change":"2024-04-26T01:36:50.237072+0000","last_active":"2024-04-26T01:41:00.725122+0000","last_peered":"2024-04-26T01:41:00.725122+0000","last_clean":"2024-04-26T01:41:00.725122+0000","last_became_active":"2024-04-26T01:36:50.236682+0000","last_became_peered":"2024-04-26T01:36:50.236682+0000","last_unstale":"2024-04-26T01:41:00.725122+0000","last_undegraded":"2024-04-26T01:41:00.725122+0000","last_fullsized":"2024-04-26T01:41:00.725122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:58:17.553492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":44,"num_write":84,"num_write_kb":1075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206902+0000","last_change":"2024-04-26T01:36:54.289997+0000","last_active":"2024-04-26T01:40:58.206902+0000","last_peered":"2024-04-26T01:40:58.206902+0000","last_clean":"2024-04-26T01:40:58.206902+0000","last_became_active":"2024-04-26T01:36:54.289505+0000","last_became_peered":"2024-04-26T01:36:54.289505+0000","last_unstale":"2024-04-26T01:40:58.206902+0000","last_undegraded":"2024-04-26T01:40:58.206902+0000","last_fullsized":"2024-04-26T01:40:58.206902+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:58:09.340231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.950038+0000","last_change":"2024-04-26T01:36:50.245231+0000","last_active":"2024-04-26T01:40:57.950038+0000","last_peered":"2024-04-26T01:40:57.950038+0000","last_clean":"2024-04-26T01:40:57.950038+0000","last_became_active":"2024-04-26T01:36:50.244811+0000","last_became_peered":"2024-04-26T01:36:50.244811+0000","last_unstale":"2024-04-26T01:40:57.950038+0000","last_undegraded":"2024-04-26T01:40:57.950038+0000","last_fullsized":"2024-04-26T01:40:57.950038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:49.209504+0000","last_clean_scrub_stamp":"2024-04-26T01:36:49.209504+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:12:08.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206908+0000","last_change":"2024-04-26T01:36:54.272197+0000","last_active":"2024-04-26T01:40:58.206908+0000","last_peered":"2024-04-26T01:40:58.206908+0000","last_clean":"2024-04-26T01:40:58.206908+0000","last_became_active":"2024-04-26T01:36:54.271776+0000","last_became_peered":"2024-04-26T01:36:54.271776+0000","last_unstale":"2024-04-26T01:40:58.206908+0000","last_undegraded":"2024-04-26T01:40:58.206908+0000","last_fullsized":"2024-04-26T01:40:58.206908+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:20.024890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205783+0000","last_change":"2024-04-26T01:36:54.271120+0000","last_active":"2024-04-26T01:40:58.205783+0000","last_peered":"2024-04-26T01:40:58.205783+0000","last_clean":"2024-04-26T01:40:58.205783+0000","last_became_active":"2024-04-26T01:36:54.270796+0000","last_became_peered":"2024-04-26T01:36:54.270796+0000","last_unstale":"2024-04-26T01:40:58.205783+0000","last_undegraded":"2024-04-26T01:40:58.205783+0000","last_fullsized":"2024-04-26T01:40:58.205783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:04:35.093847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939435+0000","last_change":"2024-04-26T01:36:54.273013+0000","last_active":"2024-04-26T01:40:57.939435+0000","last_peered":"2024-04-26T01:40:57.939435+0000","last_clean":"2024-04-26T01:40:57.939435+0000","last_became_active":"2024-04-26T01:36:54.272666+0000","last_became_peered":"2024-04-26T01:36:54.272666+0000","last_unstale":"2024-04-26T01:40:57.939435+0000","last_undegraded":"2024-04-26T01:40:57.939435+0000","last_fullsized":"2024-04-26T01:40:57.939435+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:45.203185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939523+0000","last_change":"2024-04-26T01:36:54.289427+0000","last_active":"2024-04-26T01:40:57.939523+0000","last_peered":"2024-04-26T01:40:57.939523+0000","last_clean":"2024-04-26T01:40:57.939523+0000","last_became_active":"2024-04-26T01:36:54.289090+0000","last_became_peered":"2024-04-26T01:36:54.289090+0000","last_unstale":"2024-04-26T01:40:57.939523+0000","last_undegraded":"2024-04-26T01:40:57.939523+0000","last_fullsized":"2024-04-26T01:40:57.939523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:18:18.544777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.938573+0000","last_change":"2024-04-26T01:36:54.294308+0000","last_active":"2024-04-26T01:40:57.938573+0000","last_peered":"2024-04-26T01:40:57.938573+0000","last_clean":"2024-04-26T01:40:57.938573+0000","last_became_active":"2024-04-26T01:36:54.291910+0000","last_became_peered":"2024-04-26T01:36:54.291910+0000","last_unstale":"2024-04-26T01:40:57.938573+0000","last_undegraded":"2024-04-26T01:40:57.938573+0000","last_fullsized":"2024-04-26T01:40:57.938573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:35:16.683265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949562+0000","last_change":"2024-04-26T01:36:54.285479+0000","last_active":"2024-04-26T01:40:57.949562+0000","last_peered":"2024-04-26T01:40:57.949562+0000","last_clean":"2024-04-26T01:40:57.949562+0000","last_became_active":"2024-04-26T01:36:54.284139+0000","last_became_peered":"2024-04-26T01:36:54.284139+0000","last_unstale":"2024-04-26T01:40:57.949562+0000","last_undegraded":"2024-04-26T01:40:57.949562+0000","last_fullsized":"2024-04-26T01:40:57.949562+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:55.506496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205871+0000","last_change":"2024-04-26T01:36:54.270567+0000","last_active":"2024-04-26T01:40:58.205871+0000","last_peered":"2024-04-26T01:40:58.205871+0000","last_clean":"2024-04-26T01:40:58.205871+0000","last_became_active":"2024-04-26T01:36:54.270218+0000","last_became_peered":"2024-04-26T01:36:54.270218+0000","last_unstale":"2024-04-26T01:40:58.205871+0000","last_undegraded":"2024-04-26T01:40:58.205871+0000","last_fullsized":"2024-04-26T01:40:58.205871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:50:43.685086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.939427+0000","last_change":"2024-04-26T01:36:54.269746+0000","last_active":"2024-04-26T01:40:57.939427+0000","last_peered":"2024-04-26T01:40:57.939427+0000","last_clean":"2024-04-26T01:40:57.939427+0000","last_became_active":"2024-04-26T01:36:54.269143+0000","last_became_peered":"2024-04-26T01:36:54.269143+0000","last_unstale":"2024-04-26T01:40:57.939427+0000","last_undegraded":"2024-04-26T01:40:57.939427+0000","last_fullsized":"2024-04-26T01:40:57.939427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:07:02.911645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953933+0000","last_change":"2024-04-26T01:36:54.279194+0000","last_active":"2024-04-26T01:40:57.953933+0000","last_peered":"2024-04-26T01:40:57.953933+0000","last_clean":"2024-04-26T01:40:57.953933+0000","last_became_active":"2024-04-26T01:36:54.278709+0000","last_became_peered":"2024-04-26T01:36:54.278709+0000","last_unstale":"2024-04-26T01:40:57.953933+0000","last_undegraded":"2024-04-26T01:40:57.953933+0000","last_fullsized":"2024-04-26T01:40:57.953933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:19.604884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.969830+0000","last_change":"2024-04-26T01:36:54.277540+0000","last_active":"2024-04-26T01:40:57.969830+0000","last_peered":"2024-04-26T01:40:57.969830+0000","last_clean":"2024-04-26T01:40:57.969830+0000","last_became_active":"2024-04-26T01:36:54.277193+0000","last_became_peered":"2024-04-26T01:36:54.277193+0000","last_unstale":"2024-04-26T01:40:57.969830+0000","last_undegraded":"2024-04-26T01:40:57.969830+0000","last_fullsized":"2024-04-26T01:40:57.969830+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:34.079094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.953565+0000","last_change":"2024-04-26T01:36:54.294593+0000","last_active":"2024-04-26T01:40:57.953565+0000","last_peered":"2024-04-26T01:40:57.953565+0000","last_clean":"2024-04-26T01:40:57.953565+0000","last_became_active":"2024-04-26T01:36:54.291894+0000","last_became_peered":"2024-04-26T01:36:54.291894+0000","last_unstale":"2024-04-26T01:40:57.953565+0000","last_undegraded":"2024-04-26T01:40:57.953565+0000","last_fullsized":"2024-04-26T01:40:57.953565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:16.715108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.206030+0000","last_change":"2024-04-26T01:36:54.292433+0000","last_active":"2024-04-26T01:40:58.206030+0000","last_peered":"2024-04-26T01:40:58.206030+0000","last_clean":"2024-04-26T01:40:58.206030+0000","last_became_active":"2024-04-26T01:36:54.291445+0000","last_became_peered":"2024-04-26T01:36:54.291445+0000","last_unstale":"2024-04-26T01:40:58.206030+0000","last_undegraded":"2024-04-26T01:40:58.206030+0000","last_fullsized":"2024-04-26T01:40:58.206030+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:49.081871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:57.949742+0000","last_change":"2024-04-26T01:36:54.291892+0000","last_active":"2024-04-26T01:40:57.949742+0000","last_peered":"2024-04-26T01:40:57.949742+0000","last_clean":"2024-04-26T01:40:57.949742+0000","last_became_active":"2024-04-26T01:36:54.291549+0000","last_became_peered":"2024-04-26T01:36:54.291549+0000","last_unstale":"2024-04-26T01:40:57.949742+0000","last_undegraded":"2024-04-26T01:40:57.949742+0000","last_fullsized":"2024-04-26T01:40:57.949742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:09:22.954651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:40:58.205662+0000","last_change":"2024-04-26T01:37:25.498138+0000","last_active":"2024-04-26T01:40:58.205662+0000","last_peered":"2024-04-26T01:40:58.205662+0000","last_clean":"2024-04-26T01:40:58.205662+0000","last_became_active":"2024-04-26T01:37:25.497681+0000","last_became_peered":"2024-04-26T01:37:25.497681+0000","last_unstale":"2024-04-26T01:40:58.205662+0000","last_undegraded":"2024-04-26T01:40:58.205662+0000","last_fullsized":"2024-04-26T01:40:58.205662+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:53.239337+0000","last_clean_scrub_stamp":"2024-04-26T01:36:53.239337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:03:25.016623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107964,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799569,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1888,"num_read_kb":1214,"num_write":484,"num_write_kb":2626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2398707,"data_compressed":108978,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383749,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488830,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645385,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36512,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708640,"statfs":{"total":95995035648,"available":95957647360,"internally_reserved":0,"allocated":1196032,"data_stored":1481718,"data_compressed":55907,"data_compressed_allocated":73728,"data_compressed_original":913408,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":882260,"data_compressed":22185,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070549,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":882158,"data_compressed":22185,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332187,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":1082518,"data_compressed":38877,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495712,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488433,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":562658,"data_compressed":4717,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738499,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":489704,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411450,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":561338,"data_compressed":4717,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182513,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":1210192,"data_compressed":37387,"data_compressed_allocated":49152,"data_compressed_original":708608,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920876,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":489683,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757352,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":1409733,"data_compressed":54079,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35988,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":74432,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":73112,"data_compressed":1828,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34498,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":331022,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19296,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19296,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":403007,"data_compressed":17030,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:41:12.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:13.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:13 smithi169 ceph-mon[32851]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:13.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:13 smithi169 ceph-mon[32851]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 61 op/s 2024-04-26T01:41:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:13 smithi053 ceph-mon[32593]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:13 smithi053 ceph-mon[32593]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 61 op/s 2024-04-26T01:41:13.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:13 smithi028 ceph-mon[28247]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:41:13.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:13 smithi028 ceph-mon[28247]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 61 op/s 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 -- 172.21.15.28:0/298267237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4101aa0 msgr2=0x7f16a4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 --2- 172.21.15.28:0/298267237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4101aa0 0x7f16a4105a90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1690009900 tx=0x7f169002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 -- 172.21.15.28:0/298267237 shutdown_connections 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 --2- 172.21.15.28:0/298267237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a4106170 0x7f16a410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 --2- 172.21.15.28:0/298267237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4101aa0 0x7f16a4105a90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 --2- 172.21.15.28:0/298267237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16a41010f0 0x7f16a41014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 -- 172.21.15.28:0/298267237 >> 172.21.15.28:0/298267237 conn(0x7f16a4077fd0 msgr2=0x7f16a40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 -- 172.21.15.28:0/298267237 shutdown_connections 2024-04-26T01:41:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.058+0000 7f16ac000640 1 -- 172.21.15.28:0/298267237 wait complete. 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.059+0000 7f16ac000640 1 Processor -- start 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.059+0000 7f16ac000640 1 -- start start 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16ac000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a41010f0 0x7f16a41a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16ac000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16a4106170 0x7f16a41a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16ac000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a419a700 0x7f16a419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16ac000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a4111100 con 0x7f16a41010f0 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16ac000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f16a4110f80 con 0x7f16a419a700 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16ac000640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f16a4111280 con 0x7f16a4106170 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16a9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a41010f0 0x7f16a41a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16a9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a41010f0 0x7f16a41a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36344/0 (socket says 172.21.15.28:36344) 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16a9600640 1 -- 172.21.15.28:0/4204777312 learned_addr learned my addr 172.21.15.28:0/4204777312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:14.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16a8c00640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16a4106170 0x7f16a41a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.060+0000 7f16aa000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a419a700 0x7f16a419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a9600640 1 -- 172.21.15.28:0/4204777312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16a4106170 msgr2=0x7f16a41a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a9600640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16a4106170 0x7f16a41a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a9600640 1 -- 172.21.15.28:0/4204777312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a419a700 msgr2=0x7f16a419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a9600640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a419a700 0x7f16a419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a9600640 1 -- 172.21.15.28:0/4204777312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a419b300 con 0x7f16a41010f0 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16aa000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a419a700 0x7f16a419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a9600640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a41010f0 0x7f16a41a0640 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f169800e9f0 tx=0x7f169800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a2000640 1 -- 172.21.15.28:0/4204777312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169800cd80 con 0x7f16a41010f0 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a2000640 1 -- 172.21.15.28:0/4204777312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16980054d0 con 0x7f16a41010f0 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16a2000640 1 -- 172.21.15.28:0/4204777312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698010690 con 0x7f16a41010f0 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16ac000640 1 -- 172.21.15.28:0/4204777312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a41a73e0 con 0x7f16a41010f0 2024-04-26T01:41:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.061+0000 7f16ac000640 1 -- 172.21.15.28:0/4204777312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a41a7790 con 0x7f16a41010f0 2024-04-26T01:41:14.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16aed82740 1 Processor -- start 2024-04-26T01:41:14.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16aed82740 1 -- start start 2024-04-26T01:41:14.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16a2000640 1 -- 172.21.15.28:0/4204777312 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f1698004e90 con 0x7f16a41010f0 2024-04-26T01:41:14.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16aed82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b46c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16aed82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b514c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16aed82740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b51a00 0x55f546b53df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16aed82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f546aac900 con 0x55f546b47140 2024-04-26T01:41:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16aed82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f546a52ef0 con 0x55f546b46820 2024-04-26T01:41:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.065+0000 7f16aed82740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f546aa88c0 con 0x55f546b51a00 2024-04-26T01:41:14.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a8c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b51a00 0x55f546b53df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a8c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b51a00 0x55f546b53df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54876/0 (socket says 172.21.15.28:54876) 2024-04-26T01:41:14.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a8c00640 1 -- 172.21.15.28:0/2161276895 learned_addr learned my addr 172.21.15.28:0/2161276895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36360/0 (socket says 172.21.15.28:36360) 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16aa000640 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b46c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a9600640 1 -- 172.21.15.28:0/2161276895 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b51a00 msgr2=0x55f546b53df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a9600640 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b51a00 0x55f546b53df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a9600640 1 -- 172.21.15.28:0/2161276895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 msgr2=0x55f546b46c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a9600640 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b46c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a9600640 1 -- 172.21.15.28:0/2161276895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5467bf210 con 0x55f546b47140 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16a8c00640 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b51a00 0x55f546b53df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.066+0000 7f16aa000640 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b46c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.067+0000 7f16a9600640 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b514c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f16980ce520 tx=0x7f16980d4f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ba257960fc5575 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.067+0000 7f1696600640 1 -- 172.21.15.28:0/2161276895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16980e4070 con 0x55f546b47140 2024-04-26T01:41:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.067+0000 7f1696600640 1 -- 172.21.15.28:0/2161276895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16980d9c80 con 0x55f546b47140 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.067+0000 7f1696600640 1 -- 172.21.15.28:0/2161276895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16980de5e0 con 0x55f546b47140 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.067+0000 7f16a2000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f168c0a3ee0 0x7f168c0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.067+0000 7f16a2000640 1 -- 172.21.15.28:0/4204777312 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f16980d0f80 con 0x7f16a41010f0 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.067+0000 7f16aa000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f168c0a3ee0 0x7f168c0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 -- 172.21.15.28:0/2161276895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 msgr2=0x55f546b514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b514c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f16980ce520 tx=0x7f16980d4f70 comp rx=0 tx=0).stop 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 -- 172.21.15.28:0/2161276895 shutdown_connections 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b51a00 0x55f546b53df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b514c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 --2- 172.21.15.28:0/2161276895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b46c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 -- 172.21.15.28:0/2161276895 >> 172.21.15.28:0/2161276895 conn(0x55f546b42d30 msgr2=0x55f546b43110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 -- 172.21.15.28:0/2161276895 shutdown_connections 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aa000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f168c0a3ee0 0x7f168c0a63a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f169c005fd0 tx=0x7f169c009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.068+0000 7f16aed82740 1 -- 172.21.15.28:0/2161276895 wait complete. 2024-04-26T01:41:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 1 -- start start 2024-04-26T01:41:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 1 Processor -- start 2024-04-26T01:41:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b46820 0x55f546b4e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b4ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b4f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f546b54630 con 0x55f546b46820 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f546a66310 con 0x55f546b51a00 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f546b547b0 con 0x55f546b47140 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.069+0000 7f16aed82740 10 client.0 _ll_register_callbacks cb 0x55f546a34cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b46820 0x55f546b4e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b46820 0x55f546b4e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36372/0 (socket says 172.21.15.28:36372) 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a8c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b4ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 -- 172.21.15.28:0/188861222 learned_addr learned my addr 172.21.15.28:0/188861222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16aa000640 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b4f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 -- 172.21.15.28:0/188861222 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 msgr2=0x55f546b4ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b4ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 -- 172.21.15.28:0/188861222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 msgr2=0x55f546b4f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b4f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 -- 172.21.15.28:0/188861222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54688b620 con 0x55f546b46820 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16aa000640 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b4f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f16a9600640 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b46820 0x55f546b4e940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1698099100 tx=0x7f16980d1200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da14d68233ce3969 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.070+0000 7f168aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f168aa00640 1 -- 172.21.15.28:0/188861222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16980bbe60 con 0x55f546b46820 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f168aa00640 1 -- 172.21.15.28:0/188861222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f16980dfb60 con 0x55f546b46820 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f16aed82740 1 -- 172.21.15.28:0/188861222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16980d9c80 con 0x55f546b46820 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f168aa00640 1 -- 172.21.15.28:0/188861222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1698098f40 con 0x55f546b46820 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f16aed82740 10 client.15051 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f16aed82740 1 -- 172.21.15.28:0/188861222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f546b54630 con 0x55f546b46820 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f168aa00640 1 -- 172.21.15.28:0/188861222 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16980d9c80 con 0x55f546b46820 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f168aa00640 1 -- 172.21.15.28:0/188861222 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f16980bd0b0 con 0x55f546b46820 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f16aed82740 10 client.15051 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f16aed82740 10 client.15051 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.071+0000 7f16aed82740 20 client.15051 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.072+0000 7f16aed82740 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f546b619c0 0x55f546b63db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.072+0000 7f16aed82740 4 client.15051 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.072+0000 7f16aed82740 1 -- 172.21.15.28:0/188861222 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f546ac08b0 con 0x55f546b619c0 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.072+0000 7f16a8c00640 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f546b619c0 0x55f546b63db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.072+0000 7f16a8c00640 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f546b619c0 0x55f546b63db0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=668c19c755b1f9f4 server_cookie=ec05df35baeeaa33 in_seq=0 out_seq=0 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.072+0000 7f168aa00640 10 client.15051 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.073+0000 7f168aa00640 1 -- 172.21.15.28:0/188861222 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f546ac08b0 con 0x55f546b619c0 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f168aa00640 10 client.15051 handle_command_reply: tid=0 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 client.15051 shutdown 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 -- 172.21.15.28:0/188861222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b46820 msgr2=0x55f546b4e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b46820 0x55f546b4e940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1698099100 tx=0x7f16980d1200 comp rx=0 tx=0).stop 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 -- 172.21.15.28:0/188861222 shutdown_connections 2024-04-26T01:41:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f546b619c0 0x55f546b63db0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b4f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b4ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f168aa00640 0 client.15051 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 --2- 172.21.15.28:0/188861222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b46820 0x55f546b4e940 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 -- 172.21.15.28:0/188861222 >> 172.21.15.28:0/188861222 conn(0x55f546b42d30 msgr2=0x55f546b48ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 -- 172.21.15.28:0/188861222 shutdown_connections 2024-04-26T01:41:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 1 -- 172.21.15.28:0/188861222 wait complete. 2024-04-26T01:41:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.074+0000 7f16aed82740 20 client.15051 trim_cache size 0 max 16384 2024-04-26T01:41:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.090+0000 7f16aed82740 1 Processor -- start 2024-04-26T01:41:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aed82740 1 -- start start 2024-04-26T01:41:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aed82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b5d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aed82740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b491a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aed82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b51a00 0x55f546b4f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aed82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41b2580 con 0x55f546b51a00 2024-04-26T01:41:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aed82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41afe50 con 0x55f546b46820 2024-04-26T01:41:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aed82740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41adca0 con 0x55f546b47140 2024-04-26T01:41:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b51a00 0x55f546b4f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b51a00 0x55f546b4f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36388/0 (socket says 172.21.15.28:36388) 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16aa000640 1 -- 172.21.15.28:0/1977796336 learned_addr learned my addr 172.21.15.28:0/1977796336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16a8c00640 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.091+0000 7f16a9600640 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b5d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f16a8c00640 1 -- 172.21.15.28:0/1977796336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 msgr2=0x55f546b5d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f16a8c00640 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b5d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f16a8c00640 1 -- 172.21.15.28:0/1977796336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b51a00 msgr2=0x55f546b4f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f16a8c00640 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b51a00 0x55f546b4f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f16a8c00640 1 -- 172.21.15.28:0/1977796336 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f546b4ce90 con 0x55f546b47140 2024-04-26T01:41:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f16a8c00640 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b491a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1690005ab0 tx=0x7f1690005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17fa8e87a55d2590 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f1695c00640 1 -- 172.21.15.28:0/1977796336 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690035040 con 0x55f546b47140 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f1695c00640 1 -- 172.21.15.28:0/1977796336 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1690046dc0 con 0x55f546b47140 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.092+0000 7f1695c00640 1 -- 172.21.15.28:0/1977796336 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690005560 con 0x55f546b47140 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 -- 172.21.15.28:0/1977796336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 msgr2=0x55f546b491a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b491a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1690005ab0 tx=0x7f1690005a20 comp rx=0 tx=0).stop 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 -- 172.21.15.28:0/1977796336 shutdown_connections 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b51a00 0x55f546b4f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b47140 0x55f546b491a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 --2- 172.21.15.28:0/1977796336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b46820 0x55f546b5d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 -- 172.21.15.28:0/1977796336 >> 172.21.15.28:0/1977796336 conn(0x55f546b42d30 msgr2=0x55f546b4e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 -- 172.21.15.28:0/1977796336 shutdown_connections 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.093+0000 7f16aed82740 1 -- 172.21.15.28:0/1977796336 wait complete. 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.094+0000 7f16aed82740 1 -- start start 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.094+0000 7f16aed82740 1 Processor -- start 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.094+0000 7f16aed82740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b46820 0x55f546b57370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.094+0000 7f16aed82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b578b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.094+0000 7f16aed82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b57df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.094+0000 7f16aed82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41ae310 con 0x55f546b47140 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.094+0000 7f16aed82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41af780 con 0x55f546b51a00 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.094+0000 7f16aed82740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41ae9e0 con 0x55f546b46820 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16aed82740 10 client.0 _ll_register_callbacks cb 0x7f1698005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a9600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b46820 0x55f546b57370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a9600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b46820 0x55f546b57370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54904/0 (socket says 172.21.15.28:54904) 2024-04-26T01:41:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a9600640 1 -- 172.21.15.28:0/1065857761 learned_addr learned my addr 172.21.15.28:0/1065857761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16aa000640 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b57df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a8c00640 1 -- 172.21.15.28:0/1065857761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b46820 msgr2=0x55f546b57370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a8c00640 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b46820 0x55f546b57370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a8c00640 1 -- 172.21.15.28:0/1065857761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 msgr2=0x55f546b57df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a8c00640 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b57df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a8c00640 1 -- 172.21.15.28:0/1065857761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f546bc6c90 con 0x55f546b47140 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f16a8c00640 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b578b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55f546b47ca0 tx=0x7f1690004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a392af87d83bc8a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.095+0000 7f168aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.096+0000 7f168aa00640 1 -- 172.21.15.28:0/1065857761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690035070 con 0x55f546b47140 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.096+0000 7f16aed82740 1 -- 172.21.15.28:0/1065857761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1690046dc0 con 0x55f546b47140 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.096+0000 7f168aa00640 1 -- 172.21.15.28:0/1065857761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f16900054a0 con 0x55f546b47140 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.096+0000 7f168aa00640 1 -- 172.21.15.28:0/1065857761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1690046340 con 0x55f546b47140 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.096+0000 7f16aed82740 10 client.15057 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.096+0000 7f16aed82740 1 -- 172.21.15.28:0/1065857761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f546bc7ec0 con 0x55f546b47140 2024-04-26T01:41:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.096+0000 7f168aa00640 1 -- 172.21.15.28:0/1065857761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1690046dc0 con 0x55f546b47140 2024-04-26T01:41:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.097+0000 7f168aa00640 1 -- 172.21.15.28:0/1065857761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f1690004bd0 con 0x55f546b47140 2024-04-26T01:41:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.097+0000 7f16aed82740 10 client.15057 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.097+0000 7f16aed82740 10 client.15057 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:41:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.097+0000 7f16aed82740 20 client.15057 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.097+0000 7f16aed82740 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f546bc85f0 0x55f546bcaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.097+0000 7f16aed82740 4 client.15057 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:41:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.097+0000 7f16aed82740 1 -- 172.21.15.28:0/1065857761 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f546aba320 con 0x55f546bc85f0 2024-04-26T01:41:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.097+0000 7f16a9600640 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f546bc85f0 0x55f546bcaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.098+0000 7f16a9600640 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f546bc85f0 0x55f546bcaa10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a299da2abb1f83c3 server_cookie=e879f3869ae1f8fa in_seq=0 out_seq=0 2024-04-26T01:41:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.098+0000 7f168aa00640 10 client.15057 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.110+0000 7f168aa00640 1 -- 172.21.15.28:0/1065857761 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f546aba320 con 0x55f546bc85f0 2024-04-26T01:41:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.110+0000 7f168aa00640 10 client.15057 handle_command_reply: tid=0 2024-04-26T01:41:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.110+0000 7f16aed82740 1 client.15057 shutdown 2024-04-26T01:41:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.110+0000 7f16aed82740 1 -- 172.21.15.28:0/1065857761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 msgr2=0x55f546b578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.110+0000 7f16aed82740 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b578b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55f546b47ca0 tx=0x7f1690004600 comp rx=0 tx=0).stop 2024-04-26T01:41:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.110+0000 7f16aed82740 1 -- 172.21.15.28:0/1065857761 shutdown_connections 2024-04-26T01:41:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f16aed82740 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f546bc85f0 0x55f546bcaa10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f16aed82740 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f546b51a00 0x55f546b57df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f16aed82740 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f546b47140 0x55f546b578b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f16aed82740 1 --2- 172.21.15.28:0/1065857761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f546b46820 0x55f546b57370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f168aa00640 0 client.15057 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f16aed82740 1 -- 172.21.15.28:0/1065857761 >> 172.21.15.28:0/1065857761 conn(0x55f546b42d30 msgr2=0x55f546b4f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f16aed82740 1 -- 172.21.15.28:0/1065857761 shutdown_connections 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f16aed82740 1 -- 172.21.15.28:0/1065857761 wait complete. 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.111+0000 7f16aed82740 20 client.15057 trim_cache size 0 max 16384 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:41:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:41:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:41:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 -- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f168c0a3ee0 msgr2=0x7f168c0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f168c0a3ee0 0x7f168c0a63a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f169c005fd0 tx=0x7f169c009500 comp rx=0 tx=0).stop 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 -- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a41010f0 msgr2=0x7f16a41a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a41010f0 0x7f16a41a0640 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f169800e9f0 tx=0x7f169800edf0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 -- 172.21.15.28:0/4204777312 shutdown_connections 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f168c0a3ee0 0x7f168c0a63a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a419a700 0x7f16a419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16a4106170 0x7f16a41a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 --2- 172.21.15.28:0/4204777312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a41010f0 0x7f16a41a0640 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 -- 172.21.15.28:0/4204777312 >> 172.21.15.28:0/4204777312 conn(0x7f16a4077fd0 msgr2=0x7f16a41000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 -- 172.21.15.28:0/4204777312 shutdown_connections 2024-04-26T01:41:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:14.112+0000 7f16ac000640 1 -- 172.21.15.28:0/4204777312 wait complete. 2024-04-26T01:41:15.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:15 smithi028 ceph-mon[28247]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 98 KiB/s wr, 63 op/s 2024-04-26T01:41:15.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:15 smithi169 ceph-mon[32851]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 98 KiB/s wr, 63 op/s 2024-04-26T01:41:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:15 smithi053 ceph-mon[32593]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 98 KiB/s wr, 63 op/s 2024-04-26T01:41:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 -- 172.21.15.28:0/3835434641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc108a30 msgr2=0x7f1ffc108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 --2- 172.21.15.28:0/3835434641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc108a30 0x7f1ffc108e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009990 tx=0x7f1fec009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 -- 172.21.15.28:0/3835434641 shutdown_connections 2024-04-26T01:41:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 --2- 172.21.15.28:0/3835434641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc06b0f0 0x7f1ffc10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 --2- 172.21.15.28:0/3835434641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc108a30 0x7f1ffc108e10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 --2- 172.21.15.28:0/3835434641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc06a750 0x7f1ffc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 -- 172.21.15.28:0/3835434641 >> 172.21.15.28:0/3835434641 conn(0x7f1ffc0788f0 msgr2=0x7f1ffc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:15.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 -- 172.21.15.28:0/3835434641 shutdown_connections 2024-04-26T01:41:15.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.852+0000 7f2001c00700 1 -- 172.21.15.28:0/3835434641 wait complete. 2024-04-26T01:41:15.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.853+0000 7f2001c00700 1 Processor -- start 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.853+0000 7f2001c00700 1 -- start start 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f2001c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc06a750 0x7f1ffc101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f2001c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc06b0f0 0x7f1ffc101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f2001c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc108a30 0x7f1ffc102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f2001c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc112bb0 con 0x7f1ffc06a750 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f2001c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc112a30 con 0x7f1ffc108a30 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f2001c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc112d30 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f1ff2000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc06b0f0 0x7f1ffc101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f1ffaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc06a750 0x7f1ffc101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f1ff2000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc06b0f0 0x7f1ffc101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54924/0 (socket says 172.21.15.28:54924) 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f1ff2000700 1 -- 172.21.15.28:0/2542834529 learned_addr learned my addr 172.21.15.28:0/2542834529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.854+0000 7f1ffb400700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc108a30 0x7f1ffc102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ff2000700 1 -- 172.21.15.28:0/2542834529 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc108a30 msgr2=0x7f1ffc102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ff2000700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc108a30 0x7f1ffc102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ff2000700 1 -- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc06a750 msgr2=0x7f1ffc101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ff2000700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc06a750 0x7f1ffc101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ff2000700 1 -- 172.21.15.28:0/2542834529 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc1aba90 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ffb400700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc108a30 0x7f1ffc102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ffaa00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc06a750 0x7f1ffc101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ff2000700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc06b0f0 0x7f1ffc101f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400d980 tx=0x7f1fe400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ff3e00700 1 -- 172.21.15.28:0/2542834529 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4016070 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.855+0000 7f1ff3e00700 1 -- 172.21.15.28:0/2542834529 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fe4004830 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.856+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc1abd20 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.856+0000 7f1ff3e00700 1 -- 172.21.15.28:0/2542834529 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe40055e0 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.856+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc1ac1b0 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.857+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ffc104170 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.859+0000 7f1ff3e00700 1 -- 172.21.15.28:0/2542834529 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f1fe4004220 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.860+0000 7f1ff3e00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.861+0000 7f1ff3e00700 1 -- 172.21.15.28:0/2542834529 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f1fe40d1260 con 0x7f1ffc06b0f0 2024-04-26T01:41:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.861+0000 7f1ffaa00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.861+0000 7f1ffaa00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0098e0 tx=0x7f1fec005960 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:15.863+0000 7f1ff3e00700 1 -- 172.21.15.28:0/2542834529 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1fe401f020 con 0x7f1ffc06b0f0 2024-04-26T01:41:16.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.143+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ffc02e140 con 0x7f1ffc06b0f0 2024-04-26T01:41:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.145+0000 7f1ff3e00700 1 -- 172.21.15.28:0/2542834529 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11861 (secure 0 0 0) 0x7f1fe4047360 con 0x7f1ffc06b0f0 2024-04-26T01:41:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:16.143 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:40:57.900766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24509,24523]},"id":1}]} 2024-04-26T01:41:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1fe00a3ed0 msgr2=0x7f1fe00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0098e0 tx=0x7f1fec005960 comp rx=0 tx=0).stop 2024-04-26T01:41:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc06b0f0 msgr2=0x7f1ffc101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc06b0f0 0x7f1ffc101f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400d980 tx=0x7f1fe400dc90 comp rx=0 tx=0).stop 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 shutdown_connections 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc108a30 0x7f1ffc102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc06a750 0x7f1ffc101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 --2- 172.21.15.28:0/2542834529 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc06b0f0 0x7f1ffc101f50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 >> 172.21.15.28:0/2542834529 conn(0x7f1ffc0788f0 msgr2=0x7f1ffc0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 shutdown_connections 2024-04-26T01:41:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:16.148+0000 7f2001c00700 1 -- 172.21.15.28:0/2542834529 wait complete. 2024-04-26T01:41:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-26T01:41:16.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:16 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2542834529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:16.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:16 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2542834529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:16.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:16 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2542834529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:16.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:17.452 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:17.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:17 smithi028 ceph-mon[28247]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 62 op/s 2024-04-26T01:41:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:17 smithi169 ceph-mon[32851]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 62 op/s 2024-04-26T01:41:17.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:17 smithi053 ceph-mon[32593]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 62 op/s 2024-04-26T01:41:19.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:19 smithi028 ceph-mon[28247]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:41:19.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:19 smithi169 ceph-mon[32851]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:41:19.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:19 smithi053 ceph-mon[32593]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 -- 172.21.15.28:0/1754486327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a5010e3b0 msgr2=0x7f7a50115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 --2- 172.21.15.28:0/1754486327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a5010e3b0 0x7f7a50115f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7a480099c0 tx=0x7f7a48009cd0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 -- 172.21.15.28:0/1754486327 shutdown_connections 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 --2- 172.21.15.28:0/1754486327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a5010e3b0 0x7f7a50115f80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 --2- 172.21.15.28:0/1754486327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a501092d0 0x7f7a501096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 --2- 172.21.15.28:0/1754486327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a50109c80 0x7f7a5010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 -- 172.21.15.28:0/1754486327 >> 172.21.15.28:0/1754486327 conn(0x7f7a50076210 msgr2=0x7f7a50107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 -- 172.21.15.28:0/1754486327 shutdown_connections 2024-04-26T01:41:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.886+0000 7f7a56600700 1 -- 172.21.15.28:0/1754486327 wait complete. 2024-04-26T01:41:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.887+0000 7f7a56600700 1 Processor -- start 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a56600700 1 -- start start 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a56600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a501092d0 0x7f7a5019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a56600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a50109c80 0x7f7a5019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a56600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a5010e3b0 0x7f7a501a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a56600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a50119250 con 0x7f7a501092d0 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a56600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501190d0 con 0x7f7a50109c80 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a56600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7a501193d0 con 0x7f7a5010e3b0 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a4fe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a5010e3b0 0x7f7a501a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a4f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a501092d0 0x7f7a5019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a4fe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a5010e3b0 0x7f7a501a3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58056/0 (socket says 172.21.15.28:58056) 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a4fe00700 1 -- 172.21.15.28:0/398447099 learned_addr learned my addr 172.21.15.28:0/398447099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.888+0000 7f7a47e00700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a50109c80 0x7f7a5019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.889+0000 7f7a4fe00700 1 -- 172.21.15.28:0/398447099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a50109c80 msgr2=0x7f7a5019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.889+0000 7f7a4fe00700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a50109c80 0x7f7a5019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.889+0000 7f7a4fe00700 1 -- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a501092d0 msgr2=0x7f7a5019f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.889+0000 7f7a4fe00700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a501092d0 0x7f7a5019f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.889+0000 7f7a4fe00700 1 -- 172.21.15.28:0/398447099 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a501a46b0 con 0x7f7a5010e3b0 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.889+0000 7f7a4f400700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a501092d0 0x7f7a5019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.889+0000 7f7a4fe00700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a5010e3b0 0x7f7a501a3f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a48005910 tx=0x7f7a480055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.890+0000 7f7a4cc00700 1 -- 172.21.15.28:0/398447099 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a48026070 con 0x7f7a5010e3b0 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.890+0000 7f7a4cc00700 1 -- 172.21.15.28:0/398447099 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a4800be60 con 0x7f7a5010e3b0 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.890+0000 7f7a4cc00700 1 -- 172.21.15.28:0/398447099 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4800fa60 con 0x7f7a5010e3b0 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.890+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a501a4940 con 0x7f7a5010e3b0 2024-04-26T01:41:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.890+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a501ac220 con 0x7f7a5010e3b0 2024-04-26T01:41:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.891+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a50069730 con 0x7f7a5010e3b0 2024-04-26T01:41:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.893+0000 7f7a4cc00700 1 -- 172.21.15.28:0/398447099 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f7a48029070 con 0x7f7a5010e3b0 2024-04-26T01:41:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.895+0000 7f7a4cc00700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7a340a3f50 0x7f7a340a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.895+0000 7f7a4cc00700 1 -- 172.21.15.28:0/398447099 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f7a480d1980 con 0x7f7a5010e3b0 2024-04-26T01:41:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.895+0000 7f7a4f400700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7a340a3f50 0x7f7a340a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.896+0000 7f7a4f400700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7a340a3f50 0x7f7a340a6410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009480 tx=0x7f7a4000b040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:19.897+0000 7f7a4cc00700 1 -- 172.21.15.28:0/398447099 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a48030020 con 0x7f7a5010e3b0 2024-04-26T01:41:20.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.182+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a5002df80 con 0x7f7a5010e3b0 2024-04-26T01:41:20.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.184+0000 7f7a4cc00700 1 -- 172.21.15.28:0/398447099 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11861 (secure 0 0 0) 0x7f7a48013070 con 0x7f7a5010e3b0 2024-04-26T01:41:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:20.181 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:40:57.900766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24509,24523]},"id":1}]} 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7a340a3f50 msgr2=0x7f7a340a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7a340a3f50 0x7f7a340a6410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009480 tx=0x7f7a4000b040 comp rx=0 tx=0).stop 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a5010e3b0 msgr2=0x7f7a501a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a5010e3b0 0x7f7a501a3f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a48005910 tx=0x7f7a480055f0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 shutdown_connections 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7a340a3f50 0x7f7a340a6410 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a5010e3b0 0x7f7a501a3f70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a501092d0 0x7f7a5019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 --2- 172.21.15.28:0/398447099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a50109c80 0x7f7a5019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.187+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 >> 172.21.15.28:0/398447099 conn(0x7f7a50076210 msgr2=0x7f7a50076790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.188+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 shutdown_connections 2024-04-26T01:41:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:20.188+0000 7f7a56600700 1 -- 172.21.15.28:0/398447099 wait complete. 2024-04-26T01:41:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-26T01:41:20.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:20 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/398447099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:20.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:20 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/398447099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:20.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:20 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/398447099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:21.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:41:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:21.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:21 smithi169 ceph-mon[32851]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-26T01:41:21.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:21 smithi053 ceph-mon[32593]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-26T01:41:21.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:21 smithi028 ceph-mon[28247]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-26T01:41:23.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:23 smithi028 ceph-mon[28247]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:23.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:23 smithi053 ceph-mon[32593]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 -- 172.21.15.28:0/1199968261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8109fa0 msgr2=0x7f1df8111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 --2- 172.21.15.28:0/1199968261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8109fa0 0x7f1df8111b70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1df000b0d0 tx=0x7f1df000b3e0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 -- 172.21.15.28:0/1199968261 shutdown_connections 2024-04-26T01:41:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 --2- 172.21.15.28:0/1199968261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8109fa0 0x7f1df8111b70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 --2- 172.21.15.28:0/1199968261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1df8104ec0 0x7f1df81052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 --2- 172.21.15.28:0/1199968261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df8105870 0x7f1df81098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 -- 172.21.15.28:0/1199968261 >> 172.21.15.28:0/1199968261 conn(0x7f1df8100710 msgr2=0x7f1df8102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 -- 172.21.15.28:0/1199968261 shutdown_connections 2024-04-26T01:41:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.855+0000 7f1dff200700 1 -- 172.21.15.28:0/1199968261 wait complete. 2024-04-26T01:41:23.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.856+0000 7f1dff200700 1 Processor -- start 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.856+0000 7f1dff200700 1 -- start start 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1dff200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df8104ec0 0x7f1df819b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1dff200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8105870 0x7f1df819b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1dff200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1df8109fa0 0x7f1df819fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1dff200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8114e30 con 0x7f1df8105870 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1dff200700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8114cb0 con 0x7f1df8104ec0 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1dff200700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8114fb0 con 0x7f1df8109fa0 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1df7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8105870 0x7f1df819b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1df7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8105870 0x7f1df819b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38874/0 (socket says 172.21.15.28:38874) 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1df7400700 1 -- 172.21.15.28:0/1977114126 learned_addr learned my addr 172.21.15.28:0/1977114126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1dfd200700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1df8109fa0 0x7f1df819fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.857+0000 7f1df7e00700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df8104ec0 0x7f1df819b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.858+0000 7f1df7400700 1 -- 172.21.15.28:0/1977114126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1df8109fa0 msgr2=0x7f1df819fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.858+0000 7f1df7400700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1df8109fa0 0x7f1df819fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.858+0000 7f1df7400700 1 -- 172.21.15.28:0/1977114126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df8104ec0 msgr2=0x7f1df819b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.858+0000 7f1df7400700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df8104ec0 0x7f1df819b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.858+0000 7f1df7400700 1 -- 172.21.15.28:0/1977114126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df81a02b0 con 0x7f1df8105870 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.858+0000 7f1df7400700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8105870 0x7f1df819b7e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1de400ea30 tx=0x7f1de400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.858+0000 7f1df4c00700 1 -- 172.21.15.28:0/1977114126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de400cbd0 con 0x7f1df8105870 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.859+0000 7f1df4c00700 1 -- 172.21.15.28:0/1977114126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1de40044d0 con 0x7f1df8105870 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.859+0000 7f1df4c00700 1 -- 172.21.15.28:0/1977114126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de4003dc0 con 0x7f1df8105870 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.859+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df81a05a0 con 0x7f1df8105870 2024-04-26T01:41:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.859+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df81a7e20 con 0x7f1df8105870 2024-04-26T01:41:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.860+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df8069730 con 0x7f1df8105870 2024-04-26T01:41:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.862+0000 7f1df4c00700 1 -- 172.21.15.28:0/1977114126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f1de4003f80 con 0x7f1df8105870 2024-04-26T01:41:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.863+0000 7f1df4c00700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ddc0a3ed0 0x7f1ddc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.864+0000 7f1df4c00700 1 -- 172.21.15.28:0/1977114126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f1de40d0b10 con 0x7f1df8105870 2024-04-26T01:41:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.864+0000 7f1df7e00700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ddc0a3ed0 0x7f1ddc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.865+0000 7f1df7e00700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ddc0a3ed0 0x7f1ddc0a6390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1de8005950 tx=0x7f1de800b410 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:23.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:23.869+0000 7f1df4c00700 1 -- 172.21.15.28:0/1977114126 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1de409cf30 con 0x7f1df8105870 2024-04-26T01:41:23.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:23 smithi169 ceph-mon[32851]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:24.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.104+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1df802df80 con 0x7f1df8105870 2024-04-26T01:41:24.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.108+0000 7f1df4c00700 1 -- 172.21.15.28:0/1977114126 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23007 (secure 0 0 0) 0x7f1de409c830 con 0x7f1df8105870 2024-04-26T01:41:24.107 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:24.107 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:40:57.926724+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":7,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3431203710":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/1295766352":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3441292878":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6800/3431203710":"2024-04-27T01:40:57.926618+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/218486566":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3010978828":"2024-04-27T01:40:57.926618+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ddc0a3ed0 msgr2=0x7f1ddc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ddc0a3ed0 0x7f1ddc0a6390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1de8005950 tx=0x7f1de800b410 comp rx=0 tx=0).stop 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8105870 msgr2=0x7f1df819b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8105870 0x7f1df819b7e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1de400ea30 tx=0x7f1de400ed40 comp rx=0 tx=0).stop 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 shutdown_connections 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ddc0a3ed0 0x7f1ddc0a6390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1df8109fa0 0x7f1df819fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df8104ec0 0x7f1df819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 --2- 172.21.15.28:0/1977114126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1df8105870 0x7f1df819b7e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 >> 172.21.15.28:0/1977114126 conn(0x7f1df8100710 msgr2=0x7f1df81029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.113+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 shutdown_connections 2024-04-26T01:41:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:24.114+0000 7f1dff200700 1 -- 172.21.15.28:0/1977114126 wait complete. 2024-04-26T01:41:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:24 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1977114126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:24.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:24 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1977114126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:24.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:24 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1977114126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:25.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-04-26T01:41:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:25 smithi028 ceph-mon[28247]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:41:25.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:25 smithi053 ceph-mon[32593]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:41:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:25 smithi169 ceph-mon[32851]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:41:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 -- 172.21.15.28:0/2384925248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 msgr2=0x7f9528110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 --2- 172.21.15.28:0/2384925248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 0x7f9528110e90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9520009ca0 tx=0x7f952002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 -- 172.21.15.28:0/2384925248 shutdown_connections 2024-04-26T01:41:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 --2- 172.21.15.28:0/2384925248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 0x7f9528110e90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 --2- 172.21.15.28:0/2384925248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9528104ba0 0x7f9528108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 --2- 172.21.15.28:0/2384925248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95281041f0 0x7f95281045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 -- 172.21.15.28:0/2384925248 >> 172.21.15.28:0/2384925248 conn(0x7f95280ffa30 msgr2=0x7f9528101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 -- 172.21.15.28:0/2384925248 shutdown_connections 2024-04-26T01:41:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.260+0000 7f9527e00640 1 -- 172.21.15.28:0/2384925248 wait complete. 2024-04-26T01:41:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.262+0000 7f9527e00640 1 Processor -- start 2024-04-26T01:41:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.262+0000 7f9527e00640 1 -- start start 2024-04-26T01:41:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9527e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95281041f0 0x7f95281a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9527e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 0x7f95281a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9527e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f952819a620 0x7f952819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9527e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9528114260 con 0x7f95281092c0 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9527e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f95281140e0 con 0x7f952819a620 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9527e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f95281143e0 con 0x7f95281041f0 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9526000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 0x7f95281a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9526a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95281041f0 0x7f95281a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9526000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 0x7f95281a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38882/0 (socket says 172.21.15.28:38882) 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9526000640 1 -- 172.21.15.28:0/245781620 learned_addr learned my addr 172.21.15.28:0/245781620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.263+0000 7f9527400640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f952819a620 0x7f952819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9526a00640 1 -- 172.21.15.28:0/245781620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f952819a620 msgr2=0x7f952819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9526a00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f952819a620 0x7f952819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9526a00640 1 -- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 msgr2=0x7f95281a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9526a00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 0x7f95281a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9526a00640 1 -- 172.21.15.28:0/245781620 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952819b140 con 0x7f95281041f0 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9526000640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 0x7f95281a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9526a00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95281041f0 0x7f95281a0470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f951800e9f0 tx=0x7f951800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f951f400640 1 -- 172.21.15.28:0/245781620 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951800ccc0 con 0x7f95281041f0 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9527e00640 1 -- 172.21.15.28:0/245781620 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f952819b430 con 0x7f95281041f0 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f951f400640 1 -- 172.21.15.28:0/245781620 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95180108e0 con 0x7f95281041f0 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f951f400640 1 -- 172.21.15.28:0/245781620 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9518018880 con 0x7f95281041f0 2024-04-26T01:41:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.264+0000 7f9527e00640 1 -- 172.21.15.28:0/245781620 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95281a76d0 con 0x7f95281041f0 2024-04-26T01:41:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.269+0000 7f953056e740 1 Processor -- start 2024-04-26T01:41:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.269+0000 7f953056e740 1 -- start start 2024-04-26T01:41:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.269+0000 7f953056e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bca6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.269+0000 7f953056e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca72d0 0x55b66bcb1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f953056e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 0x55b66bcb3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f953056e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b66bc0c900 con 0x55b66bca69b0 2024-04-26T01:41:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f953056e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b66bbb2ef0 con 0x55b66bca72d0 2024-04-26T01:41:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f953056e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55b66bc088c0 con 0x55b66bcb1b90 2024-04-26T01:41:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f951f400640 1 -- 172.21.15.28:0/245781620 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f9518010430 con 0x7f95281041f0 2024-04-26T01:41:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9527400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bca6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9527400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bca6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38886/0 (socket says 172.21.15.28:38886) 2024-04-26T01:41:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9527400640 1 -- 172.21.15.28:0/2150063049 learned_addr learned my addr 172.21.15.28:0/2150063049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9526000640 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 0x55b66bcb3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9526a00640 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca72d0 0x55b66bcb1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9527400640 1 -- 172.21.15.28:0/2150063049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 msgr2=0x55b66bcb3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9527400640 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 0x55b66bcb3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9527400640 1 -- 172.21.15.28:0/2150063049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca72d0 msgr2=0x55b66bcb1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9527400640 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca72d0 0x55b66bcb1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.270+0000 7f9527400640 1 -- 172.21.15.28:0/2150063049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66b91f210 con 0x55b66bca69b0 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.271+0000 7f9526a00640 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca72d0 0x55b66bcb1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.271+0000 7f9527400640 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bca6d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9520009ca0 tx=0x7f95200093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73abbb0450591bb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.271+0000 7f9513e00640 1 -- 172.21.15.28:0/2150063049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520031cf0 con 0x55b66bca69b0 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.271+0000 7f9513e00640 1 -- 172.21.15.28:0/2150063049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9520036030 con 0x55b66bca69b0 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.271+0000 7f9513e00640 1 -- 172.21.15.28:0/2150063049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520004260 con 0x55b66bca69b0 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.271+0000 7f951f400640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f950c0a3eb0 0x7f950c0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.272+0000 7f951f400640 1 -- 172.21.15.28:0/245781620 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f95180d0d60 con 0x7f95281041f0 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.272+0000 7f9526a00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f950c0a3eb0 0x7f950c0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.272+0000 7f953056e740 1 -- 172.21.15.28:0/2150063049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 msgr2=0x55b66bca6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.272+0000 7f953056e740 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bca6d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9520009ca0 tx=0x7f95200093e0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f9526a00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f950c0a3eb0 0x7f950c0a6370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95180ce360 tx=0x7f9518009700 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 -- 172.21.15.28:0/2150063049 shutdown_connections 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 0x55b66bcb3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca72d0 0x55b66bcb1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 --2- 172.21.15.28:0/2150063049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bca6d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 -- 172.21.15.28:0/2150063049 >> 172.21.15.28:0/2150063049 conn(0x55b66bca2ec0 msgr2=0x55b66bca32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 -- 172.21.15.28:0/2150063049 shutdown_connections 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 -- 172.21.15.28:0/2150063049 wait complete. 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 -- start start 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.273+0000 7f953056e740 1 Processor -- start 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f953056e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca69b0 0x55b66bcaea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f953056e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f953056e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 0x55b66bcaf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f953056e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b66bcb45e0 con 0x55b66bca72d0 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f953056e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b66bbc6310 con 0x55b66bcb1b90 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f953056e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55b66bcb4760 con 0x55b66bca69b0 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f953056e740 10 client.0 _ll_register_callbacks cb 0x55b66bb94cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f9526000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f9527400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca69b0 0x55b66bcaea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f9526000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38894/0 (socket says 172.21.15.28:38894) 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f9527400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca69b0 0x55b66bcaea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58128/0 (socket says 172.21.15.28:58128) 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f9526000640 1 -- 172.21.15.28:0/2841608975 learned_addr learned my addr 172.21.15.28:0/2841608975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.274+0000 7f9527400640 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 0x55b66bcaf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9526000640 1 -- 172.21.15.28:0/2841608975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca69b0 msgr2=0x55b66bcaea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9526000640 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca69b0 0x55b66bcaea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9526000640 1 -- 172.21.15.28:0/2841608975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 msgr2=0x55b66bcaf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9526000640 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 0x55b66bcaf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9526000640 1 -- 172.21.15.28:0/2841608975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b66b9eb620 con 0x55b66bca72d0 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9526000640 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaefd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f95140058c0 tx=0x7f951400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=177071eb50ccca45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9510c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9510c00640 1 -- 172.21.15.28:0/2841608975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514012110 con 0x55b66bca72d0 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f9510c00640 1 -- 172.21.15.28:0/2841608975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f9514004050 con 0x55b66bca72d0 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.275+0000 7f953056e740 1 -- 172.21.15.28:0/2841608975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9520036030 con 0x55b66bca72d0 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.276+0000 7f9510c00640 1 -- 172.21.15.28:0/2841608975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f95140122b0 con 0x55b66bca72d0 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.276+0000 7f953056e740 10 client.15087 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.276+0000 7f953056e740 1 -- 172.21.15.28:0/2841608975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b66bcc0800 con 0x55b66bca72d0 2024-04-26T01:41:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.276+0000 7f9510c00640 1 -- 172.21.15.28:0/2841608975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9520036030 con 0x55b66bca72d0 2024-04-26T01:41:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.276+0000 7f9510c00640 1 -- 172.21.15.28:0/2841608975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f9514010430 con 0x55b66bca72d0 2024-04-26T01:41:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.276+0000 7f953056e740 10 client.15087 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.276+0000 7f953056e740 10 client.15087 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:41:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.276+0000 7f953056e740 20 client.15087 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.277+0000 7f953056e740 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b66bcc1d10 0x55b66bcc4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.277+0000 7f953056e740 4 client.15087 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:41:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.277+0000 7f953056e740 1 -- 172.21.15.28:0/2841608975 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b66bc208b0 con 0x55b66bcc1d10 2024-04-26T01:41:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.277+0000 7f9527400640 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b66bcc1d10 0x55b66bcc4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.277+0000 7f9527400640 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b66bcc1d10 0x55b66bcc4100 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d8060377858f8ed server_cookie=c7be90984116f150 in_seq=0 out_seq=0 2024-04-26T01:41:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.277+0000 7f9510c00640 10 client.15087 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.278+0000 7f9510c00640 1 -- 172.21.15.28:0/2841608975 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b66bc208b0 con 0x55b66bcc1d10 2024-04-26T01:41:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.278+0000 7f9510c00640 10 client.15087 handle_command_reply: tid=0 2024-04-26T01:41:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.278+0000 7f953056e740 1 client.15087 shutdown 2024-04-26T01:41:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 -- 172.21.15.28:0/2841608975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 msgr2=0x55b66bcaefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaefd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f95140058c0 tx=0x7f951400f790 comp rx=0 tx=0).stop 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 -- 172.21.15.28:0/2841608975 shutdown_connections 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b66bcc1d10 0x55b66bcc4100 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 0x55b66bcaf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaefd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 --2- 172.21.15.28:0/2841608975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca69b0 0x55b66bcaea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 -- 172.21.15.28:0/2841608975 >> 172.21.15.28:0/2841608975 conn(0x55b66bca2ec0 msgr2=0x55b66bca8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f9510c00640 0 client.15087 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 -- 172.21.15.28:0/2841608975 shutdown_connections 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 1 -- 172.21.15.28:0/2841608975 wait complete. 2024-04-26T01:41:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.279+0000 7f953056e740 20 client.15087 trim_cache size 0 max 16384 2024-04-26T01:41:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.296+0000 7f953056e740 1 Processor -- start 2024-04-26T01:41:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.296+0000 7f953056e740 1 -- start start 2024-04-26T01:41:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.296+0000 7f953056e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca69b0 0x55b66bcbd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.296+0000 7f953056e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.296+0000 7f953056e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 0x55b66bcc9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.296+0000 7f953056e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95281b2410 con 0x55b66bca72d0 2024-04-26T01:41:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.296+0000 7f953056e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f95281afce0 con 0x55b66bca69b0 2024-04-26T01:41:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.296+0000 7f953056e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f95281adb50 con 0x55b66bcb1b90 2024-04-26T01:41:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca69b0 0x55b66bcbd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca69b0 0x55b66bcbd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:45714/0 (socket says 172.21.15.28:45714) 2024-04-26T01:41:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 -- 172.21.15.28:0/1231670323 learned_addr learned my addr 172.21.15.28:0/1231670323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 0x55b66bcc9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9526000640 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 -- 172.21.15.28:0/1231670323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 msgr2=0x55b66bcc9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 0x55b66bcc9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 -- 172.21.15.28:0/1231670323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 msgr2=0x55b66bcaf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9527400640 1 -- 172.21.15.28:0/1231670323 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66bc9d470 con 0x55b66bca69b0 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.297+0000 7f9526000640 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.298+0000 7f9527400640 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca69b0 0x55b66bcbd4a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9520003b30 tx=0x7f9520005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa4badbe73cb7070 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.298+0000 7f9513400640 1 -- 172.21.15.28:0/1231670323 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520031af0 con 0x55b66bca69b0 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.298+0000 7f9513400640 1 -- 172.21.15.28:0/1231670323 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9520036040 con 0x55b66bca69b0 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.298+0000 7f9513400640 1 -- 172.21.15.28:0/1231670323 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520004140 con 0x55b66bca69b0 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 -- 172.21.15.28:0/1231670323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca69b0 msgr2=0x55b66bcbd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca69b0 0x55b66bcbd4a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9520003b30 tx=0x7f9520005d80 comp rx=0 tx=0).stop 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 -- 172.21.15.28:0/1231670323 shutdown_connections 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bcb1b90 0x55b66bcc9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca72d0 0x55b66bcaf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 --2- 172.21.15.28:0/1231670323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bca69b0 0x55b66bcbd4a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 -- 172.21.15.28:0/1231670323 >> 172.21.15.28:0/1231670323 conn(0x55b66bca2ec0 msgr2=0x55b66bca8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 -- 172.21.15.28:0/1231670323 shutdown_connections 2024-04-26T01:41:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.299+0000 7f953056e740 1 -- 172.21.15.28:0/1231670323 wait complete. 2024-04-26T01:41:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 1 -- start start 2024-04-26T01:41:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 1 Processor -- start 2024-04-26T01:41:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bcb9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca72d0 0x55b66bcb6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 0x55b66bcb74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95281ae1a0 con 0x55b66bca69b0 2024-04-26T01:41:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f95281af610 con 0x55b66bcb1b90 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f95281ae870 con 0x55b66bca72d0 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.300+0000 7f953056e740 10 client.0 _ll_register_callbacks cb 0x7f9514012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bcb9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bcb9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38900/0 (socket says 172.21.15.28:38900) 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 -- 172.21.15.28:0/3682598841 learned_addr learned my addr 172.21.15.28:0/3682598841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9526000640 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca72d0 0x55b66bcb6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 0x55b66bcb74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 -- 172.21.15.28:0/3682598841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca72d0 msgr2=0x55b66bcb6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca72d0 0x55b66bcb6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 -- 172.21.15.28:0/3682598841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 msgr2=0x55b66bcb74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 0x55b66bcb74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 -- 172.21.15.28:0/3682598841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b66bcb79e0 con 0x55b66bca69b0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9527400640 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bcb9a90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f952002f3e0 tx=0x7f9520031f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f265fb0040ef9a6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.301+0000 7f9510c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.302+0000 7f9510c00640 1 -- 172.21.15.28:0/3682598841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520069060 con 0x55b66bca69b0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.302+0000 7f9510c00640 1 -- 172.21.15.28:0/3682598841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f952005d040 con 0x55b66bca69b0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.302+0000 7f953056e740 1 -- 172.21.15.28:0/3682598841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9520036040 con 0x55b66bca69b0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.302+0000 7f9510c00640 1 -- 172.21.15.28:0/3682598841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9520031bb0 con 0x55b66bca69b0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.302+0000 7f953056e740 10 client.15093 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.302+0000 7f953056e740 1 -- 172.21.15.28:0/3682598841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b66bd27fb0 con 0x55b66bca69b0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.302+0000 7f9510c00640 1 -- 172.21.15.28:0/3682598841 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9520036040 con 0x55b66bca69b0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.303+0000 7f9510c00640 1 -- 172.21.15.28:0/3682598841 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f9520043070 con 0x55b66bca69b0 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.303+0000 7f953056e740 10 client.15093 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.303+0000 7f953056e740 10 client.15093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:41:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.303+0000 7f953056e740 20 client.15093 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:27.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.303+0000 7f953056e740 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b66bd283c0 0x55b66bd2a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:27.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.303+0000 7f953056e740 4 client.15093 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:41:27.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.303+0000 7f953056e740 1 -- 172.21.15.28:0/3682598841 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b66bc1a320 con 0x55b66bd283c0 2024-04-26T01:41:27.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.303+0000 7f9526000640 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b66bd283c0 0x55b66bd2a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:27.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.304+0000 7f9526000640 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b66bd283c0 0x55b66bd2a7e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5185d29e8b775789 server_cookie=30da9db7fde77499 in_seq=0 out_seq=0 2024-04-26T01:41:27.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.304+0000 7f9510c00640 10 client.15093 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f9510c00640 1 -- 172.21.15.28:0/3682598841 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b66bc1a320 con 0x55b66bd283c0 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f9510c00640 10 client.15093 handle_command_reply: tid=0 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 client.15093 shutdown 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 -- 172.21.15.28:0/3682598841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 msgr2=0x55b66bcb9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bcb9a90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f952002f3e0 tx=0x7f9520031f60 comp rx=0 tx=0).stop 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 -- 172.21.15.28:0/3682598841 shutdown_connections 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b66bd283c0 0x55b66bd2a7e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b66bcb1b90 0x55b66bcb74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b66bca72d0 0x55b66bcb6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 --2- 172.21.15.28:0/3682598841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66bca69b0 0x55b66bcb9a90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f9510c00640 0 client.15093 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 -- 172.21.15.28:0/3682598841 >> 172.21.15.28:0/3682598841 conn(0x55b66bca2ec0 msgr2=0x55b66bca1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 -- 172.21.15.28:0/3682598841 shutdown_connections 2024-04-26T01:41:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 1 -- 172.21.15.28:0/3682598841 wait complete. 2024-04-26T01:41:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.311+0000 7f953056e740 20 client.15093 trim_cache size 0 max 16384 2024-04-26T01:41:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:41:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:41:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:41:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 -- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f950c0a3eb0 msgr2=0x7f950c0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f950c0a3eb0 0x7f950c0a6370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95180ce360 tx=0x7f9518009700 comp rx=0 tx=0).stop 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 -- 172.21.15.28:0/245781620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95281041f0 msgr2=0x7f95281a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95281041f0 0x7f95281a0470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f951800e9f0 tx=0x7f951800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 -- 172.21.15.28:0/245781620 shutdown_connections 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f950c0a3eb0 0x7f950c0a6370 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f952819a620 0x7f952819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95281092c0 0x7f95281a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 --2- 172.21.15.28:0/245781620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95281041f0 0x7f95281a0470 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 -- 172.21.15.28:0/245781620 >> 172.21.15.28:0/245781620 conn(0x7f95280ffa30 msgr2=0x7f9528101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:27.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 -- 172.21.15.28:0/245781620 shutdown_connections 2024-04-26T01:41:27.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:27.312+0000 7f9527e00640 1 -- 172.21.15.28:0/245781620 wait complete. 2024-04-26T01:41:27.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:27 smithi028 ceph-mon[28247]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:27.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:27 smithi053 ceph-mon[32593]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:27.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:27 smithi169 ceph-mon[32851]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:28.812 INFO:teuthology.orchestra.run.smithi028.stdout:{"run_dir":"/var/run/ceph"} 2024-04-26T01:41:28.813 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-04-26T01:41:28.813 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 3448 2024-04-26T01:41:28.813 DEBUG:tasks.quiescer:Yielding 2024-04-26T01:41:28.813 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T01:41:28.824 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:41:28.824 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:41:28.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-26T01:41:28.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:28.928 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:28.965 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-26T01:41:28.965 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:3300,172.21.15.53:3300,172.21.15.169:3300:/volumes/qa/sv_1/57e9c16f-4990-4d2e-88b0-536ce8dec643 340G 0 340G 0% /tmp/cephfs 2024-04-26T01:41:28.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-26T01:41:29.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-26T01:41:29.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-26T01:41:29.287 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:29.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:29 smithi028 ceph-mon[28247]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:29.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:29 smithi053 ceph-mon[32593]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:29.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:29 smithi169 ceph-mon[32851]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:41:31.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:31 smithi053 ceph-mon[32593]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-26T01:41:31.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:31 smithi028 ceph-mon[28247]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-26T01:41:31.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-26T01:41:31.845 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-26T01:41:31.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-26T01:41:31.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:31 smithi169 ceph-mon[32851]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-26T01:41:31.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/: 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 .. 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 data 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data: 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:32.089 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 .. 2024-04-26T01:41:32.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-26T01:41:32.219 INFO:teuthology.orchestra.run.smithi028.stdout:[Service] 2024-04-26T01:41:32.220 INFO:teuthology.orchestra.run.smithi028.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-26T01:41:32.220 INFO:teuthology.orchestra.run.smithi028.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-26T01:41:32.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-26T01:41:32.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.419+0000 7f7d70400700 1 -- 172.21.15.28:0/3494561383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d6806bb10 msgr2=0x7f7d6810f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.419+0000 7f7d70400700 1 --2- 172.21.15.28:0/3494561383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d6806bb10 0x7f7d6810f900 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000b0a0 tx=0x7f7d6000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.420+0000 7f7d70400700 1 -- 172.21.15.28:0/3494561383 shutdown_connections 2024-04-26T01:41:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.420+0000 7f7d70400700 1 --2- 172.21.15.28:0/3494561383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d6806bb10 0x7f7d6810f900 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.420+0000 7f7d70400700 1 --2- 172.21.15.28:0/3494561383 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d68108a30 0x7f7d68108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.420+0000 7f7d70400700 1 --2- 172.21.15.28:0/3494561383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d6806b170 0x7f7d6806b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.420+0000 7f7d70400700 1 -- 172.21.15.28:0/3494561383 >> 172.21.15.28:0/3494561383 conn(0x7f7d680788f0 msgr2=0x7f7d68078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.420+0000 7f7d70400700 1 -- 172.21.15.28:0/3494561383 shutdown_connections 2024-04-26T01:41:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.420+0000 7f7d70400700 1 -- 172.21.15.28:0/3494561383 wait complete. 2024-04-26T01:41:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.421+0000 7f7d70400700 1 Processor -- start 2024-04-26T01:41:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.421+0000 7f7d70400700 1 -- start start 2024-04-26T01:41:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.421+0000 7f7d70400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d6806b170 0x7f7d68101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.421+0000 7f7d70400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d6806bb10 0x7f7d68101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d70400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d68108a30 0x7f7d681024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d70400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d68112c20 con 0x7f7d6806bb10 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d70400700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7d68112aa0 con 0x7f7d6806b170 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d70400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7d68112da0 con 0x7f7d68108a30 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d6806bb10 0x7f7d68101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6da00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d6806b170 0x7f7d68101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d6806bb10 0x7f7d68101f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49954/0 (socket says 172.21.15.28:49954) 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6d000700 1 -- 172.21.15.28:0/902753526 learned_addr learned my addr 172.21.15.28:0/902753526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6e400700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d68108a30 0x7f7d681024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:32.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6da00700 1 -- 172.21.15.28:0/902753526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d68108a30 msgr2=0x7f7d681024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:32.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6da00700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d68108a30 0x7f7d681024c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6da00700 1 -- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d6806bb10 msgr2=0x7f7d68101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:32.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6da00700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d6806bb10 0x7f7d68101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.422+0000 7f7d6da00700 1 -- 172.21.15.28:0/902753526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d681a76e0 con 0x7f7d6806b170 2024-04-26T01:41:32.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.423+0000 7f7d6da00700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d6806b170 0x7f7d68101a40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00e9f0 tx=0x7f7d5c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:32.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.423+0000 7f7d66000700 1 -- 172.21.15.28:0/902753526 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c004220 con 0x7f7d6806b170 2024-04-26T01:41:32.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.423+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d681a79d0 con 0x7f7d6806b170 2024-04-26T01:41:32.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.423+0000 7f7d66000700 1 -- 172.21.15.28:0/902753526 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d5c0043c0 con 0x7f7d6806b170 2024-04-26T01:41:32.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.423+0000 7f7d66000700 1 -- 172.21.15.28:0/902753526 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c01c070 con 0x7f7d6806b170 2024-04-26T01:41:32.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.423+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d681a7f10 con 0x7f7d6806b170 2024-04-26T01:41:32.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.425+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d68069730 con 0x7f7d6806b170 2024-04-26T01:41:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.427+0000 7f7d66000700 1 -- 172.21.15.28:0/902753526 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f7d5c004750 con 0x7f7d6806b170 2024-04-26T01:41:32.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.428+0000 7f7d66000700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7d500a3ed0 0x7f7d500a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:32.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.428+0000 7f7d66000700 1 -- 172.21.15.28:0/902753526 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f7d5c0d08b0 con 0x7f7d6806b170 2024-04-26T01:41:32.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.428+0000 7f7d6d000700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7d500a3ed0 0x7f7d500a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:32.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.429+0000 7f7d6d000700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7d500a3ed0 0x7f7d500a6390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7d54009680 tx=0x7f7d54009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:32.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.433+0000 7f7d66000700 1 -- 172.21.15.28:0/902753526 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d5c09cda0 con 0x7f7d6806b170 2024-04-26T01:41:32.453 INFO:teuthology.orchestra.run.smithi028.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-26T01:41:32.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.723+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d68102f70 con 0x7f7d6806b170 2024-04-26T01:41:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.724+0000 7f7d66000700 1 -- 172.21.15.28:0/902753526 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11861 (secure 0 0 0) 0x7f7d5c09c6a0 con 0x7f7d6806b170 2024-04-26T01:41:32.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:32.721 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:40:57.900766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24509,24523]},"id":1}]} 2024-04-26T01:41:32.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7d500a3ed0 msgr2=0x7f7d500a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:32.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7d500a3ed0 0x7f7d500a6390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7d54009680 tx=0x7f7d54009260 comp rx=0 tx=0).stop 2024-04-26T01:41:32.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d6806b170 msgr2=0x7f7d68101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:32.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d6806b170 0x7f7d68101a40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00e9f0 tx=0x7f7d5c00ed00 comp rx=0 tx=0).stop 2024-04-26T01:41:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 shutdown_connections 2024-04-26T01:41:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7d500a3ed0 0x7f7d500a6390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d68108a30 0x7f7d681024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d6806b170 0x7f7d68101a40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 --2- 172.21.15.28:0/902753526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d6806bb10 0x7f7d68101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:32.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.727+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 >> 172.21.15.28:0/902753526 conn(0x7f7d680788f0 msgr2=0x7f7d680ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:32.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.728+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 shutdown_connections 2024-04-26T01:41:32.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:32.728+0000 7f7d70400700 1 -- 172.21.15.28:0/902753526 wait complete. 2024-04-26T01:41:32.725 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-26T01:41:33.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:36:57.152667+0000', 'modified': '2024-04-26T01:40:57.900766+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24523, 'mds_1': 24509, 'mds_2': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24491': {'gid': 24491, 'name': 'g', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/2114874192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2114874192}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2114874192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24497': {'gid': 24497, 'name': 'a', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/2299937376', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 2299937376}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 2299937376}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.169:6835/1802608598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6834', 'nonce': 1802608598}, {'type': 'v1', 'addr': '172.21.15.169:6835', 'nonce': 1802608598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.169:6837/3777955238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6836', 'nonce': 3777955238}, {'type': 'v1', 'addr': '172.21.15.169:6837', 'nonce': 3777955238}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.53:6839/955096275', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6838', 'nonce': 955096275}, {'type': 'v1', 'addr': '172.21.15.53:6839', 'nonce': 955096275}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.169:6839/2157805186', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6838', 'nonce': 2157805186}, {'type': 'v1', 'addr': '172.21.15.169:6839', 'nonce': 2157805186}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24523, 'qdb_cluster': [24505, 24509, 24523]} 2024-04-26T01:41:33.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:41:33.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-26T01:41:33.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:33 smithi053 ceph-mon[32593]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:41:33.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:33 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/902753526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:33.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:33 smithi028 ceph-mon[28247]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:41:33.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:33 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/902753526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:33.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:33 smithi169 ceph-mon[32851]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:41:33.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:33 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/902753526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:35.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:35 smithi028 ceph-mon[28247]: pgmap v21: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 53 KiB/s wr, 11 op/s 2024-04-26T01:41:35.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:35 smithi053 ceph-mon[32593]: pgmap v21: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 53 KiB/s wr, 11 op/s 2024-04-26T01:41:35.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:35 smithi169 ceph-mon[32851]: pgmap v21: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 53 KiB/s wr, 11 op/s 2024-04-26T01:41:36.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 -- 172.21.15.28:0/1890582255 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 msgr2=0x7fb20c1052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 --2- 172.21.15.28:0/1890582255 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 0x7fb20c1052c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0099c0 tx=0x7fb1fc009cd0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 -- 172.21.15.28:0/1890582255 shutdown_connections 2024-04-26T01:41:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 --2- 172.21.15.28:0/1890582255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb20c109fc0 0x7fb20c111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 --2- 172.21.15.28:0/1890582255 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 0x7fb20c1052c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 --2- 172.21.15.28:0/1890582255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20c105890 0x7fb20c1098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 -- 172.21.15.28:0/1890582255 >> 172.21.15.28:0/1890582255 conn(0x7fb20c100770 msgr2=0x7fb20c102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 -- 172.21.15.28:0/1890582255 shutdown_connections 2024-04-26T01:41:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.621+0000 7fb212e00700 1 -- 172.21.15.28:0/1890582255 wait complete. 2024-04-26T01:41:36.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.622+0000 7fb212e00700 1 Processor -- start 2024-04-26T01:41:36.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb212e00700 1 -- start start 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb212e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 0x7fb20c19b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb212e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb20c105890 0x7fb20c19b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb212e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20c109fc0 0x7fb20c19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb212e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c114ea0 con 0x7fb20c105890 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb212e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c114d20 con 0x7fb20c109fc0 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb212e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c115020 con 0x7fb20c104ee0 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb20b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb20c105890 0x7fb20c19b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb20be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 0x7fb20c19b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb20b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb20c105890 0x7fb20c19b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49978/0 (socket says 172.21.15.28:49978) 2024-04-26T01:41:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.623+0000 7fb20b400700 1 -- 172.21.15.28:0/2936573486 learned_addr learned my addr 172.21.15.28:0/2936573486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:36.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.624+0000 7fb210e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20c109fc0 0x7fb20c19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.624+0000 7fb20b400700 1 -- 172.21.15.28:0/2936573486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 msgr2=0x7fb20c19b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.624+0000 7fb20b400700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 0x7fb20c19b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.624+0000 7fb20b400700 1 -- 172.21.15.28:0/2936573486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20c109fc0 msgr2=0x7fb20c19fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.624+0000 7fb20b400700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20c109fc0 0x7fb20c19fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.624+0000 7fb20b400700 1 -- 172.21.15.28:0/2936573486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20c1a03e0 con 0x7fb20c105890 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.624+0000 7fb20be00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 0x7fb20c19b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.624+0000 7fb210e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20c109fc0 0x7fb20c19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.625+0000 7fb20b400700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb20c105890 0x7fb20c19b910 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800e970 tx=0x7fb1f800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.625+0000 7fb208c00700 1 -- 172.21.15.28:0/2936573486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800cbd0 con 0x7fb20c105890 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.625+0000 7fb208c00700 1 -- 172.21.15.28:0/2936573486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1f800cd70 con 0x7fb20c105890 2024-04-26T01:41:36.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.625+0000 7fb208c00700 1 -- 172.21.15.28:0/2936573486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8005610 con 0x7fb20c105890 2024-04-26T01:41:36.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.626+0000 7fb212e00700 1 -- 172.21.15.28:0/2936573486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb20c1a06d0 con 0x7fb20c105890 2024-04-26T01:41:36.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.626+0000 7fb212e00700 1 -- 172.21.15.28:0/2936573486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb20c1a7f50 con 0x7fb20c105890 2024-04-26T01:41:36.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.630+0000 7fb208c00700 1 -- 172.21.15.28:0/2936573486 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fb1f8004750 con 0x7fb20c105890 2024-04-26T01:41:36.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.630+0000 7fb21d695b80 1 Processor -- start 2024-04-26T01:41:36.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb21d695b80 1 -- start start 2024-04-26T01:41:36.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 0x55d83d74fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d7500a0 0x55d83d7499b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d749ef0 0x55d83d74a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb21d695b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d83d69be40 con 0x55d83d74f780 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb21d695b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d83d6426e0 con 0x55d83d749ef0 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb21d695b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d83d697e00 con 0x55d83d7500a0 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb208c00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1f00a3ed0 0x7fb1f00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb210e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 0x55d83d74fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.631+0000 7fb20be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d7500a0 0x55d83d7499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb20be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d7500a0 0x55d83d7499b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52762/0 (socket says 172.21.15.28:52762) 2024-04-26T01:41:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb20be00700 1 -- 172.21.15.28:0/2370921098 learned_addr learned my addr 172.21.15.28:0/2370921098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb208c00700 1 -- 172.21.15.28:0/2936573486 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fb1f80d0ce0 con 0x7fb20c105890 2024-04-26T01:41:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb20be00700 1 -- 172.21.15.28:0/2370921098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d749ef0 msgr2=0x55d83d74a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:41:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb20be00700 1 --2- 172.21.15.28:0/2370921098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d749ef0 0x55d83d74a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb20be00700 1 -- 172.21.15.28:0/2370921098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 msgr2=0x55d83d74fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb210e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1f00a3ed0 0x7fb1f00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb20be00700 1 --2- 172.21.15.28:0/2370921098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 0x55d83d74fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb20be00700 1 -- 172.21.15.28:0/2370921098 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d83d638630 con 0x55d83d7500a0 2024-04-26T01:41:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.632+0000 7fb20be00700 1 --2- 172.21.15.28:0/2370921098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d7500a0 0x55d83d7499b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc015000 tx=0x7fb1fc003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d95dc88bd2bf0bec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.633+0000 7fb201600700 1 -- 172.21.15.28:0/2370921098 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc027070 con 0x55d83d7500a0 2024-04-26T01:41:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.633+0000 7fb201600700 1 -- 172.21.15.28:0/2370921098 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1fc0037c0 con 0x55d83d7500a0 2024-04-26T01:41:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.633+0000 7fb201600700 1 -- 172.21.15.28:0/2370921098 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc00b9f0 con 0x55d83d7500a0 2024-04-26T01:41:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.633+0000 7fb210e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1f00a3ed0 0x7fb1f00a6390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb204009da0 tx=0x7fb204004b40 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 -- 172.21.15.28:0/2370921098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d7500a0 msgr2=0x55d83d7499b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 --2- 172.21.15.28:0/2370921098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d7500a0 0x55d83d7499b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc015000 tx=0x7fb1fc003700 comp rx=0 tx=0).stop 2024-04-26T01:41:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 -- 172.21.15.28:0/2370921098 shutdown_connections 2024-04-26T01:41:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 --2- 172.21.15.28:0/2370921098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d749ef0 0x55d83d74a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 --2- 172.21.15.28:0/2370921098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 0x55d83d74fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 --2- 172.21.15.28:0/2370921098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d7500a0 0x55d83d7499b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 -- 172.21.15.28:0/2370921098 >> 172.21.15.28:0/2370921098 conn(0x55d83d677f00 msgr2=0x55d83d7448a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 -- 172.21.15.28:0/2370921098 shutdown_connections 2024-04-26T01:41:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.634+0000 7fb21d695b80 1 -- 172.21.15.28:0/2370921098 wait complete. 2024-04-26T01:41:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.635+0000 7fb21d695b80 1 -- start start 2024-04-26T01:41:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.635+0000 7fb21d695b80 1 Processor -- start 2024-04-26T01:41:36.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d755d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d74f780 0x55d83d753030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d753570 0x55d83d7539f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb21d695b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d83d7568c0 con 0x55d83d749ef0 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb21d695b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d83d655810 con 0x55d83d74f780 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb21d695b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d83d756a40 con 0x55d83d753570 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb21d695b80 10 client.0 _ll_register_callbacks cb 0x55d83d6346d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb210e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d753570 0x55d83d7539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb210e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d753570 0x55d83d7539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52766/0 (socket says 172.21.15.28:52766) 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb210e00700 1 -- 172.21.15.28:0/1282560410 learned_addr learned my addr 172.21.15.28:0/1282560410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb20b400700 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d74f780 0x55d83d753030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.636+0000 7fb20be00700 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d755d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb20b400700 1 -- 172.21.15.28:0/1282560410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d753570 msgr2=0x55d83d7539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb20b400700 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d753570 0x55d83d7539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb20b400700 1 -- 172.21.15.28:0/1282560410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 msgr2=0x55d83d755d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb20b400700 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d755d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb20b400700 1 -- 172.21.15.28:0/1282560410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d83d5e9880 con 0x55d83d74f780 2024-04-26T01:41:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb210e00700 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d753570 0x55d83d7539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb20b400700 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d74f780 0x55d83d753030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80ce280 tx=0x7fb1f800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3797a3c8aae16de server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb1f6000700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:41:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb1f6000700 1 -- 172.21.15.28:0/1282560410 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f809c360 con 0x55d83d74f780 2024-04-26T01:41:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb1f6000700 1 -- 172.21.15.28:0/1282560410 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fb1f80a53e0 con 0x55d83d74f780 2024-04-26T01:41:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.637+0000 7fb21d695b80 1 -- 172.21.15.28:0/1282560410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1fc0037c0 con 0x55d83d74f780 2024-04-26T01:41:36.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.638+0000 7fb1f6000700 1 -- 172.21.15.28:0/1282560410 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb1f809d100 con 0x55d83d74f780 2024-04-26T01:41:36.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.638+0000 7fb21d695b80 10 client.25217 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.638+0000 7fb21d695b80 1 -- 172.21.15.28:0/1282560410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d83d632aa0 con 0x55d83d74f780 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.638+0000 7fb1f6000700 1 -- 172.21.15.28:0/1282560410 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1fc0037c0 con 0x55d83d74f780 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.638+0000 7fb1f6000700 1 -- 172.21.15.28:0/1282560410 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fb1f80ce560 con 0x55d83d74f780 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.638+0000 7fb21d695b80 10 client.25217 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.639+0000 7fb21d695b80 10 client.25217 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.639+0000 7fb21d695b80 20 client.25217 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.639+0000 7fb21d695b80 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x55d83d763a00 0x55d83d764e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.639+0000 7fb21d695b80 4 client.25217 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.639+0000 7fb21d695b80 1 -- 172.21.15.28:0/1282560410 --> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d83d6a9630 con 0x55d83d763a00 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.639+0000 7fb20be00700 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x55d83d763a00 0x55d83d764e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.640+0000 7fb20be00700 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x55d83d763a00 0x55d83d764e70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a5f0e5fe3d5f683 server_cookie=86fa23390456af79 in_seq=0 out_seq=0 2024-04-26T01:41:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.640+0000 7fb1f6000700 10 client.25217 ms_handle_connect on v2:172.21.15.169:6836/3777955238 2024-04-26T01:41:36.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb1f6000700 1 -- 172.21.15.28:0/1282560410 <== mds.2 v2:172.21.15.169:6836/3777955238 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d83d6a9630 con 0x55d83d763a00 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb1f6000700 10 client.25217 handle_command_reply: tid=0 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 client.25217 shutdown 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 -- 172.21.15.28:0/1282560410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d74f780 msgr2=0x55d83d753030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d74f780 0x55d83d753030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80ce280 tx=0x7fb1f800ef20 comp rx=0 tx=0).stop 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 -- 172.21.15.28:0/1282560410 shutdown_connections 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x55d83d763a00 0x55d83d764e70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d753570 0x55d83d7539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d755d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 --2- 172.21.15.28:0/1282560410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d74f780 0x55d83d753030 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 -- 172.21.15.28:0/1282560410 >> 172.21.15.28:0/1282560410 conn(0x55d83d677f00 msgr2=0x55d83d743e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb1f6000700 0 client.25217 ms_handle_reset on v2:172.21.15.169:6836/3777955238 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.641+0000 7fb21d695b80 1 -- 172.21.15.28:0/1282560410 shutdown_connections 2024-04-26T01:41:36.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.642+0000 7fb21d695b80 1 -- 172.21.15.28:0/1282560410 wait complete. 2024-04-26T01:41:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.642+0000 7fb21d695b80 20 client.25217 trim_cache size 0 max 16384 2024-04-26T01:41:36.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.686+0000 7fb21d695b80 1 Processor -- start 2024-04-26T01:41:36.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.687+0000 7fb21d695b80 1 -- start start 2024-04-26T01:41:36.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.687+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d749ef0 0x55d83d7499b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.687+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 0x55d83d75fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.687+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d750ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.687+0000 7fb21d695b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d83d7631b0 con 0x55d83d74f780 2024-04-26T01:41:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.687+0000 7fb21d695b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d83d763880 con 0x55d83d753570 2024-04-26T01:41:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.687+0000 7fb21d695b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d83d763f50 con 0x55d83d749ef0 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb20be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d749ef0 0x55d83d7499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb20be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d749ef0 0x55d83d7499b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52776/0 (socket says 172.21.15.28:52776) 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb210e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d750ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb20be00700 1 -- 172.21.15.28:0/2741457654 learned_addr learned my addr 172.21.15.28:0/2741457654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb20b400700 1 --2- 172.21.15.28:0/2741457654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 0x55d83d75fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb210e00700 1 -- 172.21.15.28:0/2741457654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d749ef0 msgr2=0x55d83d7499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb210e00700 1 --2- 172.21.15.28:0/2741457654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d749ef0 0x55d83d7499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb210e00700 1 -- 172.21.15.28:0/2741457654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 msgr2=0x55d83d75fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb210e00700 1 --2- 172.21.15.28:0/2741457654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 0x55d83d75fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.688+0000 7fb210e00700 1 -- 172.21.15.28:0/2741457654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d83d73c690 con 0x55d83d753570 2024-04-26T01:41:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.689+0000 7fb210e00700 1 --2- 172.21.15.28:0/2741457654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d750ba0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb20400ed70 tx=0x7fb204009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c86938abc683d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.689+0000 7fb200c00700 1 -- 172.21.15.28:0/2741457654 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204004bb0 con 0x55d83d753570 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.689+0000 7fb200c00700 1 -- 172.21.15.28:0/2741457654 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb204004d50 con 0x55d83d753570 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.689+0000 7fb200c00700 1 -- 172.21.15.28:0/2741457654 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204005940 con 0x55d83d753570 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 -- 172.21.15.28:0/2741457654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 msgr2=0x55d83d750ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 --2- 172.21.15.28:0/2741457654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d750ba0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb20400ed70 tx=0x7fb204009920 comp rx=0 tx=0).stop 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 -- 172.21.15.28:0/2741457654 shutdown_connections 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 --2- 172.21.15.28:0/2741457654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d750ba0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 --2- 172.21.15.28:0/2741457654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d749ef0 0x55d83d7499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 --2- 172.21.15.28:0/2741457654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d74f780 0x55d83d75fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 -- 172.21.15.28:0/2741457654 >> 172.21.15.28:0/2741457654 conn(0x55d83d677f00 msgr2=0x55d83d750490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 -- 172.21.15.28:0/2741457654 shutdown_connections 2024-04-26T01:41:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.690+0000 7fb21d695b80 1 -- 172.21.15.28:0/2741457654 wait complete. 2024-04-26T01:41:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.691+0000 7fb21d695b80 1 -- start start 2024-04-26T01:41:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.691+0000 7fb21d695b80 1 Processor -- start 2024-04-26T01:41:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.691+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d743cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.691+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d74f780 0x55d83d744210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.691+0000 7fb21d695b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d76c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.691+0000 7fb21d695b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d83d764cf0 con 0x55d83d749ef0 2024-04-26T01:41:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.691+0000 7fb21d695b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d83d764620 con 0x55d83d753570 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.691+0000 7fb21d695b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d83d7653c0 con 0x55d83d74f780 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb21d695b80 10 client.0 _ll_register_callbacks cb 0x55d83d6346d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d743cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d743cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50016/0 (socket says 172.21.15.28:50016) 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20be00700 1 -- 172.21.15.28:0/3437326943 learned_addr learned my addr 172.21.15.28:0/3437326943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20b400700 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d74f780 0x55d83d744210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb210e00700 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d76c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20be00700 1 -- 172.21.15.28:0/3437326943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d74f780 msgr2=0x55d83d744210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20be00700 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d74f780 0x55d83d744210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20be00700 1 -- 172.21.15.28:0/3437326943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 msgr2=0x55d83d76c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20be00700 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d76c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.692+0000 7fb20be00700 1 -- 172.21.15.28:0/3437326943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d83d76ccf0 con 0x55d83d749ef0 2024-04-26T01:41:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.693+0000 7fb20be00700 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d743cd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc005ea0 tx=0x7fb1fc005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35ff01fd19f42f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.693+0000 7fb1f6000700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.693+0000 7fb1f6000700 1 -- 172.21.15.28:0/3437326943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc032e60 con 0x55d83d749ef0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.693+0000 7fb1f6000700 1 -- 172.21.15.28:0/3437326943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fb1fc0052d0 con 0x55d83d749ef0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.693+0000 7fb21d695b80 1 -- 172.21.15.28:0/3437326943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb204004d50 con 0x55d83d749ef0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.693+0000 7fb1f6000700 1 -- 172.21.15.28:0/3437326943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb1fc027040 con 0x55d83d749ef0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.693+0000 7fb21d695b80 10 client.15117 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.693+0000 7fb21d695b80 1 -- 172.21.15.28:0/3437326943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d83d7cb1b0 con 0x55d83d749ef0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.694+0000 7fb1f6000700 1 -- 172.21.15.28:0/3437326943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb204004d50 con 0x55d83d749ef0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.694+0000 7fb1f6000700 1 -- 172.21.15.28:0/3437326943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fb1fc021030 con 0x55d83d749ef0 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.694+0000 7fb21d695b80 10 client.15117 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.694+0000 7fb21d695b80 10 client.15117 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.694+0000 7fb21d695b80 20 client.15117 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.694+0000 7fb21d695b80 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x55d83d7cf8c0 0x55d83d7ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.694+0000 7fb21d695b80 4 client.15117 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:41:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.694+0000 7fb21d695b80 1 -- 172.21.15.28:0/3437326943 --> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] -- command(tid 0: {"prefix": "status"}) -- 0x55d83d6a36d0 con 0x55d83d7cf8c0 2024-04-26T01:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.695+0000 7fb20b400700 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x55d83d7cf8c0 0x55d83d7ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.695+0000 7fb20b400700 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x55d83d7cf8c0 0x55d83d7ce5d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=24474dee797e3af4 server_cookie=3ff14e9876f98678 in_seq=0 out_seq=0 2024-04-26T01:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.695+0000 7fb1f6000700 10 client.15117 ms_handle_connect on v2:172.21.15.169:6836/3777955238 2024-04-26T01:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.695+0000 7fb1f6000700 1 -- 172.21.15.28:0/3437326943 <== mds.2 v2:172.21.15.169:6836/3777955238 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d83d6a36d0 con 0x55d83d7cf8c0 2024-04-26T01:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.695+0000 7fb1f6000700 10 client.15117 handle_command_reply: tid=0 2024-04-26T01:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.695+0000 7fb21d695b80 1 client.15117 shutdown 2024-04-26T01:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 -- 172.21.15.28:0/3437326943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 msgr2=0x55d83d743cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d743cd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc005ea0 tx=0x7fb1fc005c00 comp rx=0 tx=0).stop 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 -- 172.21.15.28:0/3437326943 shutdown_connections 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.169:6836/3777955238,v1:172.21.15.169:6837/3777955238] conn(0x55d83d7cf8c0 0x55d83d7ce5d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d83d753570 0x55d83d76c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d83d749ef0 0x55d83d743cd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 --2- 172.21.15.28:0/3437326943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d83d74f780 0x55d83d744210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 -- 172.21.15.28:0/3437326943 >> 172.21.15.28:0/3437326943 conn(0x55d83d677f00 msgr2=0x55d83d751920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb1f6000700 0 client.15117 ms_handle_reset on v2:172.21.15.169:6836/3777955238 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 -- 172.21.15.28:0/3437326943 shutdown_connections 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 1 -- 172.21.15.28:0/3437326943 wait complete. 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.696+0000 7fb21d695b80 20 client.15117 trim_cache size 0 max 16384 2024-04-26T01:41:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24505, 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 262.29880446200002, 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 25, 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 320.50487188599999 2024-04-26T01:41:36.697 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:41:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 -- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1f00a3ed0 msgr2=0x7fb1f00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1f00a3ed0 0x7fb1f00a6390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb204009da0 tx=0x7fb204004b40 comp rx=0 tx=0).stop 2024-04-26T01:41:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 -- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb20c105890 msgr2=0x7fb20c19b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb20c105890 0x7fb20c19b910 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800e970 tx=0x7fb1f800ed30 comp rx=0 tx=0).stop 2024-04-26T01:41:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 -- 172.21.15.28:0/2936573486 shutdown_connections 2024-04-26T01:41:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1f00a3ed0 0x7fb1f00a6390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20c109fc0 0x7fb20c19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c104ee0 0x7fb20c19b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 --2- 172.21.15.28:0/2936573486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb20c105890 0x7fb20c19b910 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.697+0000 7fb212e00700 1 -- 172.21.15.28:0/2936573486 >> 172.21.15.28:0/2936573486 conn(0x7fb20c100770 msgr2=0x7fb20c109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.698+0000 7fb212e00700 1 -- 172.21.15.28:0/2936573486 shutdown_connections 2024-04-26T01:41:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:36.698+0000 7fb212e00700 1 -- 172.21.15.28:0/2936573486 wait complete. 2024-04-26T01:41:37.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:41:37.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:37 smithi053 ceph-mon[32593]: pgmap v22: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 9 op/s 2024-04-26T01:41:37.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:37 smithi028 ceph-mon[28247]: pgmap v22: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 9 op/s 2024-04-26T01:41:37.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:37 smithi169 ceph-mon[32851]: pgmap v22: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 9 op/s 2024-04-26T01:41:37.948 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:39.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:39 smithi053 ceph-mon[32593]: pgmap v23: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 8 op/s 2024-04-26T01:41:39.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:39 smithi028 ceph-mon[28247]: pgmap v23: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 8 op/s 2024-04-26T01:41:39.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:39 smithi169 ceph-mon[32851]: pgmap v23: 97 pgs: 97 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 8 op/s 2024-04-26T01:41:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 -- 172.21.15.28:0/2582958277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428104eb0 msgr2=0x7f4428105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 --2- 172.21.15.28:0/2582958277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428104eb0 0x7f4428105290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f44180098f0 tx=0x7f4418009c00 comp rx=0 tx=0).stop 2024-04-26T01:41:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 -- 172.21.15.28:0/2582958277 shutdown_connections 2024-04-26T01:41:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 --2- 172.21.15.28:0/2582958277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4428109f90 0x7f4428111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 --2- 172.21.15.28:0/2582958277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428104eb0 0x7f4428105290 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 --2- 172.21.15.28:0/2582958277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4428105860 0x7f44281098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 -- 172.21.15.28:0/2582958277 >> 172.21.15.28:0/2582958277 conn(0x7f4428100740 msgr2=0x7f4428102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 -- 172.21.15.28:0/2582958277 shutdown_connections 2024-04-26T01:41:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.412+0000 7f442d000700 1 -- 172.21.15.28:0/2582958277 wait complete. 2024-04-26T01:41:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.414+0000 7f442d000700 1 Processor -- start 2024-04-26T01:41:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.414+0000 7f442d000700 1 -- start start 2024-04-26T01:41:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.414+0000 7f442d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4428104eb0 0x7f442819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.414+0000 7f442d000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428105860 0x7f442819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.414+0000 7f442d000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4428109f90 0x7f442819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.414+0000 7f442d000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4428114e20 con 0x7f4428104eb0 2024-04-26T01:41:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.414+0000 7f442d000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4428114ca0 con 0x7f4428105860 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.414+0000 7f442d000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4428114fa0 con 0x7f4428109f90 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4428104eb0 0x7f442819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4428104eb0 0x7f442819b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34866/0 (socket says 172.21.15.28:34866) 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425e00700 1 -- 172.21.15.28:0/3811424957 learned_addr learned my addr 172.21.15.28:0/3811424957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4426800700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4428109f90 0x7f442819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425400700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428105860 0x7f442819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425e00700 1 -- 172.21.15.28:0/3811424957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4428109f90 msgr2=0x7f442819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425e00700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4428109f90 0x7f442819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425e00700 1 -- 172.21.15.28:0/3811424957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428105860 msgr2=0x7f442819b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425e00700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428105860 0x7f442819b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425e00700 1 -- 172.21.15.28:0/3811424957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44281a03a0 con 0x7f4428104eb0 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.415+0000 7f4425400700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428105860 0x7f442819b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.416+0000 7f4425e00700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4428104eb0 0x7f442819b390 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4418009230 tx=0x7f441800f9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.416+0000 7f441ea00700 1 -- 172.21.15.28:0/3811424957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418027070 con 0x7f4428104eb0 2024-04-26T01:41:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.416+0000 7f441ea00700 1 -- 172.21.15.28:0/3811424957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4418005570 con 0x7f4428104eb0 2024-04-26T01:41:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.416+0000 7f442d000700 1 -- 172.21.15.28:0/3811424957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44281a0630 con 0x7f4428104eb0 2024-04-26T01:41:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.416+0000 7f441ea00700 1 -- 172.21.15.28:0/3811424957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418004930 con 0x7f4428104eb0 2024-04-26T01:41:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.416+0000 7f442d000700 1 -- 172.21.15.28:0/3811424957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44281a7f10 con 0x7f4428104eb0 2024-04-26T01:41:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.420+0000 7f441ea00700 1 -- 172.21.15.28:0/3811424957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f441802a070 con 0x7f4428104eb0 2024-04-26T01:41:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.421+0000 7f4437a0cb80 1 Processor -- start 2024-04-26T01:41:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f441ea00700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f440c0a3f50 0x7f440c0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4437a0cb80 1 -- start start 2024-04-26T01:41:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f441ea00700 1 -- 172.21.15.28:0/3811424957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f44180d14b0 con 0x7f4428104eb0 2024-04-26T01:41:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c5bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4426800700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f440c0a3f50 0x7f440c0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5c0a0 0x562775c559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c56330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562775ba7e40 con 0x562775c5b780 2024-04-26T01:41:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562775b4e6e0 con 0x562775c5c0a0 2024-04-26T01:41:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562775ba3e00 con 0x562775c55ef0 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.422+0000 7f4425400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5c0a0 0x562775c559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5c0a0 0x562775c559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39720/0 (socket says 172.21.15.28:39720) 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425400700 1 -- 172.21.15.28:0/104200691 learned_addr learned my addr 172.21.15.28:0/104200691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425e00700 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c5bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4426800700 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c56330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425400700 1 -- 172.21.15.28:0/104200691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 msgr2=0x562775c56330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425400700 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c56330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425400700 1 -- 172.21.15.28:0/104200691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 msgr2=0x562775c5bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425400700 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c5bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425400700 1 -- 172.21.15.28:0/104200691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562775b44630 con 0x562775c5c0a0 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425e00700 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c5bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4426800700 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c56330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:41:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4426800700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f440c0a3f50 0x7f440c0a6410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f442000f540 tx=0x7f4420009920 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.423+0000 7f4425400700 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5c0a0 0x562775c559b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f44140115f0 tx=0x7f4414011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4861c999f66f655c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.424+0000 7f4413400700 1 -- 172.21.15.28:0/104200691 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4414004a30 con 0x562775c5c0a0 2024-04-26T01:41:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.424+0000 7f4413400700 1 -- 172.21.15.28:0/104200691 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44140054d0 con 0x562775c5c0a0 2024-04-26T01:41:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.424+0000 7f4413400700 1 -- 172.21.15.28:0/104200691 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441400f9c0 con 0x562775c5c0a0 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 -- 172.21.15.28:0/104200691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5c0a0 msgr2=0x562775c559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5c0a0 0x562775c559b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f44140115f0 tx=0x7f4414011900 comp rx=0 tx=0).stop 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 -- 172.21.15.28:0/104200691 shutdown_connections 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c56330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c5bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/104200691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5c0a0 0x562775c559b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 -- 172.21.15.28:0/104200691 >> 172.21.15.28:0/104200691 conn(0x562775b83f00 msgr2=0x562775c508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 -- 172.21.15.28:0/104200691 shutdown_connections 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 -- 172.21.15.28:0/104200691 wait complete. 2024-04-26T01:41:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.425+0000 7f4437a0cb80 1 -- start start 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.426+0000 7f4437a0cb80 1 Processor -- start 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.426+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c55ef0 0x562775c61ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.426+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c5b780 0x562775c5efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.426+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c5f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.426+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562775c62920 con 0x562775c5f4e0 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.426+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562775b61810 con 0x562775c55ef0 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.426+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562775c62aa0 con 0x562775c5b780 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.426+0000 7f4437a0cb80 10 client.0 _ll_register_callbacks cb 0x562775b406d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4426800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c5f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4425e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c55ef0 0x562775c61ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4426800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c5f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34884/0 (socket says 172.21.15.28:34884) 2024-04-26T01:41:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4425e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c55ef0 0x562775c61ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39726/0 (socket says 172.21.15.28:39726) 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4426800700 1 -- 172.21.15.28:0/300533636 learned_addr learned my addr 172.21.15.28:0/300533636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4425400700 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c5b780 0x562775c5efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4425e00700 1 -- 172.21.15.28:0/300533636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c5b780 msgr2=0x562775c5efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4425e00700 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c5b780 0x562775c5efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4425e00700 1 -- 172.21.15.28:0/300533636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 msgr2=0x562775c5f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4425e00700 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c5f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4425e00700 1 -- 172.21.15.28:0/300533636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562775af5880 con 0x562775c55ef0 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.427+0000 7f4426800700 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c5f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4425e00700 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c55ef0 0x562775c61ca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f44180cea50 tx=0x7f441809cfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80bfe2b8687e00cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4407e00700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:41:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4407e00700 1 -- 172.21.15.28:0/300533636 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44180b5070 con 0x562775c55ef0 2024-04-26T01:41:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4407e00700 1 -- 172.21.15.28:0/300533636 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f441809d5e0 con 0x562775c55ef0 2024-04-26T01:41:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4437a0cb80 1 -- 172.21.15.28:0/300533636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44140054d0 con 0x562775c55ef0 2024-04-26T01:41:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4407e00700 1 -- 172.21.15.28:0/300533636 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f44180a1730 con 0x562775c55ef0 2024-04-26T01:41:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4437a0cb80 10 client.25235 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4437a0cb80 1 -- 172.21.15.28:0/300533636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562775b3eaa0 con 0x562775c55ef0 2024-04-26T01:41:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.428+0000 7f4407e00700 1 -- 172.21.15.28:0/300533636 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44140054d0 con 0x562775c55ef0 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.429+0000 7f4407e00700 1 -- 172.21.15.28:0/300533636 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f44180ced30 con 0x562775c55ef0 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.429+0000 7f4437a0cb80 10 client.25235 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.429+0000 7f4437a0cb80 10 client.25235 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.429+0000 7f4437a0cb80 20 client.25235 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.429+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x562775c6fa40 0x562775c70eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.429+0000 7f4437a0cb80 4 client.25235 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.429+0000 7f4437a0cb80 1 -- 172.21.15.28:0/300533636 --> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562775bb5630 con 0x562775c6fa40 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.430+0000 7f4425400700 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x562775c6fa40 0x562775c70eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.431+0000 7f4425400700 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x562775c6fa40 0x562775c70eb0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a38de9f8e5c28049 server_cookie=e195f3a571099639 in_seq=0 out_seq=0 2024-04-26T01:41:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.431+0000 7f4407e00700 10 client.25235 ms_handle_connect on v2:172.21.15.53:6838/955096275 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4407e00700 1 -- 172.21.15.28:0/300533636 <== mds.1 v2:172.21.15.53:6838/955096275 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562775bb5630 con 0x562775c6fa40 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4407e00700 10 client.25235 handle_command_reply: tid=0 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 client.25235 shutdown 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 -- 172.21.15.28:0/300533636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c55ef0 msgr2=0x562775c61ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c55ef0 0x562775c61ca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f44180cea50 tx=0x7f441809cfb0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 -- 172.21.15.28:0/300533636 shutdown_connections 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x562775c6fa40 0x562775c70eb0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c5f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4407e00700 0 client.25235 ms_handle_reset on v2:172.21.15.53:6838/955096275 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c55ef0 0x562775c61ca0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/300533636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c5b780 0x562775c5efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.432+0000 7f4437a0cb80 1 -- 172.21.15.28:0/300533636 >> 172.21.15.28:0/300533636 conn(0x562775b83f00 msgr2=0x562775c4fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.433+0000 7f4437a0cb80 1 -- 172.21.15.28:0/300533636 shutdown_connections 2024-04-26T01:41:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.433+0000 7f4437a0cb80 1 -- 172.21.15.28:0/300533636 wait complete. 2024-04-26T01:41:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.433+0000 7f4437a0cb80 20 client.25235 trim_cache size 0 max 16384 2024-04-26T01:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.457+0000 7f4437a0cb80 1 Processor -- start 2024-04-26T01:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.457+0000 7f4437a0cb80 1 -- start start 2024-04-26T01:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.457+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c52030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5f4e0 0x562775c505d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562775c6f1f0 con 0x562775c5b780 2024-04-26T01:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562775c6f8c0 con 0x562775c5f4e0 2024-04-26T01:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562775c6ff90 con 0x562775c55ef0 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c52030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4426800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5f4e0 0x562775c505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59836/0 (socket says 172.21.15.28:59836) 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4426800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5f4e0 0x562775c505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39744/0 (socket says 172.21.15.28:39744) 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425e00700 1 -- 172.21.15.28:0/3875243820 learned_addr learned my addr 172.21.15.28:0/3875243820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425e00700 1 -- 172.21.15.28:0/3875243820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5f4e0 msgr2=0x562775c505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425e00700 1 --2- 172.21.15.28:0/3875243820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5f4e0 0x562775c505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425e00700 1 -- 172.21.15.28:0/3875243820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 msgr2=0x562775c52030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425e00700 1 --2- 172.21.15.28:0/3875243820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c52030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.458+0000 7f4425e00700 1 -- 172.21.15.28:0/3875243820 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562775c5f2d0 con 0x562775c55ef0 2024-04-26T01:41:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.459+0000 7f4425e00700 1 --2- 172.21.15.28:0/3875243820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c559b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f441809b630 tx=0x7f441800fb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e37c5b3164a02aa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.459+0000 7f4412a00700 1 -- 172.21.15.28:0/3875243820 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44180b5070 con 0x562775c55ef0 2024-04-26T01:41:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.459+0000 7f4412a00700 1 -- 172.21.15.28:0/3875243820 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44180a5f60 con 0x562775c55ef0 2024-04-26T01:41:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.459+0000 7f4412a00700 1 -- 172.21.15.28:0/3875243820 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44180990d0 con 0x562775c55ef0 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3875243820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 msgr2=0x562775c559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3875243820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c559b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f441809b630 tx=0x7f441800fb50 comp rx=0 tx=0).stop 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3875243820 shutdown_connections 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3875243820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5f4e0 0x562775c505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3875243820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c559b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3875243820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5b780 0x562775c52030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3875243820 >> 172.21.15.28:0/3875243820 conn(0x562775b83f00 msgr2=0x562775c562d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3875243820 shutdown_connections 2024-04-26T01:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.460+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3875243820 wait complete. 2024-04-26T01:41:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 -- 172.21.15.28:0/3312316087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 msgr2=0x7fe5dc074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 --2- 172.21.15.28:0/3312316087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 0x7fe5dc074630 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009900 tx=0x7fe5d002a480 comp rx=0 tx=0).stop 2024-04-26T01:41:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 -- 172.21.15.28:0/3312316087 shutdown_connections 2024-04-26T01:41:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 --2- 172.21.15.28:0/3312316087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5dc10e840 0x7fe5dc110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 --2- 172.21.15.28:0/3312316087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5dc074b70 0x7fe5dc10e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 --2- 172.21.15.28:0/3312316087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 0x7fe5dc074630 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 -- 172.21.15.28:0/3312316087 >> 172.21.15.28:0/3312316087 conn(0x7fe5dc0793e0 msgr2=0x7fe5dc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 -- 172.21.15.28:0/3312316087 shutdown_connections 2024-04-26T01:41:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.461+0000 7f4437a0cb80 1 -- start start 2024-04-26T01:41:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.461+0000 7fe5e3800640 1 -- 172.21.15.28:0/3312316087 wait complete. 2024-04-26T01:41:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.461+0000 7f4437a0cb80 1 Processor -- start 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c7a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5b780 0x562775c7a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4437a0cb80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c75300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562775c70d30 con 0x562775c5f4e0 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562775c70660 con 0x562775c5b780 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4437a0cb80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562775c71400 con 0x562775c55ef0 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4437a0cb80 10 client.0 _ll_register_callbacks cb 0x562775b406d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4425e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c7a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4425e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c7a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59850/0 (socket says 172.21.15.28:59850) 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4425400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5b780 0x562775c7a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.462+0000 7f4425e00700 1 -- 172.21.15.28:0/3520494784 learned_addr learned my addr 172.21.15.28:0/3520494784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.463+0000 7fe5e3800640 1 Processor -- start 2024-04-26T01:41:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4426800700 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c75300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4425400700 1 -- 172.21.15.28:0/3520494784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 msgr2=0x562775c7a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.463+0000 7fe5e3800640 1 -- start start 2024-04-26T01:41:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4425400700 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c7a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4425400700 1 -- 172.21.15.28:0/3520494784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 msgr2=0x562775c75300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4425400700 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c75300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4425400700 1 -- 172.21.15.28:0/3520494784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562775c75b40 con 0x562775c5b780 2024-04-26T01:41:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4425e00700 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c7a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4425400700 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5b780 0x562775c7a870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4414009a30 tx=0x7f4414005ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78e12d962b6cf53e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.463+0000 7f4407e00700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:41:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e3800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 0x7fe5dc1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e3800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5dc074b70 0x7fe5dc1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e3800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5dc10e840 0x7fe5dc19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e3800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc114430 con 0x7fe5dc074250 2024-04-26T01:41:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e3800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1142b0 con 0x7fe5dc10e840 2024-04-26T01:41:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e3800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1145b0 con 0x7fe5dc074b70 2024-04-26T01:41:40.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.464+0000 7f4407e00700 1 -- 172.21.15.28:0/3520494784 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441402cd60 con 0x562775c5b780 2024-04-26T01:41:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e1800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5dc10e840 0x7fe5dc19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 0x7fe5dc1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e1800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5dc10e840 0x7fe5dc19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39768/0 (socket says 172.21.15.28:39768) 2024-04-26T01:41:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 0x7fe5dc1a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34930/0 (socket says 172.21.15.28:34930) 2024-04-26T01:41:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5e1800640 1 -- 172.21.15.28:0/181643051 learned_addr learned my addr 172.21.15.28:0/181643051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.464+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5dc074b70 0x7fe5dc1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.464+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3520494784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44180a5f60 con 0x562775c5b780 2024-04-26T01:41:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.464+0000 7f4407e00700 1 -- 172.21.15.28:0/3520494784 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f44140050c0 con 0x562775c5b780 2024-04-26T01:41:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.465+0000 7fe5e1800640 1 -- 172.21.15.28:0/181643051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5dc074b70 msgr2=0x7fe5dc1a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.465+0000 7fe5e1800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5dc074b70 0x7fe5dc1a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.465+0000 7fe5e1800640 1 -- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 msgr2=0x7fe5dc1a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.465+0000 7fe5e1800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 0x7fe5dc1a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4407e00700 1 -- 172.21.15.28:0/3520494784 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f441403c050 con 0x562775c5b780 2024-04-26T01:41:40.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.465+0000 7fe5e1800640 1 -- 172.21.15.28:0/181643051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5dc19ae40 con 0x7fe5dc10e840 2024-04-26T01:41:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4437a0cb80 10 client.25244 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3520494784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562775cdfee0 con 0x562775c5b780 2024-04-26T01:41:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4407e00700 1 -- 172.21.15.28:0/3520494784 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44180a5f60 con 0x562775c5b780 2024-04-26T01:41:40.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.465+0000 7fe5e1800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5dc10e840 0x7fe5dc19a670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400ea60 tx=0x7fe5d400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4407e00700 1 -- 172.21.15.28:0/3520494784 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f441400f780 con 0x562775c5b780 2024-04-26T01:41:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4437a0cb80 10 client.25244 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4437a0cb80 10 client.25244 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:41:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4437a0cb80 20 client.25244 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x562775ce4910 0x562775ce3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4437a0cb80 4 client.25244 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:41:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.465+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3520494784 --> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] -- command(tid 0: {"prefix": "status"}) -- 0x562775baf6d0 con 0x562775ce4910 2024-04-26T01:41:40.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.466+0000 7fe5d9600640 1 -- 172.21.15.28:0/181643051 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d400ce40 con 0x7fe5dc10e840 2024-04-26T01:41:40.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.466+0000 7fe5d9600640 1 -- 172.21.15.28:0/181643051 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5d4005020 con 0x7fe5dc10e840 2024-04-26T01:41:40.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.466+0000 7fe5d9600640 1 -- 172.21.15.28:0/181643051 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4010670 con 0x7fe5dc10e840 2024-04-26T01:41:40.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.466+0000 7f4425e00700 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x562775ce4910 0x562775ce3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.466+0000 7fe5e3800640 1 -- 172.21.15.28:0/181643051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5dc19b130 con 0x7fe5dc10e840 2024-04-26T01:41:40.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.466+0000 7fe5e3800640 1 -- 172.21.15.28:0/181643051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5dc1a7330 con 0x7fe5dc10e840 2024-04-26T01:41:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.466+0000 7f4425e00700 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x562775ce4910 0x562775ce3620 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69ea0e2c62ea9340 server_cookie=4127c547af73e699 in_seq=0 out_seq=0 2024-04-26T01:41:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.466+0000 7f4407e00700 10 client.25244 ms_handle_connect on v2:172.21.15.53:6838/955096275 2024-04-26T01:41:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.467+0000 7f4407e00700 1 -- 172.21.15.28:0/3520494784 <== mds.1 v2:172.21.15.53:6838/955096275 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562775baf6d0 con 0x562775ce4910 2024-04-26T01:41:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.467+0000 7f4407e00700 10 client.25244 handle_command_reply: tid=0 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.467+0000 7f4437a0cb80 1 client.25244 shutdown 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3520494784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5b780 msgr2=0x562775c7a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5b780 0x562775c7a870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4414009a30 tx=0x7f4414005ab0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3520494784 shutdown_connections 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.53:6838/955096275,v1:172.21.15.53:6839/955096275] conn(0x562775ce4910 0x562775ce3620 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562775c5f4e0 0x562775c75300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562775c55ef0 0x562775c7a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4437a0cb80 1 --2- 172.21.15.28:0/3520494784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562775c5b780 0x562775c7a870 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3520494784 >> 172.21.15.28:0/3520494784 conn(0x562775b83f00 msgr2=0x562775c5c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.468+0000 7f4407e00700 0 client.25244 ms_handle_reset on v2:172.21.15.53:6838/955096275 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.469+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3520494784 shutdown_connections 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.469+0000 7f4437a0cb80 1 -- 172.21.15.28:0/3520494784 wait complete. 2024-04-26T01:41:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.469+0000 7f4437a0cb80 20 client.25244 trim_cache size 0 max 16384 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24509, 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 267.09153186100002, 2024-04-26T01:41:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 25, 2024-04-26T01:41:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-04-26T01:41:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:41:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 304.60698366600002 2024-04-26T01:41:40.482 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:41:40.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 -- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f440c0a3f50 msgr2=0x7f440c0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f440c0a3f50 0x7f440c0a6410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f442000f540 tx=0x7f4420009920 comp rx=0 tx=0).stop 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 -- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4428104eb0 msgr2=0x7f442819b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4428104eb0 0x7f442819b390 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4418009230 tx=0x7f441800f9b0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 -- 172.21.15.28:0/3811424957 shutdown_connections 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f440c0a3f50 0x7f440c0a6410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4428109f90 0x7f442819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4428104eb0 0x7f442819b390 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 --2- 172.21.15.28:0/3811424957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4428105860 0x7f442819b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 -- 172.21.15.28:0/3811424957 >> 172.21.15.28:0/3811424957 conn(0x7f4428100740 msgr2=0x7f4428109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 -- 172.21.15.28:0/3811424957 shutdown_connections 2024-04-26T01:41:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:40.470+0000 7f442d000700 1 -- 172.21.15.28:0/3811424957 wait complete. 2024-04-26T01:41:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.471+0000 7fe5e662a740 1 Processor -- start 2024-04-26T01:41:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.472+0000 7fe5d9600640 1 -- 172.21.15.28:0/181643051 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fe5d40054b0 con 0x7fe5dc10e840 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.472+0000 7fe5e662a740 1 -- start start 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.472+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.472+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e662a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d5740a900 con 0x555d574a4820 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e662a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555d573b0ef0 con 0x555d574a5140 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e662a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x555d574068c0 con 0x555d574afa00 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574af4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39776/0 (socket says 172.21.15.28:39776) 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e0e00640 1 -- 172.21.15.28:0/2609669980 learned_addr learned my addr 172.21.15.28:0/2609669980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e0e00640 1 -- 172.21.15.28:0/2609669980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 msgr2=0x555d574b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574b1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e0e00640 1 -- 172.21.15.28:0/2609669980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 msgr2=0x555d574a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5e0e00640 1 -- 172.21.15.28:0/2609669980 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d5711d210 con 0x555d574a5140 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.473+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.474+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574af4c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d0004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1663a7a177d1f04 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.474+0000 7fe5d9600640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe5c40a4040 0x7fe5c40a6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.474+0000 7fe5d9600640 1 -- 172.21.15.28:0/181643051 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fe5d40d0e40 con 0x7fe5dc10e840 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.474+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe5c40a4040 0x7fe5c40a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.474+0000 7fe5ce000640 1 -- 172.21.15.28:0/2609669980 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0035040 con 0x555d574a5140 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.474+0000 7fe5ce000640 1 -- 172.21.15.28:0/2609669980 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5d00054f0 con 0x555d574a5140 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.474+0000 7fe5ce000640 1 -- 172.21.15.28:0/2609669980 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d00035f0 con 0x555d574a5140 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 -- 172.21.15.28:0/2609669980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 msgr2=0x555d574af4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574af4c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d0004040 comp rx=0 tx=0).stop 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe5c40a4040 0x7fe5c40a6500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x555d574a2150 tx=0x7fe5c8015040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 -- 172.21.15.28:0/2609669980 shutdown_connections 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574af4c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 --2- 172.21.15.28:0/2609669980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 -- 172.21.15.28:0/2609669980 >> 172.21.15.28:0/2609669980 conn(0x555d574a0d30 msgr2=0x555d574a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 -- 172.21.15.28:0/2609669980 shutdown_connections 2024-04-26T01:41:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.475+0000 7fe5e662a740 1 -- 172.21.15.28:0/2609669980 wait complete. 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.476+0000 7fe5e662a740 1 -- start start 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.476+0000 7fe5e662a740 1 Processor -- start 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.476+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a4820 0x555d574ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.476+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.476+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574ad340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.476+0000 7fe5e662a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d574b25d0 con 0x555d574a5140 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e662a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555d573c4310 con 0x555d574a4820 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e662a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x555d574b2750 con 0x555d574afa00 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e662a740 10 client.0 _ll_register_callbacks cb 0x555d57392cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34954/0 (socket says 172.21.15.28:34954) 2024-04-26T01:41:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5dbe00640 1 -- 172.21.15.28:0/3803985239 learned_addr learned my addr 172.21.15.28:0/3803985239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e1800640 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a4820 0x555d574ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e1800640 1 -- 172.21.15.28:0/3803985239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a4820 msgr2=0x555d574ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e1800640 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a4820 0x555d574ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e1800640 1 -- 172.21.15.28:0/3803985239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 msgr2=0x555d574ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e1800640 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5e1800640 1 -- 172.21.15.28:0/3803985239 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d571e9620 con 0x555d574afa00 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.477+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.478+0000 7fe5e1800640 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574ad340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe5d40ce3e0 tx=0x7fe5d409cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=113008168c062d49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.478+0000 7fe5c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.478+0000 7fe5c2a00640 1 -- 172.21.15.28:0/3803985239 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d40a5340 con 0x555d574afa00 2024-04-26T01:41:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.478+0000 7fe5c2a00640 1 -- 172.21.15.28:0/3803985239 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fe5d40a54e0 con 0x555d574afa00 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.478+0000 7fe5e662a740 1 -- 172.21.15.28:0/3803985239 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5d00054f0 con 0x555d574afa00 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.478+0000 7fe5c2a00640 1 -- 172.21.15.28:0/3803985239 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe5d40ce6c0 con 0x555d574afa00 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.478+0000 7fe5e662a740 10 client.25294 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.478+0000 7fe5e662a740 1 -- 172.21.15.28:0/3803985239 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555d574be7d0 con 0x555d574afa00 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.479+0000 7fe5c2a00640 1 -- 172.21.15.28:0/3803985239 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5d00054f0 con 0x555d574afa00 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.479+0000 7fe5c2a00640 1 -- 172.21.15.28:0/3803985239 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fe5d409ba20 con 0x555d574afa00 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.479+0000 7fe5e662a740 10 client.25294 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.479+0000 7fe5e662a740 10 client.25294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.479+0000 7fe5e662a740 20 client.25294 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.479+0000 7fe5e662a740 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555d574bfc90 0x555d574c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.479+0000 7fe5e662a740 4 client.25294 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:41:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.479+0000 7fe5e662a740 1 -- 172.21.15.28:0/3803985239 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d5741e8b0 con 0x555d574bfc90 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.480+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555d574bfc90 0x555d574c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.481+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555d574bfc90 0x555d574c20c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=686aa909c3566c98 server_cookie=43374768b7893b7 in_seq=0 out_seq=0 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.481+0000 7fe5c2a00640 10 client.25294 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5c2a00640 1 -- 172.21.15.28:0/3803985239 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d5741e8b0 con 0x555d574bfc90 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5c2a00640 10 client.25294 handle_command_reply: tid=0 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 client.25294 shutdown 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 -- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 msgr2=0x555d574ad340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574ad340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe5d40ce3e0 tx=0x7fe5d409cc20 comp rx=0 tx=0).stop 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 -- 172.21.15.28:0/3803985239 shutdown_connections 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555d574bfc90 0x555d574c20c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574ad340 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5c2a00640 0 client.25294 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 --2- 172.21.15.28:0/3803985239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a4820 0x555d574ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 -- 172.21.15.28:0/3803985239 >> 172.21.15.28:0/3803985239 conn(0x555d574a0d30 msgr2=0x555d574a6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 -- 172.21.15.28:0/3803985239 shutdown_connections 2024-04-26T01:41:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 1 -- 172.21.15.28:0/3803985239 wait complete. 2024-04-26T01:41:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.482+0000 7fe5e662a740 20 client.25294 trim_cache size 0 max 16384 2024-04-26T01:41:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.500+0000 7fe5e662a740 1 Processor -- start 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.500+0000 7fe5e662a740 1 -- start start 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.500+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574a4820 0x555d574bb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.500+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.500+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574afa00 0x555d574c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.500+0000 7fe5e662a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1b2520 con 0x555d574a5140 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.500+0000 7fe5e662a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1afdf0 con 0x555d574afa00 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.500+0000 7fe5e662a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1adc40 con 0x555d574a4820 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ad160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34960/0 (socket says 172.21.15.28:34960) 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5e0e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574a4820 0x555d574bb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5dbe00640 1 -- 172.21.15.28:0/4022085503 learned_addr learned my addr 172.21.15.28:0/4022085503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5e1800640 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574afa00 0x555d574c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5e0e00640 1 -- 172.21.15.28:0/4022085503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574afa00 msgr2=0x555d574c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574afa00 0x555d574c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5e0e00640 1 -- 172.21.15.28:0/4022085503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 msgr2=0x555d574ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5e0e00640 1 -- 172.21.15.28:0/4022085503 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d5749b2a0 con 0x555d574a4820 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.501+0000 7fe5e1800640 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574afa00 0x555d574c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.502+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574a4820 0x555d574bb4c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0005ab0 tx=0x7fe5d0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46d35d32a1cce2f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.502+0000 7fe5cd600640 1 -- 172.21.15.28:0/4022085503 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0035040 con 0x555d574a4820 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.502+0000 7fe5cd600640 1 -- 172.21.15.28:0/4022085503 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5d00545d0 con 0x555d574a4820 2024-04-26T01:41:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.502+0000 7fe5cd600640 1 -- 172.21.15.28:0/4022085503 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0003ac0 con 0x555d574a4820 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 -- 172.21.15.28:0/4022085503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574a4820 msgr2=0x555d574bb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574a4820 0x555d574bb4c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0005ab0 tx=0x7fe5d0004ab0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 -- 172.21.15.28:0/4022085503 shutdown_connections 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574afa00 0x555d574c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a5140 0x555d574ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 --2- 172.21.15.28:0/4022085503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574a4820 0x555d574bb4c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 -- 172.21.15.28:0/4022085503 >> 172.21.15.28:0/4022085503 conn(0x555d574a0d30 msgr2=0x555d574a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 -- 172.21.15.28:0/4022085503 shutdown_connections 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.503+0000 7fe5e662a740 1 -- 172.21.15.28:0/4022085503 wait complete. 2024-04-26T01:41:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.504+0000 7fe5e662a740 1 -- start start 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.504+0000 7fe5e662a740 1 Processor -- start 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574b7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e662a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e662a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1ae2b0 con 0x555d574a4820 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e662a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1af720 con 0x555d574a5140 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e662a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1ae980 con 0x555d574afa00 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e662a740 10 client.0 _ll_register_callbacks cb 0x7fe5d409d260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574b7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34976/0 (socket says 172.21.15.28:34976) 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e0e00640 1 -- 172.21.15.28:0/168853942 learned_addr learned my addr 172.21.15.28:0/168853942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e1800640 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e0e00640 1 -- 172.21.15.28:0/168853942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 msgr2=0x555d574b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574b4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e0e00640 1 -- 172.21.15.28:0/168853942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 msgr2=0x555d574b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.505+0000 7fe5e0e00640 1 -- 172.21.15.28:0/168853942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d574b5250 con 0x555d574a4820 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.506+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.506+0000 7fe5e0e00640 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574b7640 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5d002a9a0 tx=0x7fe5d0009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1548aeb859f39d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.506+0000 7fe5c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.506+0000 7fe5c2a00640 1 -- 172.21.15.28:0/168853942 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0035040 con 0x555d574a4820 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.506+0000 7fe5c2a00640 1 -- 172.21.15.28:0/168853942 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fe5d0054120 con 0x555d574a4820 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.506+0000 7fe5e662a740 1 -- 172.21.15.28:0/168853942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5d00545d0 con 0x555d574a4820 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.506+0000 7fe5c2a00640 1 -- 172.21.15.28:0/168853942 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe5d0065050 con 0x555d574a4820 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5e662a740 10 client.15153 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5e662a740 1 -- 172.21.15.28:0/168853942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555d57525cb0 con 0x555d574a4820 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5c2a00640 1 -- 172.21.15.28:0/168853942 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5d00545d0 con 0x555d574a4820 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5c2a00640 1 -- 172.21.15.28:0/168853942 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fe5d00314a0 con 0x555d574a4820 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5e662a740 10 client.15153 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5e662a740 10 client.15153 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5e662a740 20 client.15153 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5e662a740 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555d57525ee0 0x555d575282d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5e662a740 4 client.15153 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:41:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.507+0000 7fe5e662a740 1 -- 172.21.15.28:0/168853942 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d57418320 con 0x555d57525ee0 2024-04-26T01:41:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.508+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555d57525ee0 0x555d575282d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.509+0000 7fe5dbe00640 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555d57525ee0 0x555d575282d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be848e8ae74525cb server_cookie=c61b4b6ebb80b523 in_seq=0 out_seq=0 2024-04-26T01:41:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.509+0000 7fe5c2a00640 10 client.15153 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.785+0000 7fe5c2a00640 1 -- 172.21.15.28:0/168853942 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d57418320 con 0x555d57525ee0 2024-04-26T01:41:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5c2a00640 10 client.15153 handle_command_reply: tid=0 2024-04-26T01:41:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 client.15153 shutdown 2024-04-26T01:41:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 -- 172.21.15.28:0/168853942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 msgr2=0x555d574b7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574b7640 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5d002a9a0 tx=0x7fe5d0009230 comp rx=0 tx=0).stop 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 -- 172.21.15.28:0/168853942 shutdown_connections 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555d57525ee0 0x555d575282d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555d574afa00 0x555d574b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555d574a5140 0x555d574b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5c2a00640 0 client.15153 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 --2- 172.21.15.28:0/168853942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d574a4820 0x555d574b7640 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 -- 172.21.15.28:0/168853942 >> 172.21.15.28:0/168853942 conn(0x555d574a0d30 msgr2=0x555d5749f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 -- 172.21.15.28:0/168853942 shutdown_connections 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 1 -- 172.21.15.28:0/168853942 wait complete. 2024-04-26T01:41:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.786+0000 7fe5e662a740 20 client.15153 trim_cache size 0 max 16384 2024-04-26T01:41:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:41:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:41:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:41:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 -- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe5c40a4040 msgr2=0x7fe5c40a6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe5c40a4040 0x7fe5c40a6500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x555d574a2150 tx=0x7fe5c8015040 comp rx=0 tx=0).stop 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 -- 172.21.15.28:0/181643051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5dc10e840 msgr2=0x7fe5dc19a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5dc10e840 0x7fe5dc19a670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400ea60 tx=0x7fe5d400ef20 comp rx=0 tx=0).stop 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 -- 172.21.15.28:0/181643051 shutdown_connections 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe5c40a4040 0x7fe5c40a6500 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5dc10e840 0x7fe5dc19a670 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5dc074b70 0x7fe5dc1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 --2- 172.21.15.28:0/181643051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc074250 0x7fe5dc1a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 -- 172.21.15.28:0/181643051 >> 172.21.15.28:0/181643051 conn(0x7fe5dc0793e0 msgr2=0x7fe5dc0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 -- 172.21.15.28:0/181643051 shutdown_connections 2024-04-26T01:41:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:40.787+0000 7fe5e3800640 1 -- 172.21.15.28:0/181643051 wait complete. 2024-04-26T01:41:41.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:41:41.712 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:41 smithi169 ceph-mon[32851]: pgmap v24: 97 pgs: 97 active+clean; 25 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-04-26T01:41:41.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:41 smithi053 ceph-mon[32593]: pgmap v24: 97 pgs: 97 active+clean; 25 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-04-26T01:41:41.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:41 smithi028 ceph-mon[28247]: pgmap v24: 97 pgs: 97 active+clean; 25 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-04-26T01:41:41.816 INFO:teuthology.orchestra.run.smithi028.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-26T01:41:41.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-26T01:41:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/: 2024-04-26T01:41:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:41:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 .. 2024-04-26T01:41:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 data 2024-04-26T01:41:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data: 2024-04-26T01:41:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:total 35 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 . 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 .. 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 base 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 26 01:41 global 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 log 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_commit_ts 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_dynshmem 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 26 01:41 pg_hba.conf 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 26 01:41 pg_ident.conf 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 pg_logical 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 pg_multixact 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_notify 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_replslot 2024-04-26T01:41:41.932 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_serial 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_snapshots 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_stat 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_stat_tmp 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 pg_subtrans 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_tblspc 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_twophase 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 PG_VERSION 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 pg_wal 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 pg_xact 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 26 01:41 postgresql.auto.conf 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 26 01:41 postgresql.conf 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:41.933 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base: 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 . 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 1 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 13493 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 13494 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:total 7777 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 . 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 .. 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 112 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 113 2024-04-26T01:41:41.934 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 1247 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1247_fsm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1247_vm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 01:41 1249 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1249_fsm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1249_vm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 01:41 1255 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1255_fsm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1255_vm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 01:41 1259 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1259_fsm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1259_vm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 01:41 13344 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13344_fsm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13344_vm 2024-04-26T01:41:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13346 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13348 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13349_fsm 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349_vm 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13351 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13353 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13354_fsm 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354_vm 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13356 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13358 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13359_fsm 2024-04-26T01:41:41.936 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359_vm 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13361 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13363 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1417 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1418 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 174 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 175 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2187 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2224 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2328 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2336 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2337 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2579 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2600 2024-04-26T01:41:41.937 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2600_fsm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2600_vm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2601_fsm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601_vm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2602 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2602_fsm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2602_vm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2603 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2603_fsm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2603_vm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2604 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2605 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2605_fsm 2024-04-26T01:41:41.938 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2605_vm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2606_fsm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606_vm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2607 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2607_fsm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2607_vm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 01:41 2608 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2608_fsm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2608_vm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 01:41 2609 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2609_fsm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2609_vm 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2610 2024-04-26T01:41:41.939 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2610_fsm 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2610_vm 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2611 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2612_fsm 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612_vm 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2613 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2615_fsm 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615_vm 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2616 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2616_fsm 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2616_vm 2024-04-26T01:41:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 01:41 2617 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2617_fsm 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2617_vm 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 01:41 2618 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2618_fsm 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2618_vm 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 01:41 2619 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2619_fsm 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2619_vm 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2620 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2650 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2651 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2652 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2653 2024-04-26T01:41:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2654 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2655 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2656 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2657 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 01:41 2658 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 2659 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2660 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2661 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2662 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2663 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2664 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2665 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2666 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2667 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2668 2024-04-26T01:41:41.942 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2669 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2670 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 01:41 2673 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 01:41 2674 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 01:41 2675 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2678 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2679 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2680 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2681 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2682 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2683 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2684 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2685 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2686 2024-04-26T01:41:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2687 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2688 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2689 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 01:41 2690 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 01:41 2691 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2692 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2693 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2696 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2699 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2701 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2702 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2703 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2704 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2753 2024-04-26T01:41:41.944 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2753_fsm 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2753_vm 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2754 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2755 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2756 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2757 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2830 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2831 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2832 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2833 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2834 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2835 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2836 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2837 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 01:41 2838 2024-04-26T01:41:41.945 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2838_fsm 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2838_vm 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2839 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840_fsm 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2840_vm 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2841 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2995 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2996 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3079_fsm 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079_vm 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3080 2024-04-26T01:41:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3081 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3085 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3118 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3119 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 01:41 3164 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3256 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3257 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3258 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3350 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3351 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3379 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3380 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3381 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3394 2024-04-26T01:41:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3394_fsm 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3394_vm 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3395 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3429 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3430 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3431 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3433 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3439 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3440 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3455 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 01:41 3456 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3456_fsm 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3456_vm 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3466 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3467 2024-04-26T01:41:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3468 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3501 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3502 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3503 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3534 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3541_fsm 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541_vm 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3542 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3574 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3575 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3576 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3596 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3597 2024-04-26T01:41:41.949 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3598 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3599 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3600_fsm 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600_vm 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3601_fsm 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601_vm 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3602_fsm 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602_vm 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603_fsm 2024-04-26T01:41:41.950 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3603_vm 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3604 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3605 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3606 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3607 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3608 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 3609 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3712 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3764_fsm 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764_vm 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3766 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3767 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3997 2024-04-26T01:41:41.951 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4143 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4144 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4145 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4146 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4147 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4148 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4149 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4150 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4151 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4152 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4153 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4154 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4155 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4156 2024-04-26T01:41:41.952 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4157 2024-04-26T01:41:41.953 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4158 2024-04-26T01:41:41.953 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4159 2024-04-26T01:41:41.953 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4160 2024-04-26T01:41:41.953 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4161 2024-04-26T01:41:41.953 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4162 2024-04-26T01:41:41.953 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4163 2024-04-26T01:41:41.953 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4164 2024-04-26T01:41:41.953 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4165 2024-04-26T01:41:41.954 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4166 2024-04-26T01:41:41.954 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4167 2024-04-26T01:41:41.954 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4168 2024-04-26T01:41:41.954 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4169 2024-04-26T01:41:41.954 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4170 2024-04-26T01:41:41.954 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4171 2024-04-26T01:41:41.954 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4172 2024-04-26T01:41:41.954 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4173 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4174 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 5002 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 548 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 549 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6102 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6104 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6106 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6110 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6111 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6112 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6113 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6117 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 826 2024-04-26T01:41:41.955 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 827 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 828 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 01:41 pg_filenode.map 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 PG_VERSION 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:total 7777 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 . 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 .. 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 112 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 113 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 1247 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1247_fsm 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1247_vm 2024-04-26T01:41:41.956 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 01:41 1249 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1249_fsm 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1249_vm 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 01:41 1255 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1255_fsm 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1255_vm 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 01:41 1259 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1259_fsm 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1259_vm 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 01:41 13344 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13344_fsm 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13344_vm 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13346 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13348 2024-04-26T01:41:41.957 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13349_fsm 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349_vm 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13351 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13353 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13354_fsm 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354_vm 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13356 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13358 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13359_fsm 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359_vm 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13361 2024-04-26T01:41:41.958 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13363 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1417 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1418 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 174 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 175 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2187 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2224 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2328 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2336 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2337 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2579 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2600 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2600_fsm 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2600_vm 2024-04-26T01:41:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2601_fsm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601_vm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2602 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2602_fsm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2602_vm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2603 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2603_fsm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2603_vm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2604 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2605 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2605_fsm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2605_vm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2606_fsm 2024-04-26T01:41:41.960 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606_vm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2607 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2607_fsm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2607_vm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 01:41 2608 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2608_fsm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2608_vm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 01:41 2609 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2609_fsm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2609_vm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2610 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2610_fsm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2610_vm 2024-04-26T01:41:41.961 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2611 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2612_fsm 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612_vm 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2613 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2615_fsm 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615_vm 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2616 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2616_fsm 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2616_vm 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 01:41 2617 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2617_fsm 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2617_vm 2024-04-26T01:41:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 01:41 2618 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2618_fsm 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2618_vm 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 01:41 2619 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2619_fsm 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2619_vm 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2620 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2650 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2651 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2652 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2653 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2654 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2655 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2656 2024-04-26T01:41:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2657 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 01:41 2658 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 2659 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2660 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2661 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2662 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2663 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2664 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2665 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2666 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2667 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2668 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2669 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2670 2024-04-26T01:41:41.964 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 01:41 2673 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 01:41 2674 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 01:41 2675 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2678 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2679 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2680 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2681 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2682 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2683 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2684 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2685 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2686 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2687 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2688 2024-04-26T01:41:41.965 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2689 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 01:41 2690 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 01:41 2691 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2692 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2693 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2696 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2699 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2701 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2702 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2703 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2704 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2753 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2753_fsm 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2753_vm 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2754 2024-04-26T01:41:41.966 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2755 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2756 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2757 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2830 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2831 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2832 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2833 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2834 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2835 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2836 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2837 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 01:41 2838 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2838_fsm 2024-04-26T01:41:41.967 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2838_vm 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2839 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840_fsm 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2840_vm 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2841 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2995 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2996 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3079_fsm 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079_vm 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3080 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3081 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3085 2024-04-26T01:41:41.968 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3118 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3119 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 01:41 3164 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3256 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3257 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3258 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3350 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3351 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3379 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3380 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3381 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3394 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3394_fsm 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3394_vm 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3395 2024-04-26T01:41:41.969 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3429 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3430 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3431 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3433 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3439 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3440 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3455 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 01:41 3456 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3456_fsm 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3456_vm 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3466 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3467 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3468 2024-04-26T01:41:41.970 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3501 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3502 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3503 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3534 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3541_fsm 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541_vm 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3542 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3574 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3575 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3576 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3596 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3597 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3598 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3599 2024-04-26T01:41:41.971 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3600_fsm 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600_vm 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3601_fsm 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601_vm 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3602_fsm 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602_vm 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603_fsm 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3603_vm 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3604 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3605 2024-04-26T01:41:41.972 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3606 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3607 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3608 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 3609 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3712 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3764_fsm 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764_vm 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3766 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3767 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3997 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4143 2024-04-26T01:41:41.973 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4144 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4145 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4146 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4147 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4148 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4149 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4150 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4151 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4152 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4153 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4154 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4155 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4156 2024-04-26T01:41:41.974 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4157 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4158 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4159 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4160 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4161 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4162 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4163 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4164 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4165 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4166 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4167 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4168 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4169 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4170 2024-04-26T01:41:41.975 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4171 2024-04-26T01:41:41.976 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4172 2024-04-26T01:41:41.976 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4173 2024-04-26T01:41:41.976 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4174 2024-04-26T01:41:41.976 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 5002 2024-04-26T01:41:41.976 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 548 2024-04-26T01:41:41.977 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 549 2024-04-26T01:41:41.977 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6102 2024-04-26T01:41:41.977 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6104 2024-04-26T01:41:41.977 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6106 2024-04-26T01:41:41.977 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6110 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6111 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6112 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6113 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6117 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 826 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 827 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 828 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 01:41 pg_filenode.map 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 PG_VERSION 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:total 7777 2024-04-26T01:41:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 . 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 .. 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 112 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 113 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 1247 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1247_fsm 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1247_vm 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 01:41 1249 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1249_fsm 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1249_vm 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 01:41 1255 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1255_fsm 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1255_vm 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 01:41 1259 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1259_fsm 2024-04-26T01:41:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1259_vm 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 01:41 13344 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13344_fsm 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13344_vm 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13346 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13348 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13349_fsm 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349_vm 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13351 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13353 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13354_fsm 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354_vm 2024-04-26T01:41:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13356 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13358 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13359_fsm 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359_vm 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13361 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13363 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1417 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1418 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 174 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 175 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2187 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2224 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2328 2024-04-26T01:41:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2336 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2337 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2579 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2600 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2600_fsm 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2600_vm 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2601_fsm 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601_vm 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2602 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2602_fsm 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2602_vm 2024-04-26T01:41:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2603 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2603_fsm 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2603_vm 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2604 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2605 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2605_fsm 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2605_vm 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2606_fsm 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606_vm 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2607 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2607_fsm 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2607_vm 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 01:41 2608 2024-04-26T01:41:41.983 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2608_fsm 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2608_vm 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 01:41 2609 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2609_fsm 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2609_vm 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2610 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2610_fsm 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2610_vm 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2611 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2612_fsm 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612_vm 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2613 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615 2024-04-26T01:41:41.984 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2615_fsm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615_vm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2616 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2616_fsm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2616_vm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 01:41 2617 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2617_fsm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2617_vm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 01:41 2618 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2618_fsm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2618_vm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 01:41 2619 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2619_fsm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2619_vm 2024-04-26T01:41:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2620 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2650 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2651 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2652 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2653 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2654 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2655 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2656 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2657 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 01:41 2658 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 2659 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2660 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2661 2024-04-26T01:41:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2662 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2663 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2664 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2665 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2666 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2667 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2668 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2669 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2670 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 01:41 2673 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 01:41 2674 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 01:41 2675 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2678 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2679 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2680 2024-04-26T01:41:41.987 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2681 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2682 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2683 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2684 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2685 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2686 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2687 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2688 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2689 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 01:41 2690 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 01:41 2691 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2692 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2693 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2696 2024-04-26T01:41:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2699 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2701 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2702 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2703 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2704 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2753 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2753_fsm 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2753_vm 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2754 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2755 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2756 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2757 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2830 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2831 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2832 2024-04-26T01:41:41.989 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2833 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2834 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2835 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2836 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2837 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 01:41 2838 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2838_fsm 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2838_vm 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2839 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840_fsm 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2840_vm 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2841 2024-04-26T01:41:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2995 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2996 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3079_fsm 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079_vm 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3080 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3081 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3085 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3118 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3119 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 01:41 3164 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3256 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3257 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3258 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3350 2024-04-26T01:41:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3351 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3379 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3380 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3381 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3394 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3394_fsm 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3394_vm 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3395 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3429 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3430 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3431 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3433 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3439 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3440 2024-04-26T01:41:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3455 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 01:41 3456 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3456_fsm 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3456_vm 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3466 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3467 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3468 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3501 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3502 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3503 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3534 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3541_fsm 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541_vm 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3542 2024-04-26T01:41:41.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3574 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3575 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3576 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3596 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3597 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3598 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3599 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3600_fsm 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600_vm 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3601_fsm 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601_vm 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602 2024-04-26T01:41:41.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3602_fsm 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602_vm 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603_fsm 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3603_vm 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3604 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3605 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3606 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3607 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3608 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 3609 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3712 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3764_fsm 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764_vm 2024-04-26T01:41:41.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3766 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3767 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3997 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4143 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4144 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4145 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4146 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4147 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4148 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4149 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4150 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4151 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4152 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4153 2024-04-26T01:41:41.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4154 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4155 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4156 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4157 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4158 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4159 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4160 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4161 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4162 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4163 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4164 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4165 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4166 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4167 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4168 2024-04-26T01:41:41.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4169 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4170 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4171 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4172 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4173 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4174 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 5002 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 548 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 549 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6102 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6104 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6106 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6110 2024-04-26T01:41:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6111 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6112 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6113 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6117 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 826 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 827 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 828 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 01:41 pg_filenode.map 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 PG_VERSION 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/global: 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:total 537 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 26 01:41 . 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:41.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1213 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1213_fsm 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1213_vm 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1214 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1214_fsm 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1214_vm 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 1232 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 1233 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1260 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1260_fsm 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1260_vm 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1261 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1261_fsm 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1261_vm 2024-04-26T01:41:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1262 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1262_fsm 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1262_vm 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2396 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2396_fsm 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2396_vm 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2397 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2671 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2672 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2676 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2677 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2694 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2695 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2697 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2698 2024-04-26T01:41:42.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2846 2024-04-26T01:41:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2847 2024-04-26T01:41:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2964 2024-04-26T01:41:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2965 2024-04-26T01:41:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2966 2024-04-26T01:41:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2967 2024-04-26T01:41:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3592 2024-04-26T01:41:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3593 2024-04-26T01:41:42.002 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4060 2024-04-26T01:41:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4061 2024-04-26T01:41:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4175 2024-04-26T01:41:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4176 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4177 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4178 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4181 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4182 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4183 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4184 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4185 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4186 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6000 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6001 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6002 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6100 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6114 2024-04-26T01:41:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6115 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 pg_control 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 01:41 pg_filenode.map 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/log: 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:total 1 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 . 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 mappings 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 26 01:41 replorigin_checkpoint 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 snapshots 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 .. 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 .. 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 . 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 members 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 offsets 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:total 8 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 .. 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 0000 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:total 8 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 .. 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 0000 2024-04-26T01:41:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.010 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:total 8 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 0000 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-26T01:41:42.011 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:total 16384 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 . 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:41 000000010000000000000001 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 archive_status 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 .. 2024-04-26T01:41:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:41:42.013 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-26T01:41:42.013 INFO:teuthology.orchestra.run.smithi028.stdout:total 8 2024-04-26T01:41:42.013 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:41:42.013 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 01:41 .. 2024-04-26T01:41:42.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 0000 2024-04-26T01:41:42.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-26T01:41:42.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-26T01:41:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:dropping old tables... 2024-04-26T01:41:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-26T01:41:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-26T01:41:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-26T01:41:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-26T01:41:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:creating tables... 2024-04-26T01:41:42.480 INFO:teuthology.orchestra.run.smithi028.stderr:generating data (client-side)... 2024-04-26T01:41:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.11 s, remaining 3.52 s) 2024-04-26T01:41:43.102 INFO:teuthology.orchestra.run.smithi028.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.57 s, remaining 8.56 s) 2024-04-26T01:41:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.87 s, remaining 8.45 s) 2024-04-26T01:41:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:400000 of 3200000 tuples (12%) done (elapsed 1.07 s, remaining 7.49 s) 2024-04-26T01:41:43.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:43 smithi053 ceph-mon[32593]: pgmap v25: 97 pgs: 97 active+clean; 25 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 2.0 MiB/s wr, 31 op/s 2024-04-26T01:41:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:43 smithi028 ceph-mon[28247]: pgmap v25: 97 pgs: 97 active+clean; 25 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 2.0 MiB/s wr, 31 op/s 2024-04-26T01:41:43.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:43 smithi169 ceph-mon[32851]: pgmap v25: 97 pgs: 97 active+clean; 25 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 2.0 MiB/s wr, 31 op/s 2024-04-26T01:41:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:500000 of 3200000 tuples (15%) done (elapsed 1.40 s, remaining 7.58 s) 2024-04-26T01:41:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:600000 of 3200000 tuples (18%) done (elapsed 1.79 s, remaining 7.74 s) 2024-04-26T01:41:44.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 -- 172.21.15.28:0/1143984807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618075590 msgr2=0x7f36181136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 --2- 172.21.15.28:0/1143984807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618075590 0x7f36181136d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3608009990 tx=0x7f3608009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 -- 172.21.15.28:0/1143984807 shutdown_connections 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 --2- 172.21.15.28:0/1143984807 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3618113c10 0x7f3618116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 --2- 172.21.15.28:0/1143984807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3618074c70 0x7f3618075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 --2- 172.21.15.28:0/1143984807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618075590 0x7f36181136d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 -- 172.21.15.28:0/1143984807 >> 172.21.15.28:0/1143984807 conn(0x7f3618079bf0 msgr2=0x7f361807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 -- 172.21.15.28:0/1143984807 shutdown_connections 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.368+0000 7f3620800700 1 -- 172.21.15.28:0/1143984807 wait complete. 2024-04-26T01:41:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.370+0000 7f3620800700 1 Processor -- start 2024-04-26T01:41:44.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.370+0000 7f3620800700 1 -- start start 2024-04-26T01:41:44.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.370+0000 7f3620800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3618074c70 0x7f3618074230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.370+0000 7f3620800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3618075590 0x7f361806d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.370+0000 7f3620800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618113c10 0x7f361806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f3620800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3618119260 con 0x7f3618113c10 2024-04-26T01:41:44.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f3620800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f36181190e0 con 0x7f3618074c70 2024-04-26T01:41:44.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f3620800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f36181193e0 con 0x7f3618075590 2024-04-26T01:41:44.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3618074c70 0x7f3618074230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361d400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3618075590 0x7f361806d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3618074c70 0x7f3618074230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39806/0 (socket says 172.21.15.28:39806) 2024-04-26T01:41:44.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361de00700 1 -- 172.21.15.28:0/2412493192 learned_addr learned my addr 172.21.15.28:0/2412493192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:44.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361de00700 1 -- 172.21.15.28:0/2412493192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3618075590 msgr2=0x7f361806d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361de00700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3618075590 0x7f361806d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361e800700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618113c10 0x7f361806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361de00700 1 -- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618113c10 msgr2=0x7f361806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361de00700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618113c10 0x7f361806d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361de00700 1 -- 172.21.15.28:0/2412493192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361806de90 con 0x7f3618074c70 2024-04-26T01:41:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.371+0000 7f361e800700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618113c10 0x7f361806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:41:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.372+0000 7f361de00700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3618074c70 0x7f3618074230 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f360c00eb30 tx=0x7f360c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.372+0000 7f3612a00700 1 -- 172.21.15.28:0/2412493192 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c00cc90 con 0x7f3618074c70 2024-04-26T01:41:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.372+0000 7f3612a00700 1 -- 172.21.15.28:0/2412493192 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f360c0044d0 con 0x7f3618074c70 2024-04-26T01:41:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.372+0000 7f3612a00700 1 -- 172.21.15.28:0/2412493192 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c010430 con 0x7f3618074c70 2024-04-26T01:41:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.372+0000 7f3620800700 1 -- 172.21.15.28:0/2412493192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3618071d60 con 0x7f3618074c70 2024-04-26T01:41:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.373+0000 7f3620800700 1 -- 172.21.15.28:0/2412493192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3618072250 con 0x7f3618074c70 2024-04-26T01:41:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.376+0000 7f3612a00700 1 -- 172.21.15.28:0/2412493192 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f360c003ea0 con 0x7f3618074c70 2024-04-26T01:41:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.378+0000 7f3612a00700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f36000a3f50 0x7f36000a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.378+0000 7f3612a00700 1 -- 172.21.15.28:0/2412493192 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f360c0d0cf0 con 0x7f3618074c70 2024-04-26T01:41:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.378+0000 7f361d400700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f36000a3f50 0x7f36000a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.379+0000 7f361d400700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f36000a3f50 0x7f36000a6410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f361806ea80 tx=0x7f360800b260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.380+0000 7f362b1a4b80 1 Processor -- start 2024-04-26T01:41:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.380+0000 7f362b1a4b80 1 -- start start 2024-04-26T01:41:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.380+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe12a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.380+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe135570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.380+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe137ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.380+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565fe07de40 con 0x5565fe135ab0 2024-04-26T01:41:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.380+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5565fe0246e0 con 0x5565fe12a2f0 2024-04-26T01:41:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.380+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5565fe079e00 con 0x5565fe12ac10 2024-04-26T01:41:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361d400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361d400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe135570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59942/0 (socket says 172.21.15.28:59942) 2024-04-26T01:41:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361d400700 1 -- 172.21.15.28:0/266825602 learned_addr learned my addr 172.21.15.28:0/266825602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361de00700 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe12a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361e800700 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe137ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361d400700 1 -- 172.21.15.28:0/266825602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 msgr2=0x5565fe12a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361d400700 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe12a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361d400700 1 -- 172.21.15.28:0/266825602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 msgr2=0x5565fe137ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361d400700 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe137ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.381+0000 7f361d400700 1 -- 172.21.15.28:0/266825602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565fe01a630 con 0x5565fe12ac10 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.382+0000 7f361d400700 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe135570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3608015000 tx=0x7f360801f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1874bf6ecfdcba1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.382+0000 7f3607400700 1 -- 172.21.15.28:0/266825602 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360800bdb0 con 0x5565fe12ac10 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.382+0000 7f3607400700 1 -- 172.21.15.28:0/266825602 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36080095c0 con 0x5565fe12ac10 2024-04-26T01:41:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.382+0000 7f3607400700 1 -- 172.21.15.28:0/266825602 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36080308b0 con 0x5565fe12ac10 2024-04-26T01:41:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 -- 172.21.15.28:0/266825602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 msgr2=0x5565fe135570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe135570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3608015000 tx=0x7f360801f000 comp rx=0 tx=0).stop 2024-04-26T01:41:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 -- 172.21.15.28:0/266825602 shutdown_connections 2024-04-26T01:41:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe137ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe12a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/266825602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe135570 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 -- 172.21.15.28:0/266825602 >> 172.21.15.28:0/266825602 conn(0x5565fe059550 msgr2=0x5565fe126790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 -- 172.21.15.28:0/266825602 shutdown_connections 2024-04-26T01:41:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 -- 172.21.15.28:0/266825602 wait complete. 2024-04-26T01:41:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.383+0000 7f362b1a4b80 1 -- start start 2024-04-26T01:41:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.384+0000 7f362b1a4b80 1 Processor -- start 2024-04-26T01:41:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.384+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe1351e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.384+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1324e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.384+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe135ab0 0x5565fe132a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.384+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565fe138540 con 0x5565fe12ac10 2024-04-26T01:41:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.384+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5565fe037810 con 0x5565fe12a2f0 2024-04-26T01:41:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.384+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5565fe1386c0 con 0x5565fe135ab0 2024-04-26T01:41:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.384+0000 7f362b1a4b80 10 client.0 _ll_register_callbacks cb 0x5565fe0166d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe1351e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe1351e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39824/0 (socket says 172.21.15.28:39824) 2024-04-26T01:41:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 -- 172.21.15.28:0/2984224133 learned_addr learned my addr 172.21.15.28:0/2984224133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361d400700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1324e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361e800700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe135ab0 0x5565fe132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 -- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe135ab0 msgr2=0x5565fe132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe135ab0 0x5565fe132a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 -- 172.21.15.28:0/2984224133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 msgr2=0x5565fe1324e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1324e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 -- 172.21.15.28:0/2984224133 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565fdfcb880 con 0x5565fe12a2f0 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361d400700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1324e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361e800700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe135ab0 0x5565fe132a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f361de00700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe1351e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f360c09c710 tx=0x7f360c09c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=791bf9e552f87150 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f35fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.385+0000 7f35fbe00700 1 -- 172.21.15.28:0/2984224133 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c0bd070 con 0x5565fe12a2f0 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.386+0000 7f35fbe00700 1 -- 172.21.15.28:0/2984224133 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f360c09cc70 con 0x5565fe12a2f0 2024-04-26T01:41:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.386+0000 7f362b1a4b80 1 -- 172.21.15.28:0/2984224133 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5565fe08b630 con 0x5565fe12a2f0 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.386+0000 7f35fbe00700 1 -- 172.21.15.28:0/2984224133 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f360c09ca10 con 0x5565fe12a2f0 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.386+0000 7f362b1a4b80 10 client.25271 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.386+0000 7f362b1a4b80 1 -- 172.21.15.28:0/2984224133 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5565fe014aa0 con 0x5565fe12a2f0 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.386+0000 7f35fbe00700 1 -- 172.21.15.28:0/2984224133 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5565fe08b630 con 0x5565fe12a2f0 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.386+0000 7f35fbe00700 1 -- 172.21.15.28:0/2984224133 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f360c0a51f0 con 0x5565fe12a2f0 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.386+0000 7f362b1a4b80 10 client.25271 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.387+0000 7f362b1a4b80 10 client.25271 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.387+0000 7f362b1a4b80 20 client.25271 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.387+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5565fe145220 0x5565fe146690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.387+0000 7f362b1a4b80 4 client.25271 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:41:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.387+0000 7f362b1a4b80 1 -- 172.21.15.28:0/2984224133 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565fe0856d0 con 0x5565fe145220 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.387+0000 7f361d400700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5565fe145220 0x5565fe146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.388+0000 7f361d400700 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5565fe145220 0x5565fe146690 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a905c46c4f30c5c8 server_cookie=5915ce06e8bc8500 in_seq=0 out_seq=0 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.388+0000 7f35fbe00700 10 client.25271 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.388+0000 7f35fbe00700 1 -- 172.21.15.28:0/2984224133 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565fe0856d0 con 0x5565fe145220 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f35fbe00700 10 client.25271 handle_command_reply: tid=0 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 client.25271 shutdown 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 -- 172.21.15.28:0/2984224133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 msgr2=0x5565fe1351e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe1351e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f360c09c710 tx=0x7f360c09c420 comp rx=0 tx=0).stop 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 -- 172.21.15.28:0/2984224133 shutdown_connections 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5565fe145220 0x5565fe146690 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe135ab0 0x5565fe132a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe1351e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/2984224133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1324e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 -- 172.21.15.28:0/2984224133 >> 172.21.15.28:0/2984224133 conn(0x5565fe059550 msgr2=0x5565fe12cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f35fbe00700 0 client.25271 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 -- 172.21.15.28:0/2984224133 shutdown_connections 2024-04-26T01:41:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.389+0000 7f362b1a4b80 1 -- 172.21.15.28:0/2984224133 wait complete. 2024-04-26T01:41:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.390+0000 7f362b1a4b80 20 client.25271 trim_cache size 0 max 16384 2024-04-26T01:41:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.412+0000 7f362b1a4b80 1 Processor -- start 2024-04-26T01:41:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.412+0000 7f362b1a4b80 1 -- start start 2024-04-26T01:41:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12a2f0 0x5565fe126930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1329d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe135ab0 0x5565fe14faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565fe1449d0 con 0x5565fe12ac10 2024-04-26T01:41:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5565fe1450a0 con 0x5565fe135ab0 2024-04-26T01:41:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5565fe145770 con 0x5565fe12a2f0 2024-04-26T01:41:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f361e800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe135ab0 0x5565fe14faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f361d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f361de00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12a2f0 0x5565fe126930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f361e800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe135ab0 0x5565fe14faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39834/0 (socket says 172.21.15.28:39834) 2024-04-26T01:41:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f361e800700 1 -- 172.21.15.28:0/538680659 learned_addr learned my addr 172.21.15.28:0/538680659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f361e800700 1 -- 172.21.15.28:0/538680659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12a2f0 msgr2=0x5565fe126930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f361e800700 1 --2- 172.21.15.28:0/538680659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12a2f0 0x5565fe126930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.413+0000 7f361e800700 1 -- 172.21.15.28:0/538680659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 msgr2=0x5565fe1329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.414+0000 7f361e800700 1 --2- 172.21.15.28:0/538680659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1329d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.414+0000 7f361e800700 1 -- 172.21.15.28:0/538680659 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565fe130ab0 con 0x5565fe135ab0 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.414+0000 7f361e800700 1 --2- 172.21.15.28:0/538680659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe135ab0 0x5565fe14faf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3614009530 tx=0x7f361400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9b3d389f3761b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.414+0000 7f3606a00700 1 -- 172.21.15.28:0/538680659 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361401c070 con 0x5565fe135ab0 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.414+0000 7f3606a00700 1 -- 172.21.15.28:0/538680659 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36140178a0 con 0x5565fe135ab0 2024-04-26T01:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.414+0000 7f3606a00700 1 -- 172.21.15.28:0/538680659 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36140078f0 con 0x5565fe135ab0 2024-04-26T01:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 -- 172.21.15.28:0/538680659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe135ab0 msgr2=0x5565fe14faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/538680659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe135ab0 0x5565fe14faf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3614009530 tx=0x7f361400b040 comp rx=0 tx=0).stop 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 -- 172.21.15.28:0/538680659 shutdown_connections 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/538680659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe135ab0 0x5565fe14faf0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/538680659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12a2f0 0x5565fe126930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/538680659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe12ac10 0x5565fe1329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 -- 172.21.15.28:0/538680659 >> 172.21.15.28:0/538680659 conn(0x5565fe059550 msgr2=0x5565fe1322c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 -- 172.21.15.28:0/538680659 shutdown_connections 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.415+0000 7f362b1a4b80 1 -- 172.21.15.28:0/538680659 wait complete. 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.416+0000 7f362b1a4b80 1 -- start start 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.416+0000 7f362b1a4b80 1 Processor -- start 2024-04-26T01:41:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.416+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe14f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.416+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.416+0000 7f362b1a4b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe149d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.416+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565fe146510 con 0x5565fe135ab0 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5565fe145e40 con 0x5565fe12a2f0 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f362b1a4b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5565fe146be0 con 0x5565fe12ac10 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f362b1a4b80 10 client.0 _ll_register_callbacks cb 0x5565fe0166d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe149d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35034/0 (socket says 172.21.15.28:35034) 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361e800700 1 -- 172.21.15.28:0/698905678 learned_addr learned my addr 172.21.15.28:0/698905678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361de00700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe14f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361d400700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361e800700 1 -- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 msgr2=0x5565fe14feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361e800700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe14feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361e800700 1 -- 172.21.15.28:0/698905678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 msgr2=0x5565fe14f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361e800700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe14f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361e800700 1 -- 172.21.15.28:0/698905678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565fe14a580 con 0x5565fe135ab0 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361d400700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.417+0000 7f361de00700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe14f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.418+0000 7f361e800700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe149d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f361401a040 tx=0x7f3614017820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee95338950a03e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.418+0000 7f35fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.418+0000 7f35fbe00700 1 -- 172.21.15.28:0/698905678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361401c040 con 0x5565fe135ab0 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.418+0000 7f35fbe00700 1 -- 172.21.15.28:0/698905678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f3614017a50 con 0x5565fe135ab0 2024-04-26T01:41:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.418+0000 7f362b1a4b80 1 -- 172.21.15.28:0/698905678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36140178a0 con 0x5565fe135ab0 2024-04-26T01:41:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f35fbe00700 1 -- 172.21.15.28:0/698905678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f361402e050 con 0x5565fe135ab0 2024-04-26T01:41:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f362b1a4b80 10 client.15168 fetch_fsmap learned FSMap version 25 2024-04-26T01:41:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f362b1a4b80 1 -- 172.21.15.28:0/698905678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5565fe1acef0 con 0x5565fe135ab0 2024-04-26T01:41:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f35fbe00700 1 -- 172.21.15.28:0/698905678 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36140178a0 con 0x5565fe135ab0 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f35fbe00700 1 -- 172.21.15.28:0/698905678 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f361401f030 con 0x5565fe135ab0 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f362b1a4b80 10 client.15168 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f362b1a4b80 10 client.15168 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f362b1a4b80 20 client.15168 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5565fe1b1760 0x5565fe1b0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f362b1a4b80 4 client.15168 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.419+0000 7f362b1a4b80 1 -- 172.21.15.28:0/698905678 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "status"}) -- 0x5565fdefe200 con 0x5565fe1b1760 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.420+0000 7f361de00700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5565fe1b1760 0x5565fe1b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.420+0000 7f361de00700 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5565fe1b1760 0x5565fe1b0470 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0ca35af93ef15c3 server_cookie=95a8e19c2e46b1cc in_seq=0 out_seq=0 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.420+0000 7f35fbe00700 10 client.15168 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f35fbe00700 1 -- 172.21.15.28:0/698905678 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5565fdefe200 con 0x5565fe1b1760 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f35fbe00700 10 client.15168 handle_command_reply: tid=0 2024-04-26T01:41:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 client.15168 shutdown 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 -- 172.21.15.28:0/698905678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 msgr2=0x5565fe149d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe149d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f361401a040 tx=0x7f3614017820 comp rx=0 tx=0).stop 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 -- 172.21.15.28:0/698905678 shutdown_connections 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565fe135ab0 0x5565fe149d40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565fe12a2f0 0x5565fe14f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5565fe1b1760 0x5565fe1b0470 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 --2- 172.21.15.28:0/698905678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565fe12ac10 0x5565fe14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f35fbe00700 0 client.15168 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.421+0000 7f362b1a4b80 1 -- 172.21.15.28:0/698905678 >> 172.21.15.28:0/698905678 conn(0x5565fe059550 msgr2=0x5565fe1250c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.422+0000 7f362b1a4b80 1 -- 172.21.15.28:0/698905678 shutdown_connections 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.422+0000 7f362b1a4b80 1 -- 172.21.15.28:0/698905678 wait complete. 2024-04-26T01:41:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.422+0000 7f362b1a4b80 20 client.15168 trim_cache size 0 max 16384 2024-04-26T01:41:44.422 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:41:44.422 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "ead407d4-036b-11ef-bc93-c7b262605968", 2024-04-26T01:41:44.422 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-26T01:41:44.422 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24523, 2024-04-26T01:41:44.422 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-26T01:41:44.422 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-26T01:41:44.422 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-26T01:41:44.423 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 287.12733695999998, 2024-04-26T01:41:44.423 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 25, 2024-04-26T01:41:44.423 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-04-26T01:41:44.423 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:41:44.423 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 322.33762736400001 2024-04-26T01:41:44.423 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:41:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 -- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f36000a3f50 msgr2=0x7f36000a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f36000a3f50 0x7f36000a6410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f361806ea80 tx=0x7f360800b260 comp rx=0 tx=0).stop 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 -- 172.21.15.28:0/2412493192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3618074c70 msgr2=0x7f3618074230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3618074c70 0x7f3618074230 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f360c00eb30 tx=0x7f360c00ee40 comp rx=0 tx=0).stop 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 -- 172.21.15.28:0/2412493192 shutdown_connections 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f36000a3f50 0x7f36000a6410 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618113c10 0x7f361806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3618074c70 0x7f3618074230 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 --2- 172.21.15.28:0/2412493192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3618075590 0x7f361806d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 -- 172.21.15.28:0/2412493192 >> 172.21.15.28:0/2412493192 conn(0x7f3618079bf0 msgr2=0x7f361810f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 -- 172.21.15.28:0/2412493192 shutdown_connections 2024-04-26T01:41:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:41:44.423+0000 7f3620800700 1 -- 172.21.15.28:0/2412493192 wait complete. 2024-04-26T01:41:44.429 INFO:teuthology.orchestra.run.smithi028.stderr:700000 of 3200000 tuples (21%) done (elapsed 1.90 s, remaining 6.78 s) 2024-04-26T01:41:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:800000 of 3200000 tuples (25%) done (elapsed 2.28 s, remaining 6.83 s) 2024-04-26T01:41:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:900000 of 3200000 tuples (28%) done (elapsed 2.62 s, remaining 6.70 s) 2024-04-26T01:41:45.253 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-26T01:41:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:1000000 of 3200000 tuples (31%) done (elapsed 2.73 s, remaining 6.01 s) 2024-04-26T01:41:45.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:45 smithi028 ceph-mon[28247]: pgmap v26: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 12 MiB/s wr, 301 op/s 2024-04-26T01:41:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:1100000 of 3200000 tuples (34%) done (elapsed 3.26 s, remaining 6.22 s) 2024-04-26T01:41:45.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:45 smithi169 ceph-mon[32851]: pgmap v26: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 12 MiB/s wr, 301 op/s 2024-04-26T01:41:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:1200000 of 3200000 tuples (37%) done (elapsed 3.39 s, remaining 5.65 s) 2024-04-26T01:41:45.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:45 smithi053 ceph-mon[32593]: pgmap v26: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 12 MiB/s wr, 301 op/s 2024-04-26T01:41:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:1300000 of 3200000 tuples (40%) done (elapsed 3.77 s, remaining 5.51 s) 2024-04-26T01:41:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:1400000 of 3200000 tuples (43%) done (elapsed 4.27 s, remaining 5.49 s) 2024-04-26T01:41:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:1500000 of 3200000 tuples (46%) done (elapsed 4.40 s, remaining 4.98 s) 2024-04-26T01:41:47.301 INFO:teuthology.orchestra.run.smithi028.stderr:1600000 of 3200000 tuples (50%) done (elapsed 4.77 s, remaining 4.77 s) 2024-04-26T01:41:47.575 INFO:teuthology.orchestra.run.smithi028.stderr:1700000 of 3200000 tuples (53%) done (elapsed 5.04 s, remaining 4.45 s) 2024-04-26T01:41:47.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:47 smithi028 ceph-mon[28247]: pgmap v27: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 12 MiB/s wr, 296 op/s 2024-04-26T01:41:47.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:47 smithi169 ceph-mon[32851]: pgmap v27: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 12 MiB/s wr, 296 op/s 2024-04-26T01:41:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:1800000 of 3200000 tuples (56%) done (elapsed 5.40 s, remaining 4.20 s) 2024-04-26T01:41:47.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:47 smithi053 ceph-mon[32593]: pgmap v27: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 12 MiB/s wr, 296 op/s 2024-04-26T01:41:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:1900000 of 3200000 tuples (59%) done (elapsed 5.75 s, remaining 3.94 s) 2024-04-26T01:41:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2000000 of 3200000 tuples (62%) done (elapsed 5.92 s, remaining 3.55 s) 2024-04-26T01:41:48.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:48 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:2100000 of 3200000 tuples (65%) done (elapsed 6.27 s, remaining 3.28 s) 2024-04-26T01:41:48.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:48 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:48.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:48 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:49.133 INFO:teuthology.orchestra.run.smithi028.stderr:2200000 of 3200000 tuples (68%) done (elapsed 6.60 s, remaining 3.00 s) 2024-04-26T01:41:49.337 INFO:teuthology.orchestra.run.smithi028.stderr:2300000 of 3200000 tuples (71%) done (elapsed 6.80 s, remaining 2.66 s) 2024-04-26T01:41:49.740 INFO:teuthology.orchestra.run.smithi028.stderr:2400000 of 3200000 tuples (75%) done (elapsed 7.21 s, remaining 2.40 s) 2024-04-26T01:41:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:49 smithi028 ceph-mon[28247]: pgmap v28: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 12 MiB/s wr, 296 op/s 2024-04-26T01:41:49.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:49 smithi169 ceph-mon[32851]: pgmap v28: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 12 MiB/s wr, 296 op/s 2024-04-26T01:41:49.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:49 smithi053 ceph-mon[32593]: pgmap v28: 97 pgs: 97 active+clean; 93 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 12 MiB/s wr, 296 op/s 2024-04-26T01:41:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2500000 of 3200000 tuples (78%) done (elapsed 7.61 s, remaining 2.13 s) 2024-04-26T01:41:50.376 INFO:teuthology.orchestra.run.smithi028.stderr:2600000 of 3200000 tuples (81%) done (elapsed 7.84 s, remaining 1.81 s) 2024-04-26T01:41:50.774 INFO:teuthology.orchestra.run.smithi028.stderr:2700000 of 3200000 tuples (84%) done (elapsed 8.24 s, remaining 1.53 s) 2024-04-26T01:41:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2800000 of 3200000 tuples (87%) done (elapsed 8.61 s, remaining 1.23 s) 2024-04-26T01:41:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:2900000 of 3200000 tuples (90%) done (elapsed 8.80 s, remaining 0.91 s) 2024-04-26T01:41:51.684 INFO:teuthology.orchestra.run.smithi028.stderr:3000000 of 3200000 tuples (93%) done (elapsed 9.15 s, remaining 0.61 s) 2024-04-26T01:41:51.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:51 smithi028 ceph-mon[28247]: pgmap v29: 97 pgs: 97 active+clean; 253 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 40 MiB/s wr, 317 op/s 2024-04-26T01:41:51.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:51 smithi169 ceph-mon[32851]: pgmap v29: 97 pgs: 97 active+clean; 253 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 40 MiB/s wr, 317 op/s 2024-04-26T01:41:51.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:51 smithi053 ceph-mon[32593]: pgmap v29: 97 pgs: 97 active+clean; 253 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 40 MiB/s wr, 317 op/s 2024-04-26T01:41:52.019 INFO:teuthology.orchestra.run.smithi028.stderr:3100000 of 3200000 tuples (96%) done (elapsed 9.49 s, remaining 0.31 s) 2024-04-26T01:41:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:3200000 of 3200000 tuples (100%) done (elapsed 9.59 s, remaining 0.00 s) 2024-04-26T01:41:52.251 INFO:teuthology.orchestra.run.smithi028.stderr:vacuuming... 2024-04-26T01:41:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:creating primary keys... 2024-04-26T01:41:53.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:53 smithi028 ceph-mon[28247]: pgmap v30: 97 pgs: 97 active+clean; 253 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 38 MiB/s wr, 291 op/s 2024-04-26T01:41:53.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:53 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:53.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:53 smithi169 ceph-mon[32851]: pgmap v30: 97 pgs: 97 active+clean; 253 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 38 MiB/s wr, 291 op/s 2024-04-26T01:41:53.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:53 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.925+0000 7f794a000640 1 -- 172.21.15.28:0/3732667191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 msgr2=0x7f794410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.925+0000 7f794a000640 1 --2- 172.21.15.28:0/3732667191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 0x7f794410e9b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f793c009ca0 tx=0x7f793c02a950 comp rx=0 tx=0).stop 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.926+0000 7f794a000640 1 -- 172.21.15.28:0/3732667191 shutdown_connections 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.926+0000 7f794a000640 1 --2- 172.21.15.28:0/3732667191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 0x7f794410e9b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.926+0000 7f794a000640 1 --2- 172.21.15.28:0/3732667191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944103b40 0x7f794410c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.926+0000 7f794a000640 1 --2- 172.21.15.28:0/3732667191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7944103220 0x7f7944103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.926+0000 7f794a000640 1 -- 172.21.15.28:0/3732667191 >> 172.21.15.28:0/3732667191 conn(0x7f7944077fd0 msgr2=0x7f79440783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.926+0000 7f794a000640 1 -- 172.21.15.28:0/3732667191 shutdown_connections 2024-04-26T01:41:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.926+0000 7f794a000640 1 -- 172.21.15.28:0/3732667191 wait complete. 2024-04-26T01:41:53.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f794a000640 1 Processor -- start 2024-04-26T01:41:53.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f794a000640 1 -- start start 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f794a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944103220 0x7f79441a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f794a000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7944103b40 0x7f79441a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f794a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 0x7f794419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f794a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79441121d0 con 0x7f7944103220 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f794a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7944112050 con 0x7f794410c560 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f794a000640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7944112350 con 0x7f7944103b40 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f7943e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 0x7f794419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.927+0000 7f7943400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944103220 0x7f79441a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f7943e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 0x7f794419a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36482/0 (socket says 172.21.15.28:36482) 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f7943400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944103220 0x7f79441a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33348/0 (socket says 172.21.15.28:33348) 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f793be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7944103b40 0x7f79441a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f7943e00640 1 -- 172.21.15.28:0/2032231227 learned_addr learned my addr 172.21.15.28:0/2032231227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:53.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f793be00640 1 -- 172.21.15.28:0/2032231227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 msgr2=0x7f794419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f793be00640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 0x7f794419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f793be00640 1 -- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944103220 msgr2=0x7f79441a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f793be00640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944103220 0x7f79441a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f793be00640 1 -- 172.21.15.28:0/2032231227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794419af70 con 0x7f7944103b40 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f7943400640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944103220 0x7f79441a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.928+0000 7f793be00640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7944103b40 0x7f79441a0ae0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f792c00e9f0 tx=0x7f792c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.929+0000 7f7940c00640 1 -- 172.21.15.28:0/2032231227 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c00ccc0 con 0x7f7944103b40 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.929+0000 7f7940c00640 1 -- 172.21.15.28:0/2032231227 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f792c0108e0 con 0x7f7944103b40 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.929+0000 7f7940c00640 1 -- 172.21.15.28:0/2032231227 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c018880 con 0x7f7944103b40 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.929+0000 7f794a000640 1 -- 172.21.15.28:0/2032231227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794419b260 con 0x7f7944103b40 2024-04-26T01:41:53.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.929+0000 7f794a000640 1 -- 172.21.15.28:0/2032231227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79441a7520 con 0x7f7944103b40 2024-04-26T01:41:53.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.933+0000 7f794cd96740 1 Processor -- start 2024-04-26T01:41:53.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f794cd96740 1 -- start start 2024-04-26T01:41:53.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f7940c00640 1 -- 172.21.15.28:0/2032231227 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f792c010430 con 0x7f7944103b40 2024-04-26T01:41:53.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f794cd96740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b12820 0x560bb0b12c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f794cd96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b13140 0x560bb0b1d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f794cd96740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b1da00 0x560bb0b1fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f794cd96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560bb0a78900 con 0x560bb0b13140 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f794cd96740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560bb0a1eef0 con 0x560bb0b12820 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f794cd96740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x560bb0a748c0 con 0x560bb0b1da00 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.934+0000 7f7943400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b13140 0x560bb0b1d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f793be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b1da00 0x560bb0b1fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b13140 0x560bb0b1d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33358/0 (socket says 172.21.15.28:33358) 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943400640 1 -- 172.21.15.28:0/729716065 learned_addr learned my addr 172.21.15.28:0/729716065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943e00640 1 --2- 172.21.15.28:0/729716065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b12820 0x560bb0b12c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943400640 1 -- 172.21.15.28:0/729716065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b1da00 msgr2=0x560bb0b1fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943400640 1 --2- 172.21.15.28:0/729716065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b1da00 0x560bb0b1fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943400640 1 -- 172.21.15.28:0/729716065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b12820 msgr2=0x560bb0b12c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943400640 1 --2- 172.21.15.28:0/729716065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b12820 0x560bb0b12c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943400640 1 -- 172.21.15.28:0/729716065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bb078b210 con 0x560bb0b13140 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.935+0000 7f7943400640 1 --2- 172.21.15.28:0/729716065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b13140 0x560bb0b1d4c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f79340115f0 tx=0x7f7934011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff24b8108ad1bfc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.936+0000 7f7933e00640 1 -- 172.21.15.28:0/729716065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793401e070 con 0x560bb0b13140 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.936+0000 7f7933e00640 1 -- 172.21.15.28:0/729716065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f793400faf0 con 0x560bb0b13140 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.936+0000 7f7933e00640 1 -- 172.21.15.28:0/729716065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934019af0 con 0x560bb0b13140 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.936+0000 7f7940c00640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f79280a3f60 0x7f79280a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.936+0000 7f7940c00640 1 -- 172.21.15.28:0/2032231227 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f792c0d0c90 con 0x7f7944103b40 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.936+0000 7f7943e00640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f79280a3f60 0x7f79280a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 -- 172.21.15.28:0/729716065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b13140 msgr2=0x560bb0b1d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 --2- 172.21.15.28:0/729716065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b13140 0x560bb0b1d4c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f79340115f0 tx=0x7f7934011ab0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 -- 172.21.15.28:0/729716065 shutdown_connections 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 --2- 172.21.15.28:0/729716065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b1da00 0x560bb0b1fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 --2- 172.21.15.28:0/729716065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b13140 0x560bb0b1d4c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 --2- 172.21.15.28:0/729716065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b12820 0x560bb0b12c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 -- 172.21.15.28:0/729716065 >> 172.21.15.28:0/729716065 conn(0x560bb0b0ed30 msgr2=0x560bb0b0f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 -- 172.21.15.28:0/729716065 shutdown_connections 2024-04-26T01:41:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f794cd96740 1 -- 172.21.15.28:0/729716065 wait complete. 2024-04-26T01:41:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.937+0000 7f7943e00640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f79280a3f60 0x7f79280a6420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f793c0098d0 tx=0x7f793c009960 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.938+0000 7f794cd96740 1 -- start start 2024-04-26T01:41:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.938+0000 7f794cd96740 1 Processor -- start 2024-04-26T01:41:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.938+0000 7f794cd96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b1a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.938+0000 7f794cd96740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.938+0000 7f794cd96740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b1b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.938+0000 7f794cd96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560bb0b20630 con 0x560bb0b12820 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.938+0000 7f794cd96740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560bb0a32310 con 0x560bb0b1da00 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.938+0000 7f794cd96740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x560bb0b207b0 con 0x560bb0b13140 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f794cd96740 10 client.0 _ll_register_callbacks cb 0x560bb0a00cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b1a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b1a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33374/0 (socket says 172.21.15.28:33374) 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943400640 1 -- 172.21.15.28:0/1395030059 learned_addr learned my addr 172.21.15.28:0/1395030059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f793be00640 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943e00640 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b1b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943400640 1 -- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 msgr2=0x560bb0b1aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943400640 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943400640 1 -- 172.21.15.28:0/1395030059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 msgr2=0x560bb0b1b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943400640 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b1b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f7943400640 1 -- 172.21.15.28:0/1395030059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bb0857620 con 0x560bb0b12820 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.939+0000 7f793be00640 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.940+0000 7f7943400640 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b1a980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f793401c040 tx=0x7f7934009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d7aca841b382e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.940+0000 7f7930c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.940+0000 7f7930c00640 1 -- 172.21.15.28:0/1395030059 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793401e040 con 0x560bb0b12820 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.940+0000 7f7930c00640 1 -- 172.21.15.28:0/1395030059 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f793400f700 con 0x560bb0b12820 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.940+0000 7f794cd96740 1 -- 172.21.15.28:0/1395030059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f793400faf0 con 0x560bb0b12820 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.940+0000 7f7930c00640 1 -- 172.21.15.28:0/1395030059 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f793400f900 con 0x560bb0b12820 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.940+0000 7f794cd96740 10 client.15180 fetch_fsmap learned FSMap version 27 2024-04-26T01:41:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.940+0000 7f794cd96740 1 -- 172.21.15.28:0/1395030059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560bb0b2c870 con 0x560bb0b12820 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f7930c00640 1 -- 172.21.15.28:0/1395030059 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f793400faf0 con 0x560bb0b12820 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f7930c00640 1 -- 172.21.15.28:0/1395030059 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f79340035f0 con 0x560bb0b12820 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f794cd96740 10 client.15180 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f794cd96740 10 client.15180 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f794cd96740 20 client.15180 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f794cd96740 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560bb0b2dc70 0x560bb0b30060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f794cd96740 4 client.15180 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f794cd96740 1 -- 172.21.15.28:0/1395030059 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bb0a8c8b0 con 0x560bb0b2dc70 2024-04-26T01:41:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.941+0000 7f793be00640 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560bb0b2dc70 0x560bb0b30060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.942+0000 7f793be00640 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560bb0b2dc70 0x560bb0b30060 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=833a7f0e5def3632 server_cookie=4e342b614d48c580 in_seq=0 out_seq=0 2024-04-26T01:41:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.942+0000 7f7930c00640 10 client.15180 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.942+0000 7f7930c00640 1 -- 172.21.15.28:0/1395030059 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560bb0a8c8b0 con 0x560bb0b2dc70 2024-04-26T01:41:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.942+0000 7f7930c00640 10 client.15180 handle_command_reply: tid=0 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 client.15180 shutdown 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 -- 172.21.15.28:0/1395030059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 msgr2=0x560bb0b1a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b1a980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f793401c040 tx=0x7f7934009dc0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 -- 172.21.15.28:0/1395030059 shutdown_connections 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560bb0b2dc70 0x560bb0b30060 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b1b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 --2- 172.21.15.28:0/1395030059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b1a980 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 -- 172.21.15.28:0/1395030059 >> 172.21.15.28:0/1395030059 conn(0x560bb0b0ed30 msgr2=0x560bb0b14ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f7930c00640 0 client.15180 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 -- 172.21.15.28:0/1395030059 shutdown_connections 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 1 -- 172.21.15.28:0/1395030059 wait complete. 2024-04-26T01:41:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.943+0000 7f794cd96740 20 client.15180 trim_cache size 0 max 16384 2024-04-26T01:41:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.960+0000 7f794cd96740 1 Processor -- start 2024-04-26T01:41:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.960+0000 7f794cd96740 1 -- start start 2024-04-26T01:41:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.960+0000 7f794cd96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b294f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.960+0000 7f794cd96740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.960+0000 7f794cd96740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b35810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.960+0000 7f794cd96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79441b2560 con 0x560bb0b12820 2024-04-26T01:41:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.960+0000 7f794cd96740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f79441afe30 con 0x560bb0b1da00 2024-04-26T01:41:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.960+0000 7f794cd96740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f79441adc80 con 0x560bb0b13140 2024-04-26T01:41:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:43142/0 (socket says 172.21.15.28:43142) 2024-04-26T01:41:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 -- 172.21.15.28:0/3487463416 learned_addr learned my addr 172.21.15.28:0/3487463416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f7943e00640 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b35810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f7943400640 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b294f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 -- 172.21.15.28:0/3487463416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 msgr2=0x560bb0b35810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b35810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 -- 172.21.15.28:0/3487463416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 msgr2=0x560bb0b294f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b294f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 -- 172.21.15.28:0/3487463416 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bb0b18f30 con 0x560bb0b13140 2024-04-26T01:41:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.961+0000 7f793be00640 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1c8c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f792c09c7c0 tx=0x7f792c087040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d70cff7bd081baec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f7933400640 1 -- 172.21.15.28:0/3487463416 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c09c420 con 0x560bb0b13140 2024-04-26T01:41:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f7933400640 1 -- 172.21.15.28:0/3487463416 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f792c010720 con 0x560bb0b13140 2024-04-26T01:41:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f7933400640 1 -- 172.21.15.28:0/3487463416 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c010a90 con 0x560bb0b13140 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f794cd96740 1 -- 172.21.15.28:0/3487463416 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 msgr2=0x560bb0b1c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f794cd96740 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1c8c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f792c09c7c0 tx=0x7f792c087040 comp rx=0 tx=0).stop 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f794cd96740 1 -- 172.21.15.28:0/3487463416 shutdown_connections 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f794cd96740 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b35810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f794cd96740 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b1c8c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f794cd96740 1 --2- 172.21.15.28:0/3487463416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b294f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.962+0000 7f794cd96740 1 -- 172.21.15.28:0/3487463416 >> 172.21.15.28:0/3487463416 conn(0x560bb0b0ed30 msgr2=0x560bb0b1c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.963+0000 7f794cd96740 1 -- 172.21.15.28:0/3487463416 shutdown_connections 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.963+0000 7f794cd96740 1 -- 172.21.15.28:0/3487463416 wait complete. 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.963+0000 7f794cd96740 1 -- start start 2024-04-26T01:41:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.963+0000 7f794cd96740 1 Processor -- start 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f794cd96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b22c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f794cd96740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b231a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f794cd96740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b8c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f794cd96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79441ae2f0 con 0x560bb0b12820 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f794cd96740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f79441af760 con 0x560bb0b1da00 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f794cd96740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f79441ae9c0 con 0x560bb0b13140 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f794cd96740 10 client.0 _ll_register_callbacks cb 0x7f792c0d4670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f7943400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b22c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f7943400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b22c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33398/0 (socket says 172.21.15.28:33398) 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f7943400640 1 -- 172.21.15.28:0/3342427869 learned_addr learned my addr 172.21.15.28:0/3342427869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f793be00640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f7943e00640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b8c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.964+0000 7f7943400640 1 -- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 msgr2=0x560bb0b231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7943400640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7943400640 1 -- 172.21.15.28:0/3342427869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 msgr2=0x560bb0b8c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7943400640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b8c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7943400640 1 -- 172.21.15.28:0/3342427869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bb0b8c640 con 0x560bb0b12820 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f793be00640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7943e00640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b8c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7943400640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b22c60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7934011af0 tx=0x7f793400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4daea788d438dee2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7930c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7930c00640 1 -- 172.21.15.28:0/3342427869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793401e070 con 0x560bb0b12820 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f7930c00640 1 -- 172.21.15.28:0/3342427869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f793400faa0 con 0x560bb0b12820 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.965+0000 7f794cd96740 1 -- 172.21.15.28:0/3342427869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560bb0a86320 con 0x560bb0b12820 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f7930c00640 1 -- 172.21.15.28:0/3342427869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f793403d050 con 0x560bb0b12820 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f794cd96740 10 client.15186 fetch_fsmap learned FSMap version 27 2024-04-26T01:41:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f794cd96740 1 -- 172.21.15.28:0/3342427869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560bb0b939d0 con 0x560bb0b12820 2024-04-26T01:41:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f7930c00640 1 -- 172.21.15.28:0/3342427869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560bb0a86320 con 0x560bb0b12820 2024-04-26T01:41:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f7930c00640 1 -- 172.21.15.28:0/3342427869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7934035020 con 0x560bb0b12820 2024-04-26T01:41:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f794cd96740 10 client.15186 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T01:41:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f794cd96740 10 client.15186 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:41:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f794cd96740 20 client.15186 populate_metadata read hostname 'smithi028' 2024-04-26T01:41:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f794cd96740 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560bb0b94120 0x560bb0b96540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f794cd96740 4 client.15186 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:41:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.966+0000 7f794cd96740 1 -- 172.21.15.28:0/3342427869 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560bb0a80190 con 0x560bb0b94120 2024-04-26T01:41:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.967+0000 7f793be00640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560bb0b94120 0x560bb0b96540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.967+0000 7f7930c00640 10 client.15186 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:53.967+0000 7f793be00640 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560bb0b94120 0x560bb0b96540 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bd7d2d5fd48df42 server_cookie=a737c025dd73740d in_seq=0 out_seq=0 2024-04-26T01:41:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:53 smithi053 ceph-mon[32593]: pgmap v30: 97 pgs: 97 active+clean; 253 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 38 MiB/s wr, 291 op/s 2024-04-26T01:41:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:53 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.007+0000 7f7930c00640 1 -- 172.21.15.28:0/3342427869 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560bb0a80190 con 0x560bb0b94120 2024-04-26T01:41:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.007+0000 7f7930c00640 10 client.15186 handle_command_reply: tid=0 2024-04-26T01:41:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.007+0000 7f794cd96740 1 client.15186 shutdown 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 -- 172.21.15.28:0/3342427869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 msgr2=0x560bb0b22c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b22c60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7934011af0 tx=0x7f793400f3c0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 -- 172.21.15.28:0/3342427869 shutdown_connections 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560bb0b94120 0x560bb0b96540 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bb0b1da00 0x560bb0b8c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f7930c00640 0 client.15186 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560bb0b13140 0x560bb0b231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 --2- 172.21.15.28:0/3342427869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb0b12820 0x560bb0b22c60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 -- 172.21.15.28:0/3342427869 >> 172.21.15.28:0/3342427869 conn(0x560bb0b0ed30 msgr2=0x560bb0b1ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 -- 172.21.15.28:0/3342427869 shutdown_connections 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 1 -- 172.21.15.28:0/3342427869 wait complete. 2024-04-26T01:41:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.008+0000 7f794cd96740 20 client.15186 trim_cache size 0 max 16384 2024-04-26T01:41:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:41:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:41:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:41:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 -- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f79280a3f60 msgr2=0x7f79280a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f79280a3f60 0x7f79280a6420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f793c0098d0 tx=0x7f793c009960 comp rx=0 tx=0).stop 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 -- 172.21.15.28:0/2032231227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7944103b40 msgr2=0x7f79441a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7944103b40 0x7f79441a0ae0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f792c00e9f0 tx=0x7f792c00eeb0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 -- 172.21.15.28:0/2032231227 shutdown_connections 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f79280a3f60 0x7f79280a6420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794410c560 0x7f794419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7944103b40 0x7f79441a0ae0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 --2- 172.21.15.28:0/2032231227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7944103220 0x7f79441a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 -- 172.21.15.28:0/2032231227 >> 172.21.15.28:0/2032231227 conn(0x7f7944077fd0 msgr2=0x7f7944108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 -- 172.21.15.28:0/2032231227 shutdown_connections 2024-04-26T01:41:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:41:54.009+0000 7f794a000640 1 -- 172.21.15.28:0/2032231227 wait complete. 2024-04-26T01:41:55.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:55 smithi028 ceph-mon[28247]: pgmap v31: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 58 MiB/s wr, 311 op/s 2024-04-26T01:41:55.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:55 smithi169 ceph-mon[32851]: pgmap v31: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 58 MiB/s wr, 311 op/s 2024-04-26T01:41:55.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:55 smithi053 ceph-mon[32593]: pgmap v31: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 58 MiB/s wr, 311 op/s 2024-04-26T01:41:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:done in 13.56 s (drop tables 0.00 s, create tables 0.06 s, client-side generate 9.77 s, vacuum 1.10 s, primary keys 2.62 s). 2024-04-26T01:41:55.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-26T01:41:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:starting vacuum...end. 2024-04-26T01:41:57.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:57 smithi053 ceph-mon[32593]: pgmap v32: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 48 MiB/s wr, 40 op/s 2024-04-26T01:41:57.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:57 smithi028 ceph-mon[28247]: pgmap v32: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 48 MiB/s wr, 40 op/s 2024-04-26T01:41:57.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:57 smithi169 ceph-mon[32851]: pgmap v32: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 48 MiB/s wr, 40 op/s 2024-04-26T01:41:59.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:41:59 smithi028 ceph-mon[28247]: pgmap v33: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 48 MiB/s wr, 40 op/s 2024-04-26T01:41:59.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:41:59 smithi169 ceph-mon[32851]: pgmap v33: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 48 MiB/s wr, 40 op/s 2024-04-26T01:41:59.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:41:59 smithi053 ceph-mon[32593]: pgmap v33: 97 pgs: 97 active+clean; 382 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 48 MiB/s wr, 40 op/s 2024-04-26T01:42:00.254 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.6 2024-04-26T01:42:00.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:42:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 5.0 s, 2934.4 tps, lat 30.092 ms stddev 31.840 2024-04-26T01:42:01.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:01 smithi028 ceph-mon[28247]: pgmap v34: 97 pgs: 97 active+clean; 511 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 65 MiB/s wr, 130 op/s 2024-04-26T01:42:01.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:01 smithi169 ceph-mon[32851]: pgmap v34: 97 pgs: 97 active+clean; 511 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 65 MiB/s wr, 130 op/s 2024-04-26T01:42:01.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:01 smithi053 ceph-mon[32593]: pgmap v34: 97 pgs: 97 active+clean; 511 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 65 MiB/s wr, 130 op/s 2024-04-26T01:42:02.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:02 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:02.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:02 smithi169 ceph-mon[32851]: pgmap v35: 97 pgs: 97 active+clean; 511 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 37 MiB/s wr, 109 op/s 2024-04-26T01:42:02.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:02 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:02.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:02 smithi053 ceph-mon[32593]: pgmap v35: 97 pgs: 97 active+clean; 511 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 37 MiB/s wr, 109 op/s 2024-04-26T01:42:03.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:02 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:03.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:02 smithi028 ceph-mon[28247]: pgmap v35: 97 pgs: 97 active+clean; 511 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 37 MiB/s wr, 109 op/s 2024-04-26T01:42:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.930+0000 7fe58f200700 1 -- 172.21.15.28:0/319129825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe588107990 msgr2=0x7fe588107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.930+0000 7fe58f200700 1 --2- 172.21.15.28:0/319129825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe588107990 0x7fe588107d70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe5780098f0 tx=0x7fe578009c00 comp rx=0 tx=0).stop 2024-04-26T01:42:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.930+0000 7fe58f200700 1 -- 172.21.15.28:0/319129825 shutdown_connections 2024-04-26T01:42:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.930+0000 7fe58f200700 1 --2- 172.21.15.28:0/319129825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe588102300 0x7fe58810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.930+0000 7fe58f200700 1 --2- 172.21.15.28:0/319129825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe588107990 0x7fe588107d70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.930+0000 7fe58f200700 1 --2- 172.21.15.28:0/319129825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe588101940 0x7fe588101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.930+0000 7fe58f200700 1 -- 172.21.15.28:0/319129825 >> 172.21.15.28:0/319129825 conn(0x7fe5880788f0 msgr2=0x7fe588078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.930+0000 7fe58f200700 1 -- 172.21.15.28:0/319129825 shutdown_connections 2024-04-26T01:42:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.931+0000 7fe58f200700 1 -- 172.21.15.28:0/319129825 wait complete. 2024-04-26T01:42:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.932+0000 7fe58f200700 1 Processor -- start 2024-04-26T01:42:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.932+0000 7fe58f200700 1 -- start start 2024-04-26T01:42:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe58f200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe588101940 0x7fe5881a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe58f200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe588102300 0x7fe5881a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe58f200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe58819f850 0x7fe58819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe587400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe588102300 0x7fe5881a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe58f200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe588110ab0 con 0x7fe588101940 2024-04-26T01:42:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe58f200700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe588110930 con 0x7fe588102300 2024-04-26T01:42:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe58f200700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe588110c30 con 0x7fe58819f850 2024-04-26T01:42:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe587400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe588102300 0x7fe5881a5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37750/0 (socket says 172.21.15.28:37750) 2024-04-26T01:42:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe587400700 1 -- 172.21.15.28:0/3570610460 learned_addr learned my addr 172.21.15.28:0/3570610460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.933+0000 7fe58d200700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe58819f850 0x7fe58819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.934+0000 7fe587400700 1 -- 172.21.15.28:0/3570610460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe58819f850 msgr2=0x7fe58819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.934+0000 7fe587400700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe58819f850 0x7fe58819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.934+0000 7fe587400700 1 -- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe588101940 msgr2=0x7fe5881a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.934+0000 7fe587400700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe588101940 0x7fe5881a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.934+0000 7fe587400700 1 -- 172.21.15.28:0/3570610460 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5881a0410 con 0x7fe588102300 2024-04-26T01:42:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.935+0000 7fe587400700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe588102300 0x7fe5881a5c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe57400d950 tx=0x7fe57400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.935+0000 7fe584c00700 1 -- 172.21.15.28:0/3570610460 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574016070 con 0x7fe588102300 2024-04-26T01:42:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.935+0000 7fe584c00700 1 -- 172.21.15.28:0/3570610460 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe574009df0 con 0x7fe588102300 2024-04-26T01:42:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.935+0000 7fe584c00700 1 -- 172.21.15.28:0/3570610460 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57400f3c0 con 0x7fe588102300 2024-04-26T01:42:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.935+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5881ac530 con 0x7fe588102300 2024-04-26T01:42:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.937+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5881ac8d0 con 0x7fe588102300 2024-04-26T01:42:03.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.938+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe588103060 con 0x7fe588102300 2024-04-26T01:42:03.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.941+0000 7fe584c00700 1 -- 172.21.15.28:0/3570610460 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fe574019070 con 0x7fe588102300 2024-04-26T01:42:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.945+0000 7fe584c00700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe56c0a3fb0 0x7fe56c0a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:03.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.949+0000 7fe584c00700 1 -- 172.21.15.28:0/3570610460 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fe5740d00c0 con 0x7fe588102300 2024-04-26T01:42:03.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.950+0000 7fe587e00700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe56c0a3fb0 0x7fe56c0a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:03.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.950+0000 7fe584c00700 1 -- 172.21.15.28:0/3570610460 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe57409c4e0 con 0x7fe588102300 2024-04-26T01:42:03.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:03.952+0000 7fe587e00700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe56c0a3fb0 0x7fe56c0a6470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe578000c00 tx=0x7fe578009f70 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.462+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe58802df80 con 0x7fe588102300 2024-04-26T01:42:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.462+0000 7fe584c00700 1 -- 172.21.15.28:0/3570610460 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11865 (secure 0 0 0) 0x7fe5740966d0 con 0x7fe588102300 2024-04-26T01:42:04.460 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:42:04.460 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:42:00.871153+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:42:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.466+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe56c0a3fb0 msgr2=0x7fe56c0a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.466+0000 7fe58f200700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe56c0a3fb0 0x7fe56c0a6470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe578000c00 tx=0x7fe578009f70 comp rx=0 tx=0).stop 2024-04-26T01:42:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.466+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe588102300 msgr2=0x7fe5881a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.466+0000 7fe58f200700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe588102300 0x7fe5881a5c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe57400d950 tx=0x7fe57400dc60 comp rx=0 tx=0).stop 2024-04-26T01:42:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.467+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 shutdown_connections 2024-04-26T01:42:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.467+0000 7fe58f200700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe56c0a3fb0 0x7fe56c0a6470 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.467+0000 7fe58f200700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe58819f850 0x7fe58819fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.467+0000 7fe58f200700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe588101940 0x7fe5881a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.467+0000 7fe58f200700 1 --2- 172.21.15.28:0/3570610460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe588102300 0x7fe5881a5c10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.467+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 >> 172.21.15.28:0/3570610460 conn(0x7fe5880788f0 msgr2=0x7fe588100a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.467+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 shutdown_connections 2024-04-26T01:42:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:04.467+0000 7fe58f200700 1 -- 172.21.15.28:0/3570610460 wait complete. 2024-04-26T01:42:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-26T01:42:05.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:05 smithi169 ceph-mon[32851]: pgmap v36: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 74 MiB/s wr, 405 op/s 2024-04-26T01:42:05.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:05 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3570610460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:05.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:05 smithi053 ceph-mon[32593]: pgmap v36: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 74 MiB/s wr, 405 op/s 2024-04-26T01:42:05.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:05 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3570610460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:05 smithi028 ceph-mon[28247]: pgmap v36: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 74 MiB/s wr, 405 op/s 2024-04-26T01:42:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:05 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3570610460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:05.549 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 85.6 --await-for 42.0' 2024-04-26T01:42:05.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 85.6 --await-for 42.0 2024-04-26T01:42:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 10.0 s, 2247.5 tps, lat 47.906 ms stddev 124.633 2024-04-26T01:42:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:07.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.242+0000 7f5865a00640 1 -- 172.21.15.28:0/1677776678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f586010e910 msgr2=0x7f5860110d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.242+0000 7f5865a00640 1 --2- 172.21.15.28:0/1677776678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f586010e910 0x7f5860110d00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5858009ca0 tx=0x7f585802a8d0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.243+0000 7f5865a00640 1 -- 172.21.15.28:0/1677776678 shutdown_connections 2024-04-26T01:42:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.243+0000 7f5865a00640 1 --2- 172.21.15.28:0/1677776678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f586010e910 0x7f5860110d00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.243+0000 7f5865a00640 1 --2- 172.21.15.28:0/1677776678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5860101680 0x7f586010e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.243+0000 7f5865a00640 1 --2- 172.21.15.28:0/1677776678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860100d60 0x7f5860101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.243+0000 7f5865a00640 1 -- 172.21.15.28:0/1677776678 >> 172.21.15.28:0/1677776678 conn(0x7f5860077fd0 msgr2=0x7f58600783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.243+0000 7f5865a00640 1 -- 172.21.15.28:0/1677776678 shutdown_connections 2024-04-26T01:42:07.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.244+0000 7f5865a00640 1 -- 172.21.15.28:0/1677776678 wait complete. 2024-04-26T01:42:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.246+0000 7f5865a00640 1 Processor -- start 2024-04-26T01:42:07.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.247+0000 7f5865a00640 1 -- start start 2024-04-26T01:42:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.247+0000 7f5865a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5860100d60 0x7f58601a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.248+0000 7f5865a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860101680 0x7f58601a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.248+0000 7f5865a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f586010e910 0x7f586019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.248+0000 7f585ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5860100d60 0x7f58601a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.248+0000 7f585ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5860100d60 0x7f58601a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52178/0 (socket says 172.21.15.28:52178) 2024-04-26T01:42:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.249+0000 7f5865a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58601144a0 con 0x7f5860101680 2024-04-26T01:42:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.249+0000 7f5865a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5860114320 con 0x7f586010e910 2024-04-26T01:42:07.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.249+0000 7f5865a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5860114620 con 0x7f5860100d60 2024-04-26T01:42:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.250+0000 7f585f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f586010e910 0x7f586019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.250+0000 7f585f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f586010e910 0x7f586019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37774/0 (socket says 172.21.15.28:37774) 2024-04-26T01:42:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.250+0000 7f585f400640 1 -- 172.21.15.28:0/1732273065 learned_addr learned my addr 172.21.15.28:0/1732273065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:07.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.251+0000 7f585e000640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860101680 0x7f58601a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.251+0000 7f585ea00640 1 -- 172.21.15.28:0/1732273065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f586010e910 msgr2=0x7f586019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.251+0000 7f585ea00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f586010e910 0x7f586019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.251+0000 7f585ea00640 1 -- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860101680 msgr2=0x7f58601a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.251+0000 7f585ea00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860101680 0x7f58601a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.251+0000 7f585ea00640 1 -- 172.21.15.28:0/1732273065 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586019afd0 con 0x7f5860100d60 2024-04-26T01:42:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.252+0000 7f585ea00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5860100d60 0x7f58601a0690 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f585000e9f0 tx=0x7f585000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:07.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.252+0000 7f5857400640 1 -- 172.21.15.28:0/1732273065 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585000ccc0 con 0x7f5860100d60 2024-04-26T01:42:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.253+0000 7f5865a00640 1 -- 172.21.15.28:0/1732273065 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586019b2c0 con 0x7f5860100d60 2024-04-26T01:42:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.253+0000 7f5865a00640 1 -- 172.21.15.28:0/1732273065 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58601a74e0 con 0x7f5860100d60 2024-04-26T01:42:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.253+0000 7f5857400640 1 -- 172.21.15.28:0/1732273065 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58500108e0 con 0x7f5860100d60 2024-04-26T01:42:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.253+0000 7f5857400640 1 -- 172.21.15.28:0/1732273065 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850018880 con 0x7f5860100d60 2024-04-26T01:42:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.262+0000 7f5868784740 1 Processor -- start 2024-04-26T01:42:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.263+0000 7f5857400640 1 -- 172.21.15.28:0/1732273065 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f5850010430 con 0x7f5860100d60 2024-04-26T01:42:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.264+0000 7f5868784740 1 -- start start 2024-04-26T01:42:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.265+0000 7f5868784740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb84c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.266+0000 7f5868784740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 0x562b6cb8f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.266+0000 7f5868784740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 0x562b6cb91df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.266+0000 7f5868784740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b6caea900 con 0x562b6cb85140 2024-04-26T01:42:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.266+0000 7f5868784740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562b6ca90ef0 con 0x562b6cb8fa00 2024-04-26T01:42:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.266+0000 7f5868784740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562b6cae68c0 con 0x562b6cb84820 2024-04-26T01:42:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.266+0000 7f585f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb84c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.267+0000 7f585f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb84c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52186/0 (socket says 172.21.15.28:52186) 2024-04-26T01:42:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.267+0000 7f585f400640 1 -- 172.21.15.28:0/1117026007 learned_addr learned my addr 172.21.15.28:0/1117026007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:07.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.267+0000 7f585e000640 1 --2- 172.21.15.28:0/1117026007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 0x562b6cb91df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.268+0000 7f585f400640 1 -- 172.21.15.28:0/1117026007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 msgr2=0x562b6cb91df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.268+0000 7f585f400640 1 --2- 172.21.15.28:0/1117026007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 0x562b6cb91df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.268+0000 7f5857400640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f58440a3fe0 0x7f58440a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.268+0000 7f585f400640 1 -- 172.21.15.28:0/1117026007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 msgr2=0x562b6cb8f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.268+0000 7f585f400640 1 --2- 172.21.15.28:0/1117026007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 0x562b6cb8f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.268+0000 7f585f400640 1 -- 172.21.15.28:0/1117026007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b6c7fd210 con 0x562b6cb84820 2024-04-26T01:42:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.269+0000 7f5857400640 1 -- 172.21.15.28:0/1732273065 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f58500d0df0 con 0x7f5860100d60 2024-04-26T01:42:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.269+0000 7f585f400640 1 --2- 172.21.15.28:0/1117026007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb84c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5858009980 tx=0x7f58580099b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ab2327d87597fdf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.269+0000 7f584be00640 1 -- 172.21.15.28:0/1117026007 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858031d00 con 0x562b6cb84820 2024-04-26T01:42:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.269+0000 7f584be00640 1 -- 172.21.15.28:0/1117026007 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5858036030 con 0x562b6cb84820 2024-04-26T01:42:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.269+0000 7f584be00640 1 -- 172.21.15.28:0/1117026007 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58580042a0 con 0x562b6cb84820 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.271+0000 7f585f400640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f58440a3fe0 0x7f58440a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.271+0000 7f5868784740 1 -- 172.21.15.28:0/1117026007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 msgr2=0x562b6cb84c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.271+0000 7f5868784740 1 --2- 172.21.15.28:0/1117026007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb84c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5858009980 tx=0x7f58580099b0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.271+0000 7f5868784740 1 -- 172.21.15.28:0/1117026007 shutdown_connections 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.271+0000 7f5868784740 1 --2- 172.21.15.28:0/1117026007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 0x562b6cb91df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.271+0000 7f5868784740 1 --2- 172.21.15.28:0/1117026007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 0x562b6cb8f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.271+0000 7f5868784740 1 --2- 172.21.15.28:0/1117026007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb84c00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.271+0000 7f5868784740 1 -- 172.21.15.28:0/1117026007 >> 172.21.15.28:0/1117026007 conn(0x562b6cb80d30 msgr2=0x562b6cb81110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.272+0000 7f5868784740 1 -- 172.21.15.28:0/1117026007 shutdown_connections 2024-04-26T01:42:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.272+0000 7f5868784740 1 -- 172.21.15.28:0/1117026007 wait complete. 2024-04-26T01:42:07.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.273+0000 7f5868784740 1 -- start start 2024-04-26T01:42:07.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.273+0000 7f585f400640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f58440a3fe0 0x7f58440a64a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5858003cd0 tx=0x7f58580099b0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:07.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.273+0000 7f5868784740 1 Processor -- start 2024-04-26T01:42:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f5868784740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb8c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f5868784740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 0x562b6cb8cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f5868784740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 0x562b6cb8d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f5868784740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b6cb925d0 con 0x562b6cb85140 2024-04-26T01:42:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f5868784740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562b6caa4310 con 0x562b6cb8fa00 2024-04-26T01:42:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f5868784740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562b6cb92750 con 0x562b6cb84820 2024-04-26T01:42:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f585ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb8c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f5868784740 10 client.0 _ll_register_callbacks cb 0x562b6ca72cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.274+0000 7f585ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb8c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52202/0 (socket says 172.21.15.28:52202) 2024-04-26T01:42:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.275+0000 7f585ea00640 1 -- 172.21.15.28:0/1401556227 learned_addr learned my addr 172.21.15.28:0/1401556227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.275+0000 7f585e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 0x562b6cb8cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.275+0000 7f585f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 0x562b6cb8d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.276+0000 7f585ea00640 1 -- 172.21.15.28:0/1401556227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 msgr2=0x562b6cb8d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.276+0000 7f585ea00640 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 0x562b6cb8d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.276+0000 7f585ea00640 1 -- 172.21.15.28:0/1401556227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 msgr2=0x562b6cb8cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.276+0000 7f585ea00640 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 0x562b6cb8cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.276+0000 7f585ea00640 1 -- 172.21.15.28:0/1401556227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b6c8c9620 con 0x562b6cb84820 2024-04-26T01:42:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.277+0000 7f585e000640 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 0x562b6cb8cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:42:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.277+0000 7f585ea00640 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb8c9b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f585000c6a0 tx=0x7f585009c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bc8c10a42d08ab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.277+0000 7f5848c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:42:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.277+0000 7f5848c00640 1 -- 172.21.15.28:0/1401556227 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585009cb10 con 0x562b6cb84820 2024-04-26T01:42:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.277+0000 7f5848c00640 1 -- 172.21.15.28:0/1401556227 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f5850010a90 con 0x562b6cb84820 2024-04-26T01:42:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.277+0000 7f5868784740 1 -- 172.21.15.28:0/1401556227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5858036030 con 0x562b6cb84820 2024-04-26T01:42:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.277+0000 7f5848c00640 1 -- 172.21.15.28:0/1401556227 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f58500a0ec0 con 0x562b6cb84820 2024-04-26T01:42:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5868784740 10 client.25351 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5868784740 1 -- 172.21.15.28:0/1401556227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({fsmap=28}) -- 0x562b6cb99b00 con 0x562b6cb84820 2024-04-26T01:42:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5848c00640 1 -- 172.21.15.28:0/1401556227 <== mon.1 v2:172.21.15.169:3300/0 4 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f58500ce670 con 0x562b6cb84820 2024-04-26T01:42:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5868784740 10 client.25351 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5868784740 10 client.25351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:42:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5868784740 20 client.25351 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5868784740 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562b6cb9aeb0 0x562b6cb9d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5868784740 4 client.25351 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:42:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.278+0000 7f5868784740 1 -- 172.21.15.28:0/1401556227 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b6cafe8b0 con 0x562b6cb9aeb0 2024-04-26T01:42:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.279+0000 7f585e000640 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562b6cb9aeb0 0x562b6cb9d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.280+0000 7f5848c00640 10 client.25351 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.280+0000 7f585e000640 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562b6cb9aeb0 0x562b6cb9d2f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65dc4109a6f459f9 server_cookie=9f6208cbbcd95e00 in_seq=0 out_seq=0 2024-04-26T01:42:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.281+0000 7f5848c00640 1 -- 172.21.15.28:0/1401556227 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b6cafe8b0 con 0x562b6cb9aeb0 2024-04-26T01:42:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.281+0000 7f5848c00640 10 client.25351 handle_command_reply: tid=0 2024-04-26T01:42:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.281+0000 7f5868784740 1 client.25351 shutdown 2024-04-26T01:42:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.281+0000 7f5868784740 1 -- 172.21.15.28:0/1401556227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 msgr2=0x562b6cb8c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.281+0000 7f5868784740 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb8c9b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f585000c6a0 tx=0x7f585009c470 comp rx=0 tx=0).stop 2024-04-26T01:42:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 1 -- 172.21.15.28:0/1401556227 shutdown_connections 2024-04-26T01:42:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562b6cb9aeb0 0x562b6cb9d2f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb8fa00 0x562b6cb8d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5848c00640 0 client.25351 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb85140 0x562b6cb8cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 1 --2- 172.21.15.28:0/1401556227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb8c9b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 1 -- 172.21.15.28:0/1401556227 >> 172.21.15.28:0/1401556227 conn(0x562b6cb80d30 msgr2=0x562b6cb86a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:07.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 1 -- 172.21.15.28:0/1401556227 shutdown_connections 2024-04-26T01:42:07.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 1 -- 172.21.15.28:0/1401556227 wait complete. 2024-04-26T01:42:07.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.282+0000 7f5868784740 20 client.25351 trim_cache size 0 max 16384 2024-04-26T01:42:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:07 smithi028 ceph-mon[28247]: pgmap v37: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 54 MiB/s wr, 385 op/s 2024-04-26T01:42:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:07 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:42:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.315+0000 7f5868784740 1 Processor -- start 2024-04-26T01:42:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.315+0000 7f5868784740 1 -- start start 2024-04-26T01:42:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.316+0000 7f5868784740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb871a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.316+0000 7f5868784740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cbbcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.316+0000 7f5868784740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cbbd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.316+0000 7f5868784740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5840010f50 con 0x562b6cb8fa00 2024-04-26T01:42:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.316+0000 7f5868784740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f584000e7f0 con 0x562b6cb85140 2024-04-26T01:42:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.316+0000 7f5868784740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f584000c640 con 0x562b6cb84820 2024-04-26T01:42:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.316+0000 7f585ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb871a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.317+0000 7f585e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cbbcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.317+0000 7f585ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb871a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52210/0 (socket says 172.21.15.28:52210) 2024-04-26T01:42:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.317+0000 7f585f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cbbd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.317+0000 7f585ea00640 1 -- 172.21.15.28:0/4172900458 learned_addr learned my addr 172.21.15.28:0/4172900458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.317+0000 7f585e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cbbcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37810/0 (socket says 172.21.15.28:37810) 2024-04-26T01:42:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.319+0000 7f585e000640 1 -- 172.21.15.28:0/4172900458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 msgr2=0x562b6cb871a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.319+0000 7f585e000640 1 --2- 172.21.15.28:0/4172900458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb871a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.319+0000 7f585e000640 1 -- 172.21.15.28:0/4172900458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 msgr2=0x562b6cbbd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.319+0000 7f585e000640 1 --2- 172.21.15.28:0/4172900458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cbbd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.319+0000 7f585e000640 1 -- 172.21.15.28:0/4172900458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b6cb92160 con 0x562b6cb85140 2024-04-26T01:42:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.319+0000 7f585ea00640 1 --2- 172.21.15.28:0/4172900458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb871a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.320+0000 7f585e000640 1 --2- 172.21.15.28:0/4172900458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cbbcfb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x562b6cb8c210 tx=0x7f584c00a4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbb435961986a0cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.322+0000 7f584b400640 1 -- 172.21.15.28:0/4172900458 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c0169a0 con 0x562b6cb85140 2024-04-26T01:42:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.322+0000 7f584b400640 1 -- 172.21.15.28:0/4172900458 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f584c024430 con 0x562b6cb85140 2024-04-26T01:42:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.322+0000 7f584b400640 1 -- 172.21.15.28:0/4172900458 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c014630 con 0x562b6cb85140 2024-04-26T01:42:07.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.322+0000 7f5868784740 1 -- 172.21.15.28:0/4172900458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 msgr2=0x562b6cbbcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.322+0000 7f5868784740 1 --2- 172.21.15.28:0/4172900458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cbbcfb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x562b6cb8c210 tx=0x7f584c00a4e0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.323+0000 7f5868784740 1 -- 172.21.15.28:0/4172900458 shutdown_connections 2024-04-26T01:42:07.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.323+0000 7f5868784740 1 --2- 172.21.15.28:0/4172900458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cbbd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.323+0000 7f5868784740 1 --2- 172.21.15.28:0/4172900458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cbbcfb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.323+0000 7f5868784740 1 --2- 172.21.15.28:0/4172900458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cb871a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.323+0000 7f5868784740 1 -- 172.21.15.28:0/4172900458 >> 172.21.15.28:0/4172900458 conn(0x562b6cb80d30 msgr2=0x562b6cb87ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.323+0000 7f5868784740 1 -- 172.21.15.28:0/4172900458 shutdown_connections 2024-04-26T01:42:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.323+0000 7f5868784740 1 -- 172.21.15.28:0/4172900458 wait complete. 2024-04-26T01:42:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.324+0000 7f5868784740 1 -- start start 2024-04-26T01:42:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.325+0000 7f5868784740 1 Processor -- start 2024-04-26T01:42:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.326+0000 7f5868784740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cc0fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f5868784740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cc10110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f5868784740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cba2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f5868784740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f584000ccb0 con 0x562b6cb8fa00 2024-04-26T01:42:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f5868784740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f584000e120 con 0x562b6cb85140 2024-04-26T01:42:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f5868784740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f584000d380 con 0x562b6cb84820 2024-04-26T01:42:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f5868784740 10 client.0 _ll_register_callbacks cb 0x7f585009c9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f585ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cc0fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f585f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cba2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f585f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cba2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42894/0 (socket says 172.21.15.28:42894) 2024-04-26T01:42:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.327+0000 7f585f400640 1 -- 172.21.15.28:0/3743564738 learned_addr learned my addr 172.21.15.28:0/3743564738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.328+0000 7f585ea00640 1 -- 172.21.15.28:0/3743564738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 msgr2=0x562b6cc10110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:42:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.328+0000 7f585ea00640 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cc10110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.328+0000 7f585ea00640 1 -- 172.21.15.28:0/3743564738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 msgr2=0x562b6cba2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.328+0000 7f585ea00640 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cba2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.328+0000 7f585ea00640 1 -- 172.21.15.28:0/3743564738 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b6cba3070 con 0x562b6cb84820 2024-04-26T01:42:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.328+0000 7f585f400640 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cba2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:42:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.329+0000 7f585ea00640 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cc0fbd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f585009adc0 tx=0x7f585000cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d571cbbd6e2a48b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.329+0000 7f5848c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:42:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.329+0000 7f5848c00640 1 -- 172.21.15.28:0/3743564738 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500a5780 con 0x562b6cb84820 2024-04-26T01:42:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.330+0000 7f5848c00640 1 -- 172.21.15.28:0/3743564738 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f585009c170 con 0x562b6cb84820 2024-04-26T01:42:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.330+0000 7f5868784740 1 -- 172.21.15.28:0/3743564738 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f584c024430 con 0x562b6cb84820 2024-04-26T01:42:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.331+0000 7f5848c00640 1 -- 172.21.15.28:0/3743564738 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f585009cb10 con 0x562b6cb84820 2024-04-26T01:42:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.331+0000 7f5868784740 10 client.25357 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.331+0000 7f5868784740 1 -- 172.21.15.28:0/3743564738 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562b6cc28810 con 0x562b6cb84820 2024-04-26T01:42:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.331+0000 7f5848c00640 1 -- 172.21.15.28:0/3743564738 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f584c024430 con 0x562b6cb84820 2024-04-26T01:42:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.333+0000 7f5848c00640 1 -- 172.21.15.28:0/3743564738 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f58500a2050 con 0x562b6cb84820 2024-04-26T01:42:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.333+0000 7f5868784740 10 client.25357 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.333+0000 7f5868784740 10 client.25357 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:42:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.333+0000 7f5868784740 20 client.25357 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.333+0000 7f5868784740 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562b6cc28f50 0x562b6cc2b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.333+0000 7f5868784740 4 client.25357 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:42:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.333+0000 7f5868784740 1 -- 172.21.15.28:0/3743564738 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b6caf8320 con 0x562b6cc28f50 2024-04-26T01:42:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.333+0000 7f585e000640 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562b6cc28f50 0x562b6cc2b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.334+0000 7f585e000640 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562b6cc28f50 0x562b6cc2b370 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb7e0b49d239829d server_cookie=ccc79ddbd77e9823 in_seq=0 out_seq=0 2024-04-26T01:42:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.334+0000 7f5848c00640 10 client.25357 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.359+0000 7f5848c00640 1 -- 172.21.15.28:0/3743564738 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562b6caf8320 con 0x562b6cc28f50 2024-04-26T01:42:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.359+0000 7f5848c00640 10 client.25357 handle_command_reply: tid=0 2024-04-26T01:42:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.359+0000 7f5868784740 1 client.25357 shutdown 2024-04-26T01:42:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.360+0000 7f5868784740 1 -- 172.21.15.28:0/3743564738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 msgr2=0x562b6cc0fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.360+0000 7f5868784740 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cc0fbd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f585009adc0 tx=0x7f585000cef0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5868784740 1 -- 172.21.15.28:0/3743564738 shutdown_connections 2024-04-26T01:42:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5868784740 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562b6cc28f50 0x562b6cc2b370 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5868784740 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b6cb8fa00 0x562b6cba2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5868784740 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562b6cb85140 0x562b6cc10110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5868784740 1 --2- 172.21.15.28:0/3743564738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b6cb84820 0x562b6cc0fbd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5848c00640 0 client.25357 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5868784740 1 -- 172.21.15.28:0/3743564738 >> 172.21.15.28:0/3743564738 conn(0x562b6cb80d30 msgr2=0x562b6cb7f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5868784740 1 -- 172.21.15.28:0/3743564738 shutdown_connections 2024-04-26T01:42:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.361+0000 7f5868784740 1 -- 172.21.15.28:0/3743564738 wait complete. 2024-04-26T01:42:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.362+0000 7f5868784740 20 client.25357 trim_cache size 0 max 16384 2024-04-26T01:42:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:42:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:42:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:42:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:42:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 -- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f58440a3fe0 msgr2=0x7f58440a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f58440a3fe0 0x7f58440a64a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5858003cd0 tx=0x7f58580099b0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 -- 172.21.15.28:0/1732273065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5860100d60 msgr2=0x7f58601a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5860100d60 0x7f58601a0690 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f585000e9f0 tx=0x7f585000eeb0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 -- 172.21.15.28:0/1732273065 shutdown_connections 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f58440a3fe0 0x7f58440a64a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f586010e910 0x7f586019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860101680 0x7f58601a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 --2- 172.21.15.28:0/1732273065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5860100d60 0x7f58601a0690 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.363+0000 7f5865a00640 1 -- 172.21.15.28:0/1732273065 >> 172.21.15.28:0/1732273065 conn(0x7f5860077fd0 msgr2=0x7f586010fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.364+0000 7f5865a00640 1 -- 172.21.15.28:0/1732273065 shutdown_connections 2024-04-26T01:42:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:07.364+0000 7f5865a00640 1 -- 172.21.15.28:0/1732273065 wait complete. 2024-04-26T01:42:07.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:07 smithi169 ceph-mon[32851]: pgmap v37: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 54 MiB/s wr, 385 op/s 2024-04-26T01:42:07.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:07 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:42:07.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:07 smithi053 ceph-mon[32593]: pgmap v37: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 54 MiB/s wr, 385 op/s 2024-04-26T01:42:07.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:07 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:42:09.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:09 smithi028 ceph-mon[28247]: pgmap v38: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 54 MiB/s wr, 385 op/s 2024-04-26T01:42:09.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:09 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:09.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:09 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:09.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:09 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:42:09.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:09 smithi169 ceph-mon[32851]: pgmap v38: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 54 MiB/s wr, 385 op/s 2024-04-26T01:42:09.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:09 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:09.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:09 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:09.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:09 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:42:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:09 smithi053 ceph-mon[32593]: pgmap v38: 97 pgs: 97 active+clean; 935 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 54 MiB/s wr, 385 op/s 2024-04-26T01:42:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:09 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:09.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:09 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:09.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:09 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:42:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.498+0000 7f06eae00700 1 -- 172.21.15.28:0/1589406011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e4109fc0 msgr2=0x7f06e4111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.498+0000 7f06eae00700 1 --2- 172.21.15.28:0/1589406011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e4109fc0 0x7f06e4111b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00b0d0 tx=0x7f06dc00b3e0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.498+0000 7f06eae00700 1 -- 172.21.15.28:0/1589406011 shutdown_connections 2024-04-26T01:42:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.498+0000 7f06eae00700 1 --2- 172.21.15.28:0/1589406011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e4109fc0 0x7f06e4111b90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.498+0000 7f06eae00700 1 --2- 172.21.15.28:0/1589406011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e4104ee0 0x7f06e41052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.498+0000 7f06eae00700 1 --2- 172.21.15.28:0/1589406011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06e4105890 0x7f06e41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.498+0000 7f06eae00700 1 -- 172.21.15.28:0/1589406011 >> 172.21.15.28:0/1589406011 conn(0x7f06e4100770 msgr2=0x7f06e4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.499+0000 7f06eae00700 1 -- 172.21.15.28:0/1589406011 shutdown_connections 2024-04-26T01:42:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.499+0000 7f06eae00700 1 -- 172.21.15.28:0/1589406011 wait complete. 2024-04-26T01:42:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.500+0000 7f06eae00700 1 Processor -- start 2024-04-26T01:42:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.501+0000 7f06eae00700 1 -- start start 2024-04-26T01:42:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.501+0000 7f06eae00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e4104ee0 0x7f06e4111900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.501+0000 7f06eae00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e4105890 0x7f06e410e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.502+0000 7f06eae00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06e4109fc0 0x7f06e410eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.502+0000 7f06eae00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06e4114ed0 con 0x7f06e4104ee0 2024-04-26T01:42:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.502+0000 7f06eae00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06e4114d50 con 0x7f06e4109fc0 2024-04-26T01:42:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.502+0000 7f06eae00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f06e4115050 con 0x7f06e4105890 2024-04-26T01:42:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.502+0000 7f06e8e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06e4109fc0 0x7f06e410eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.502+0000 7f06e8e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06e4109fc0 0x7f06e410eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43474/0 (socket says 172.21.15.28:43474) 2024-04-26T01:42:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.502+0000 7f06e8e00700 1 -- 172.21.15.28:0/1613061462 learned_addr learned my addr 172.21.15.28:0/1613061462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.502+0000 7f06e3400700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e4105890 0x7f06e410e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.503+0000 7f06e8e00700 1 -- 172.21.15.28:0/1613061462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e4105890 msgr2=0x7f06e410e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.503+0000 7f06e8e00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e4105890 0x7f06e410e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.503+0000 7f06e8e00700 1 -- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e4104ee0 msgr2=0x7f06e4111900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.503+0000 7f06e8e00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e4104ee0 0x7f06e4111900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.503+0000 7f06e8e00700 1 -- 172.21.15.28:0/1613061462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e410f700 con 0x7f06e4109fc0 2024-04-26T01:42:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.504+0000 7f06e8e00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06e4109fc0 0x7f06e410eef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009fd0 tx=0x7f06dc007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:09.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.505+0000 7f06e0c00700 1 -- 172.21.15.28:0/1613061462 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc021070 con 0x7f06e4109fc0 2024-04-26T01:42:09.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.505+0000 7f06e0c00700 1 -- 172.21.15.28:0/1613061462 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06dc0096a0 con 0x7f06e4109fc0 2024-04-26T01:42:09.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.505+0000 7f06e0c00700 1 -- 172.21.15.28:0/1613061462 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc01d6b0 con 0x7f06e4109fc0 2024-04-26T01:42:09.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.505+0000 7f06eae00700 1 -- 172.21.15.28:0/1613061462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e41abc10 con 0x7f06e4109fc0 2024-04-26T01:42:09.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.506+0000 7f06eae00700 1 -- 172.21.15.28:0/1613061462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e41ac070 con 0x7f06e4109fc0 2024-04-26T01:42:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.510+0000 7f06e0c00700 1 -- 172.21.15.28:0/1613061462 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f06dc01d850 con 0x7f06e4109fc0 2024-04-26T01:42:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.513+0000 7f06e0c00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06c80a3ff0 0x7f06c80a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.513+0000 7f06e0c00700 1 -- 172.21.15.28:0/1613061462 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f06dc00f020 con 0x7f06e4109fc0 2024-04-26T01:42:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.514+0000 7f06e3e00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06c80a3ff0 0x7f06c80a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.518+0000 7f06f5725b80 1 Processor -- start 2024-04-26T01:42:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.517+0000 7f06e3e00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06c80a3ff0 0x7f06c80a64b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06d40097b0 tx=0x7f06d4006d20 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:09.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.519+0000 7f06f5725b80 1 -- start start 2024-04-26T01:42:09.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.520+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fd3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.520+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fdec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.520+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fe1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.520+0000 7f06f5725b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5f27020 con 0x55f7d5fd39b0 2024-04-26T01:42:09.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.520+0000 7f06f5725b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5ecd8c0 con 0x55f7d5fdf180 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.520+0000 7f06f5725b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5f22fe0 con 0x55f7d5fd42d0 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.520+0000 7f06e8e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fe1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.521+0000 7f06e8e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fe1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43480/0 (socket says 172.21.15.28:43480) 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.521+0000 7f06e8e00700 1 -- 172.21.15.28:0/743521012 learned_addr learned my addr 172.21.15.28:0/743521012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.521+0000 7f06e8e00700 1 -- 172.21.15.28:0/743521012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 msgr2=0x55f7d5fdec40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.521+0000 7f06e8e00700 1 --2- 172.21.15.28:0/743521012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fdec40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.521+0000 7f06e8e00700 1 -- 172.21.15.28:0/743521012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 msgr2=0x55f7d5fd3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.521+0000 7f06e8e00700 1 --2- 172.21.15.28:0/743521012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fd3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.521+0000 7f06e8e00700 1 -- 172.21.15.28:0/743521012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d5ec3810 con 0x55f7d5fdf180 2024-04-26T01:42:09.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.523+0000 7f06e8e00700 1 --2- 172.21.15.28:0/743521012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fe1570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f06dc09b0f0 tx=0x7f06dc007f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d0fa0bf25c737ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:09.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.523+0000 7f06d9600700 1 -- 172.21.15.28:0/743521012 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc09ec90 con 0x55f7d5fdf180 2024-04-26T01:42:09.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.523+0000 7f06d9600700 1 -- 172.21.15.28:0/743521012 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06dc0a3040 con 0x55f7d5fdf180 2024-04-26T01:42:09.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.523+0000 7f06d9600700 1 -- 172.21.15.28:0/743521012 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc09eef0 con 0x55f7d5fdf180 2024-04-26T01:42:09.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.524+0000 7f06f5725b80 1 -- 172.21.15.28:0/743521012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 msgr2=0x55f7d5fe1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.524+0000 7f06f5725b80 1 --2- 172.21.15.28:0/743521012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fe1570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f06dc09b0f0 tx=0x7f06dc007f10 comp rx=0 tx=0).stop 2024-04-26T01:42:09.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.526+0000 7f06f5725b80 1 -- 172.21.15.28:0/743521012 shutdown_connections 2024-04-26T01:42:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.526+0000 7f06f5725b80 1 --2- 172.21.15.28:0/743521012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fe1570 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.526+0000 7f06f5725b80 1 --2- 172.21.15.28:0/743521012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fd3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.526+0000 7f06f5725b80 1 --2- 172.21.15.28:0/743521012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fdec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.526+0000 7f06f5725b80 1 -- 172.21.15.28:0/743521012 >> 172.21.15.28:0/743521012 conn(0x55f7d5f02730 msgr2=0x55f7d5fcfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.526+0000 7f06f5725b80 1 -- 172.21.15.28:0/743521012 shutdown_connections 2024-04-26T01:42:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.526+0000 7f06f5725b80 1 -- 172.21.15.28:0/743521012 wait complete. 2024-04-26T01:42:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.527+0000 7f06f5725b80 1 -- start start 2024-04-26T01:42:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.527+0000 7f06f5725b80 1 Processor -- start 2024-04-26T01:42:09.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.533+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fde9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.535+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fdbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.535+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fdc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.535+0000 7f06f5725b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5fe1ab0 con 0x55f7d5fd39b0 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.535+0000 7f06f5725b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5ee09f0 con 0x55f7d5fdf180 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.535+0000 7f06f5725b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5fe1c30 con 0x55f7d5fd42d0 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.536+0000 7f06f5725b80 10 client.0 _ll_register_callbacks cb 0x55f7d5ebf8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.536+0000 7f06e3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fde9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.536+0000 7f06e3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fde9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41114/0 (socket says 172.21.15.28:41114) 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.536+0000 7f06e3400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fdbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.537+0000 7f06e3e00700 1 -- 172.21.15.28:0/1115357109 learned_addr learned my addr 172.21.15.28:0/1115357109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.537+0000 7f06e8e00700 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fdc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.538+0000 7f06e8e00700 1 -- 172.21.15.28:0/1115357109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 msgr2=0x55f7d5fdbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:09.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.538+0000 7f06e8e00700 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fdbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.538+0000 7f06e8e00700 1 -- 172.21.15.28:0/1115357109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 msgr2=0x55f7d5fde9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:09.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.538+0000 7f06e8e00700 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fde9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.538+0000 7f06e8e00700 1 -- 172.21.15.28:0/1115357109 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d5e74a60 con 0x55f7d5fdf180 2024-04-26T01:42:09.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.538+0000 7f06e3400700 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fdbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:09.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.539+0000 7f06e8e00700 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fdc220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55f7d5fd47d0 tx=0x7f06dc09d1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82ce8635fe62bf54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:09.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.539+0000 7f06ce000700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:42:09.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.539+0000 7f06ce000700 1 -- 172.21.15.28:0/1115357109 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc0a7870 con 0x55f7d5fdf180 2024-04-26T01:42:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.540+0000 7f06ce000700 1 -- 172.21.15.28:0/1115357109 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f06dc09eef0 con 0x55f7d5fdf180 2024-04-26T01:42:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.540+0000 7f06f5725b80 1 -- 172.21.15.28:0/1115357109 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06dc0a3040 con 0x55f7d5fdf180 2024-04-26T01:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.541+0000 7f06ce000700 1 -- 172.21.15.28:0/1115357109 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f06dc09f0f0 con 0x55f7d5fdf180 2024-04-26T01:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.541+0000 7f06f5725b80 10 client.25325 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.541+0000 7f06f5725b80 1 -- 172.21.15.28:0/1115357109 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f7d5ebdc80 con 0x55f7d5fdf180 2024-04-26T01:42:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.542+0000 7f06ce000700 1 -- 172.21.15.28:0/1115357109 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06dc0a3040 con 0x55f7d5fdf180 2024-04-26T01:42:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.542+0000 7f06ce000700 1 -- 172.21.15.28:0/1115357109 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f06dc011070 con 0x55f7d5fdf180 2024-04-26T01:42:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.542+0000 7f06f5725b80 10 client.25325 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.542+0000 7f06f5725b80 10 client.25325 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:42:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.542+0000 7f06f5725b80 20 client.25325 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.542+0000 7f06f5725b80 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f7d5fee6e0 0x55f7d5ff0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.542+0000 7f06f5725b80 4 client.25325 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:42:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.542+0000 7f06f5725b80 1 -- 172.21.15.28:0/1115357109 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7d5f34810 con 0x55f7d5fee6e0 2024-04-26T01:42:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.544+0000 7f06e3e00700 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f7d5fee6e0 0x55f7d5ff0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.545+0000 7f06e3e00700 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f7d5fee6e0 0x55f7d5ff0ad0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31f7dae9c6abbc32 server_cookie=c9da9007b0e4d835 in_seq=0 out_seq=0 2024-04-26T01:42:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.545+0000 7f06ce000700 10 client.25325 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.546+0000 7f06ce000700 1 -- 172.21.15.28:0/1115357109 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7d5f34810 con 0x55f7d5fee6e0 2024-04-26T01:42:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.546+0000 7f06ce000700 10 client.25325 handle_command_reply: tid=0 2024-04-26T01:42:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.546+0000 7f06f5725b80 1 client.25325 shutdown 2024-04-26T01:42:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.547+0000 7f06f5725b80 1 -- 172.21.15.28:0/1115357109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 msgr2=0x55f7d5fdc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.547+0000 7f06f5725b80 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fdc220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55f7d5fd47d0 tx=0x7f06dc09d1a0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.548+0000 7f06f5725b80 1 -- 172.21.15.28:0/1115357109 shutdown_connections 2024-04-26T01:42:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.548+0000 7f06f5725b80 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fdc220 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.548+0000 7f06f5725b80 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fde9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.548+0000 7f06f5725b80 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f7d5fee6e0 0x55f7d5ff0ad0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.548+0000 7f06f5725b80 1 --2- 172.21.15.28:0/1115357109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fdbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.548+0000 7f06f5725b80 1 -- 172.21.15.28:0/1115357109 >> 172.21.15.28:0/1115357109 conn(0x55f7d5f02730 msgr2=0x55f7d5fcfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.548+0000 7f06ce000700 0 client.25325 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.549+0000 7f06f5725b80 1 -- 172.21.15.28:0/1115357109 shutdown_connections 2024-04-26T01:42:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.549+0000 7f06f5725b80 1 -- 172.21.15.28:0/1115357109 wait complete. 2024-04-26T01:42:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.550+0000 7f06f5725b80 20 client.25325 trim_cache size 0 max 16384 2024-04-26T01:42:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.577+0000 7f06f5725b80 1 Processor -- start 2024-04-26T01:42:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06f5725b80 1 -- start start 2024-04-26T01:42:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fd7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fcffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fd8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06f5725b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06e41b6140 con 0x55f7d5fd42d0 2024-04-26T01:42:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06f5725b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06e41b39a0 con 0x55f7d5fdf180 2024-04-26T01:42:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06f5725b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f06e41b17f0 con 0x55f7d5fd39b0 2024-04-26T01:42:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06e3400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fcffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.578+0000 7f06e3400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fcffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41116/0 (socket says 172.21.15.28:41116) 2024-04-26T01:42:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.579+0000 7f06e3400700 1 -- 172.21.15.28:0/852137091 learned_addr learned my addr 172.21.15.28:0/852137091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.579+0000 7f06e8e00700 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fd8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.579+0000 7f06e3e00700 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fd7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.579+0000 7f06e3e00700 1 -- 172.21.15.28:0/852137091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 msgr2=0x55f7d5fd8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.579+0000 7f06e3e00700 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fd8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.579+0000 7f06e3e00700 1 -- 172.21.15.28:0/852137091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 msgr2=0x55f7d5fcffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.579+0000 7f06e3e00700 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fcffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.579+0000 7f06e3e00700 1 -- 172.21.15.28:0/852137091 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d5fda2b0 con 0x55f7d5fd39b0 2024-04-26T01:42:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.580+0000 7f06e3e00700 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fd7230 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55f7d5ebde20 tx=0x7f06d4006f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2c0688d21222ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.580+0000 7f06d8c00700 1 -- 172.21.15.28:0/852137091 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d40230d0 con 0x55f7d5fd39b0 2024-04-26T01:42:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.580+0000 7f06d8c00700 1 -- 172.21.15.28:0/852137091 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d4023720 con 0x55f7d5fd39b0 2024-04-26T01:42:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.580+0000 7f06d8c00700 1 -- 172.21.15.28:0/852137091 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4021cf0 con 0x55f7d5fd39b0 2024-04-26T01:42:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.581+0000 7f06f5725b80 1 -- 172.21.15.28:0/852137091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 msgr2=0x55f7d5fd7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.581+0000 7f06f5725b80 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fd7230 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55f7d5ebde20 tx=0x7f06d4006f90 comp rx=0 tx=0).stop 2024-04-26T01:42:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.582+0000 7f06f5725b80 1 -- 172.21.15.28:0/852137091 shutdown_connections 2024-04-26T01:42:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.582+0000 7f06f5725b80 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5fd8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.582+0000 7f06f5725b80 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5fd7230 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.583+0000 7f06f5725b80 1 --2- 172.21.15.28:0/852137091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5fcffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.583+0000 7f06f5725b80 1 -- 172.21.15.28:0/852137091 >> 172.21.15.28:0/852137091 conn(0x55f7d5f02730 msgr2=0x55f7d5fddda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.583+0000 7f06f5725b80 1 -- 172.21.15.28:0/852137091 shutdown_connections 2024-04-26T01:42:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.583+0000 7f06f5725b80 1 -- 172.21.15.28:0/852137091 wait complete. 2024-04-26T01:42:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.584+0000 7f06f5725b80 1 -- start start 2024-04-26T01:42:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.585+0000 7f06f5725b80 1 Processor -- start 2024-04-26T01:42:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.586+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5ff38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.586+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5ff3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.586+0000 7f06f5725b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5ff1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.586+0000 7f06f5725b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06e41b1e60 con 0x55f7d5fd42d0 2024-04-26T01:42:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.586+0000 7f06f5725b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06e41b32d0 con 0x55f7d5fdf180 2024-04-26T01:42:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.586+0000 7f06f5725b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f06e41b2530 con 0x55f7d5fd39b0 2024-04-26T01:42:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.587+0000 7f06f5725b80 10 client.0 _ll_register_callbacks cb 0x55f7d5ebf8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.586+0000 7f06e3e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5ff38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.587+0000 7f06e3e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5ff38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:42956/0 (socket says 172.21.15.28:42956) 2024-04-26T01:42:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.587+0000 7f06e3e00700 1 -- 172.21.15.28:0/4161345452 learned_addr learned my addr 172.21.15.28:0/4161345452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.587+0000 7f06e8e00700 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5ff1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.587+0000 7f06e3400700 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5ff3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.588+0000 7f06e8e00700 1 -- 172.21.15.28:0/4161345452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 msgr2=0x55f7d5ff38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.588+0000 7f06e8e00700 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5ff38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.588+0000 7f06e8e00700 1 -- 172.21.15.28:0/4161345452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 msgr2=0x55f7d5ff3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.588+0000 7f06e8e00700 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5ff3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.588+0000 7f06e8e00700 1 -- 172.21.15.28:0/4161345452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d5ff2530 con 0x55f7d5fdf180 2024-04-26T01:42:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.588+0000 7f06e3400700 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5ff3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.589+0000 7f06e8e00700 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5ff1cf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f06dc09b0f0 tx=0x7f06dc0099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ba3028a676bf989 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.589+0000 7f06ce000700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:42:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.590+0000 7f06ce000700 1 -- 172.21.15.28:0/4161345452 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc0a7870 con 0x55f7d5fdf180 2024-04-26T01:42:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.590+0000 7f06ce000700 1 -- 172.21.15.28:0/4161345452 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f06dc089020 con 0x55f7d5fdf180 2024-04-26T01:42:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.590+0000 7f06f5725b80 1 -- 172.21.15.28:0/4161345452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d4023720 con 0x55f7d5fdf180 2024-04-26T01:42:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.590+0000 7f06ce000700 1 -- 172.21.15.28:0/4161345452 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f06dc0a3040 con 0x55f7d5fdf180 2024-04-26T01:42:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.590+0000 7f06f5725b80 10 client.25331 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06f5725b80 1 -- 172.21.15.28:0/4161345452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f7d5ff2530 con 0x55f7d5fdf180 2024-04-26T01:42:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06ce000700 1 -- 172.21.15.28:0/4161345452 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d4023720 con 0x55f7d5fdf180 2024-04-26T01:42:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06ce000700 1 -- 172.21.15.28:0/4161345452 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f06dc011070 con 0x55f7d5fdf180 2024-04-26T01:42:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06f5725b80 10 client.25331 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06f5725b80 10 client.25331 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:42:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06f5725b80 20 client.25331 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06f5725b80 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f7d604f720 0x55f7d604fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06f5725b80 4 client.25331 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.6, "await_for": 42.0}] 2024-04-26T01:42:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.591+0000 7f06f5725b80 1 -- 172.21.15.28:0/4161345452 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.6, "await_for": 42.0}) -- 0x55f7d5f2e8b0 con 0x55f7d604f720 2024-04-26T01:42:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.592+0000 7f06e3e00700 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f7d604f720 0x55f7d604fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.593+0000 7f06e3e00700 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f7d604f720 0x55f7d604fb30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=456d0d4adf5444d8 server_cookie=748193ad7c083867 in_seq=0 out_seq=0 2024-04-26T01:42:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:09.593+0000 7f06ce000700 10 client.25331 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.252+0000 7f06ce000700 1 -- 172.21.15.28:0/4161345452 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+587 (crc 0 0 0) 0x55f7d5f2e8b0 con 0x55f7d604f720 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.252+0000 7f06ce000700 10 client.25331 handle_command_reply: tid=0 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.252+0000 7f06f5725b80 1 client.25331 shutdown 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 -- 172.21.15.28:0/4161345452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 msgr2=0x55f7d5ff1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5ff1cf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f06dc09b0f0 tx=0x7f06dc0099e0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 -- 172.21.15.28:0/4161345452 shutdown_connections 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f7d5fdf180 0x55f7d5ff1cf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f7d604f720 0x55f7d604fb30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f7d5fd39b0 0x55f7d5ff38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 --2- 172.21.15.28:0/4161345452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5fd42d0 0x55f7d5ff3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 -- 172.21.15.28:0/4161345452 >> 172.21.15.28:0/4161345452 conn(0x55f7d5f02730 msgr2=0x55f7d5fd58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06ce000700 0 client.25331 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:10.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 -- 172.21.15.28:0/4161345452 shutdown_connections 2024-04-26T01:42:10.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.253+0000 7f06f5725b80 1 -- 172.21.15.28:0/4161345452 wait complete. 2024-04-26T01:42:10.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.254+0000 7f06f5725b80 20 client.25331 trim_cache size 0 max 16384 2024-04-26T01:42:10.252 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:42:10.252 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 28, 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 2, 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "b141a1f": { 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 2, 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 60.0, 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 85.6, 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-26T01:42:10.253 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:10.254 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 -- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06c80a3ff0 msgr2=0x7f06c80a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06c80a3ff0 0x7f06c80a64b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06d40097b0 tx=0x7f06d4006d20 comp rx=0 tx=0).stop 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 -- 172.21.15.28:0/1613061462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06e4109fc0 msgr2=0x7f06e410eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06e4109fc0 0x7f06e410eef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009fd0 tx=0x7f06dc007a20 comp rx=0 tx=0).stop 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 -- 172.21.15.28:0/1613061462 shutdown_connections 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06c80a3ff0 0x7f06c80a64b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06e4109fc0 0x7f06e410eef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e4104ee0 0x7f06e4111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 --2- 172.21.15.28:0/1613061462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e4105890 0x7f06e410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 -- 172.21.15.28:0/1613061462 >> 172.21.15.28:0/1613061462 conn(0x7f06e4100770 msgr2=0x7f06e41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.255+0000 7f06eae00700 1 -- 172.21.15.28:0/1613061462 shutdown_connections 2024-04-26T01:42:10.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:10.256+0000 7f06eae00700 1 -- 172.21.15.28:0/1613061462 wait complete. 2024-04-26T01:42:11.070 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b141a1f', quiesce took 10.8 seconds. Will release after: 25.6 2024-04-26T01:42:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 15.0 s, 1427.1 tps, lat 47.920 ms stddev 59.938 2024-04-26T01:42:11.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:11 smithi028 ceph-mon[28247]: pgmap v39: 97 pgs: 97 active+clean; 977 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 58 MiB/s wr, 670 op/s 2024-04-26T01:42:11.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:11 smithi169 ceph-mon[32851]: pgmap v39: 97 pgs: 97 active+clean; 977 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 58 MiB/s wr, 670 op/s 2024-04-26T01:42:11.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:11 smithi053 ceph-mon[32593]: pgmap v39: 97 pgs: 97 active+clean; 977 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 58 MiB/s wr, 670 op/s 2024-04-26T01:42:13.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:13 smithi169 ceph-mon[32851]: pgmap v40: 97 pgs: 97 active+clean; 977 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 MiB/s wr, 580 op/s 2024-04-26T01:42:13.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:13 smithi053 ceph-mon[32593]: pgmap v40: 97 pgs: 97 active+clean; 977 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 MiB/s wr, 580 op/s 2024-04-26T01:42:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:13 smithi028 ceph-mon[28247]: pgmap v40: 97 pgs: 97 active+clean; 977 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 MiB/s wr, 580 op/s 2024-04-26T01:42:15.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:15 smithi169 ceph-mon[32851]: pgmap v41: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 41 MiB/s wr, 709 op/s 2024-04-26T01:42:15.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:15 smithi053 ceph-mon[32593]: pgmap v41: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 41 MiB/s wr, 709 op/s 2024-04-26T01:42:15.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:15 smithi028 ceph-mon[28247]: pgmap v41: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 41 MiB/s wr, 709 op/s 2024-04-26T01:42:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 20.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:42:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:17 smithi169 ceph-mon[32851]: pgmap v42: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 4.8 MiB/s wr, 412 op/s 2024-04-26T01:42:17.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:17 smithi053 ceph-mon[32593]: pgmap v42: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 4.8 MiB/s wr, 412 op/s 2024-04-26T01:42:17.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:17 smithi028 ceph-mon[28247]: pgmap v42: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 4.8 MiB/s wr, 412 op/s 2024-04-26T01:42:19.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:19 smithi169 ceph-mon[32851]: pgmap v43: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 4.8 MiB/s wr, 412 op/s 2024-04-26T01:42:19.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:19 smithi053 ceph-mon[32593]: pgmap v43: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 4.8 MiB/s wr, 412 op/s 2024-04-26T01:42:19.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:19 smithi028 ceph-mon[28247]: pgmap v43: 97 pgs: 97 active+clean; 981 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 4.8 MiB/s wr, 412 op/s 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 -- 172.21.15.28:0/1604832021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe97c102990 msgr2=0x7fe97c1069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 --2- 172.21.15.28:0/1604832021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe97c102990 0x7fe97c1069d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe964009860 tx=0x7fe96402a340 comp rx=0 tx=0).stop 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 -- 172.21.15.28:0/1604832021 shutdown_connections 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 --2- 172.21.15.28:0/1604832021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c1070b0 0x7fe97c10ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 --2- 172.21.15.28:0/1604832021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe97c102990 0x7fe97c1069d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 --2- 172.21.15.28:0/1604832021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c101fe0 0x7fe97c1023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 -- 172.21.15.28:0/1604832021 >> 172.21.15.28:0/1604832021 conn(0x7fe97c0fd8a0 msgr2=0x7fe97c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 -- 172.21.15.28:0/1604832021 shutdown_connections 2024-04-26T01:42:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.522+0000 7fe980a00640 1 -- 172.21.15.28:0/1604832021 wait complete. 2024-04-26T01:42:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.523+0000 7fe980a00640 1 Processor -- start 2024-04-26T01:42:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.524+0000 7fe980a00640 1 -- start start 2024-04-26T01:42:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.524+0000 7fe980a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c101fe0 0x7fe97c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.524+0000 7fe980a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c1070b0 0x7fe97c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.524+0000 7fe980a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe97c074cd0 0x7fe97c1a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.524+0000 7fe980a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c111fd0 con 0x7fe97c1070b0 2024-04-26T01:42:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.524+0000 7fe980a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c111e50 con 0x7fe97c101fe0 2024-04-26T01:42:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.524+0000 7fe980a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c112150 con 0x7fe97c074cd0 2024-04-26T01:42:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c101fe0 0x7fe97c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c1070b0 0x7fe97c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c101fe0 0x7fe97c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:40718/0 (socket says 172.21.15.28:40718) 2024-04-26T01:42:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97b400640 1 -- 172.21.15.28:0/3737820089 learned_addr learned my addr 172.21.15.28:0/3737820089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97be00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe97c074cd0 0x7fe97c1a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97aa00640 1 -- 172.21.15.28:0/3737820089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe97c074cd0 msgr2=0x7fe97c1a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97aa00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe97c074cd0 0x7fe97c1a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97aa00640 1 -- 172.21.15.28:0/3737820089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c101fe0 msgr2=0x7fe97c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97aa00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c101fe0 0x7fe97c074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97aa00640 1 -- 172.21.15.28:0/3737820089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97c1a6f70 con 0x7fe97c1070b0 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97b400640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c101fe0 0x7fe97c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.525+0000 7fe97aa00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c1070b0 0x7fe97c074790 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe96402a860 tx=0x7fe964032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.526+0000 7fe973e00640 1 -- 172.21.15.28:0/3737820089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964037040 con 0x7fe97c1070b0 2024-04-26T01:42:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.526+0000 7fe973e00640 1 -- 172.21.15.28:0/3737820089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe964004050 con 0x7fe97c1070b0 2024-04-26T01:42:20.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.526+0000 7fe973e00640 1 -- 172.21.15.28:0/3737820089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9640035f0 con 0x7fe97c1070b0 2024-04-26T01:42:20.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.526+0000 7fe980a00640 1 -- 172.21.15.28:0/3737820089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97c1a7200 con 0x7fe97c1070b0 2024-04-26T01:42:20.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.526+0000 7fe980a00640 1 -- 172.21.15.28:0/3737820089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97c1a7610 con 0x7fe97c1070b0 2024-04-26T01:42:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe973e00640 1 -- 172.21.15.28:0/3737820089 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fe964043020 con 0x7fe97c1070b0 2024-04-26T01:42:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe984dff740 1 Processor -- start 2024-04-26T01:42:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe984dff740 1 -- start start 2024-04-26T01:42:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe984dff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f59b0 0x55f4d26f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe984dff740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d2700650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe984dff740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d2702f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe984dff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4d265b900 con 0x55f4d26f59b0 2024-04-26T01:42:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe984dff740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f4d2601ef0 con 0x55f4d26f62d0 2024-04-26T01:42:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.530+0000 7fe984dff740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f4d26578c0 con 0x55f4d2700b90 2024-04-26T01:42:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f59b0 0x55f4d26f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f59b0 0x55f4d26f5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46570/0 (socket says 172.21.15.28:46570) 2024-04-26T01:42:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97be00640 1 -- 172.21.15.28:0/991807736 learned_addr learned my addr 172.21.15.28:0/991807736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97b400640 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d2700650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97aa00640 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d2702f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97b400640 1 -- 172.21.15.28:0/991807736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 msgr2=0x55f4d2702f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97b400640 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d2702f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97b400640 1 -- 172.21.15.28:0/991807736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f59b0 msgr2=0x55f4d26f5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97b400640 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f59b0 0x55f4d26f5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.531+0000 7fe97b400640 1 -- 172.21.15.28:0/991807736 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d236e210 con 0x55f4d26f62d0 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.532+0000 7fe97b400640 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d2700650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00e880 tx=0x7fe96c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ff44188ba267c38 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.532+0000 7fe973e00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe9600a3ee0 0x7fe9600a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.532+0000 7fe96be00640 1 -- 172.21.15.28:0/991807736 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c00ddd0 con 0x55f4d26f62d0 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.532+0000 7fe96be00640 1 -- 172.21.15.28:0/991807736 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe96c003e20 con 0x55f4d26f62d0 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.532+0000 7fe96be00640 1 -- 172.21.15.28:0/991807736 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c017680 con 0x55f4d26f62d0 2024-04-26T01:42:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.532+0000 7fe973e00640 1 -- 172.21.15.28:0/3737820089 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fe9640efad0 con 0x7fe97c1070b0 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.532+0000 7fe97be00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe9600a3ee0 0x7fe9600a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 -- 172.21.15.28:0/991807736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 msgr2=0x55f4d2700650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d2700650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00e880 tx=0x7fe96c0095c0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe97be00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe9600a3ee0 0x7fe9600a63a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe974009e50 tx=0x7fe974005f90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 -- 172.21.15.28:0/991807736 shutdown_connections 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d2702f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d2700650 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 --2- 172.21.15.28:0/991807736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f59b0 0x55f4d26f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 -- 172.21.15.28:0/991807736 >> 172.21.15.28:0/991807736 conn(0x55f4d26f1ec0 msgr2=0x55f4d26f22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 -- 172.21.15.28:0/991807736 shutdown_connections 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 -- 172.21.15.28:0/991807736 wait complete. 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.533+0000 7fe984dff740 1 -- start start 2024-04-26T01:42:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.534+0000 7fe984dff740 1 Processor -- start 2024-04-26T01:42:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.534+0000 7fe984dff740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d26fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.534+0000 7fe984dff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.534+0000 7fe984dff740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d26fe420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.534+0000 7fe984dff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4d27037c0 con 0x55f4d26f62d0 2024-04-26T01:42:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.534+0000 7fe984dff740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f4d2615310 con 0x55f4d26f59b0 2024-04-26T01:42:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.534+0000 7fe984dff740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f4d2703940 con 0x55f4d2700b90 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.534+0000 7fe984dff740 10 client.0 _ll_register_callbacks cb 0x55f4d25e3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d26fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d26fe420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d26fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:40740/0 (socket says 172.21.15.28:40740) 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 -- 172.21.15.28:0/2968197191 learned_addr learned my addr 172.21.15.28:0/2968197191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 -- 172.21.15.28:0/2968197191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 msgr2=0x55f4d26fe420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d26fe420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 -- 172.21.15.28:0/2968197191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 msgr2=0x55f4d26fdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 -- 172.21.15.28:0/2968197191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d243a620 con 0x55f4d26f59b0 2024-04-26T01:42:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97aa00640 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe97b400640 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d26fd9a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00d7b0 tx=0x7fe96c005af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5d9f39598dd86 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.535+0000 7fe968c00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.536+0000 7fe968c00640 1 -- 172.21.15.28:0/2968197191 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c003970 con 0x55f4d26f59b0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.536+0000 7fe984dff740 1 -- 172.21.15.28:0/2968197191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe96c003e20 con 0x55f4d26f59b0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.536+0000 7fe968c00640 1 -- 172.21.15.28:0/2968197191 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fe96c003b10 con 0x55f4d26f59b0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.536+0000 7fe968c00640 1 -- 172.21.15.28:0/2968197191 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe96c003fd0 con 0x55f4d26f59b0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.536+0000 7fe984dff740 10 client.25343 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.536+0000 7fe984dff740 1 -- 172.21.15.28:0/2968197191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f4d270f8b0 con 0x55f4d26f59b0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.536+0000 7fe968c00640 1 -- 172.21.15.28:0/2968197191 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe96c003e20 con 0x55f4d26f59b0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.537+0000 7fe968c00640 1 -- 172.21.15.28:0/2968197191 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fe96c00b040 con 0x55f4d26f59b0 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.537+0000 7fe984dff740 10 client.25343 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.537+0000 7fe984dff740 10 client.25343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.537+0000 7fe984dff740 20 client.25343 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.537+0000 7fe984dff740 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f4d2710cc0 0x55f4d27130b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.537+0000 7fe984dff740 4 client.25343 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:42:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.537+0000 7fe984dff740 1 -- 172.21.15.28:0/2968197191 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4d266f8b0 con 0x55f4d2710cc0 2024-04-26T01:42:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.537+0000 7fe97aa00640 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f4d2710cc0 0x55f4d27130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.538+0000 7fe97aa00640 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f4d2710cc0 0x55f4d27130b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=779655847f3948b6 server_cookie=7be8cd85f278c214 in_seq=0 out_seq=0 2024-04-26T01:42:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.538+0000 7fe968c00640 10 client.25343 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe968c00640 1 -- 172.21.15.28:0/2968197191 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f4d266f8b0 con 0x55f4d2710cc0 2024-04-26T01:42:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe968c00640 10 client.25343 handle_command_reply: tid=0 2024-04-26T01:42:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 client.25343 shutdown 2024-04-26T01:42:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 -- 172.21.15.28:0/2968197191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 msgr2=0x55f4d26fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d26fd9a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00d7b0 tx=0x7fe96c005af0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 -- 172.21.15.28:0/2968197191 shutdown_connections 2024-04-26T01:42:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f4d2710cc0 0x55f4d27130b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d26fe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 --2- 172.21.15.28:0/2968197191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d26fd9a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 -- 172.21.15.28:0/2968197191 >> 172.21.15.28:0/2968197191 conn(0x55f4d26f1ec0 msgr2=0x55f4d26f7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe968c00640 0 client.25343 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 -- 172.21.15.28:0/2968197191 shutdown_connections 2024-04-26T01:42:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 1 -- 172.21.15.28:0/2968197191 wait complete. 2024-04-26T01:42:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.539+0000 7fe984dff740 20 client.25343 trim_cache size 0 max 16384 2024-04-26T01:42:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.556+0000 7fe984dff740 1 Processor -- start 2024-04-26T01:42:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.556+0000 7fe984dff740 1 -- start start 2024-04-26T01:42:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.556+0000 7fe984dff740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d270c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.556+0000 7fe984dff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.556+0000 7fe984dff740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d26f6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.556+0000 7fe984dff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c1b2420 con 0x55f4d26f62d0 2024-04-26T01:42:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.556+0000 7fe984dff740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c1afcc0 con 0x55f4d26f59b0 2024-04-26T01:42:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.556+0000 7fe984dff740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c1adb10 con 0x55f4d2700b90 2024-04-26T01:42:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d270c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d26f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fcb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46580/0 (socket says 172.21.15.28:46580) 2024-04-26T01:42:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 -- 172.21.15.28:0/1191231354 learned_addr learned my addr 172.21.15.28:0/1191231354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 -- 172.21.15.28:0/1191231354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 msgr2=0x55f4d26f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 --2- 172.21.15.28:0/1191231354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d26f6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 -- 172.21.15.28:0/1191231354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 msgr2=0x55f4d270c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:20.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 --2- 172.21.15.28:0/1191231354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d270c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 -- 172.21.15.28:0/1191231354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d26f6100 con 0x55f4d26f62d0 2024-04-26T01:42:20.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97b400640 1 --2- 172.21.15.28:0/1191231354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d270c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:20.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.557+0000 7fe97aa00640 1 --2- 172.21.15.28:0/1191231354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fcb20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe96402a930 tx=0x7fe9640f1f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a1119a04944c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:20.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe96b400640 1 -- 172.21.15.28:0/1191231354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9640b7ba0 con 0x55f4d26f62d0 2024-04-26T01:42:20.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe96b400640 1 -- 172.21.15.28:0/1191231354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9640328f0 con 0x55f4d26f62d0 2024-04-26T01:42:20.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe96b400640 1 -- 172.21.15.28:0/1191231354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9640bb260 con 0x55f4d26f62d0 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe984dff740 1 -- 172.21.15.28:0/1191231354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 msgr2=0x55f4d26fcb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe984dff740 1 --2- 172.21.15.28:0/1191231354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fcb20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe96402a930 tx=0x7fe9640f1f20 comp rx=0 tx=0).stop 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe984dff740 1 -- 172.21.15.28:0/1191231354 shutdown_connections 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe984dff740 1 --2- 172.21.15.28:0/1191231354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d2700b90 0x55f4d26f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe984dff740 1 --2- 172.21.15.28:0/1191231354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d26f62d0 0x55f4d26fcb20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe984dff740 1 --2- 172.21.15.28:0/1191231354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f59b0 0x55f4d270c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe984dff740 1 -- 172.21.15.28:0/1191231354 >> 172.21.15.28:0/1191231354 conn(0x55f4d26f1ec0 msgr2=0x55f4d26ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.558+0000 7fe984dff740 1 -- 172.21.15.28:0/1191231354 shutdown_connections 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 -- 172.21.15.28:0/1191231354 wait complete. 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 -- start start 2024-04-26T01:42:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 Processor -- start 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d26f59b0 0x55f4d2708bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d27090f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d2700b90 0x55f4d2706340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c1ae180 con 0x55f4d2700b90 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c1af5f0 con 0x55f4d26f62d0 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.559+0000 7fe984dff740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c1ae850 con 0x55f4d26f59b0 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe984dff740 10 client.0 _ll_register_callbacks cb 0x7fe9640f3430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d2700b90 0x55f4d2706340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97b400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d26f59b0 0x55f4d2708bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d2700b90 0x55f4d2706340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46588/0 (socket says 172.21.15.28:46588) 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97be00640 1 -- 172.21.15.28:0/579754174 learned_addr learned my addr 172.21.15.28:0/579754174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97aa00640 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d27090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97b400640 1 -- 172.21.15.28:0/579754174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 msgr2=0x55f4d27090f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97b400640 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d27090f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97b400640 1 -- 172.21.15.28:0/579754174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d2700b90 msgr2=0x55f4d2706340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97b400640 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d2700b90 0x55f4d2706340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97b400640 1 -- 172.21.15.28:0/579754174 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d2706b10 con 0x55f4d26f59b0 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97aa00640 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d27090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.560+0000 7fe97be00640 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d2700b90 0x55f4d2706340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.561+0000 7fe97b400640 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d26f59b0 0x55f4d2708bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe96c035cf0 tx=0x7fe96c00e450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e9ed860e2a4f292 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.561+0000 7fe968c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.561+0000 7fe968c00640 1 -- 172.21.15.28:0/579754174 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c00dea0 con 0x55f4d26f59b0 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.561+0000 7fe984dff740 1 -- 172.21.15.28:0/579754174 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9640328f0 con 0x55f4d26f59b0 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.561+0000 7fe968c00640 1 -- 172.21.15.28:0/579754174 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fe96c004170 con 0x55f4d26f59b0 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.561+0000 7fe968c00640 1 -- 172.21.15.28:0/579754174 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe96c0043b0 con 0x55f4d26f59b0 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.561+0000 7fe984dff740 10 client.25390 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.561+0000 7fe984dff740 1 -- 172.21.15.28:0/579754174 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f4d2776d00 con 0x55f4d26f59b0 2024-04-26T01:42:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe968c00640 1 -- 172.21.15.28:0/579754174 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9640328f0 con 0x55f4d26f59b0 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe968c00640 1 -- 172.21.15.28:0/579754174 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fe96c00b050 con 0x55f4d26f59b0 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe984dff740 10 client.25390 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe984dff740 10 client.25390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe984dff740 20 client.25390 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe984dff740 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f4d2777420 0x55f4d2779840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe984dff740 4 client.25390 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe984dff740 1 -- 172.21.15.28:0/579754174 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4d2669320 con 0x55f4d2777420 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.562+0000 7fe97aa00640 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f4d2777420 0x55f4d2779840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.563+0000 7fe97aa00640 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f4d2777420 0x55f4d2779840 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=806d605617ec2a23 server_cookie=6ca1d70137cf60f6 in_seq=0 out_seq=0 2024-04-26T01:42:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.563+0000 7fe968c00640 10 client.25390 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe968c00640 1 -- 172.21.15.28:0/579754174 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f4d2669320 con 0x55f4d2777420 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe968c00640 10 client.25390 handle_command_reply: tid=0 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 client.25390 shutdown 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 -- 172.21.15.28:0/579754174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d26f59b0 msgr2=0x55f4d2708bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d26f59b0 0x55f4d2708bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe96c035cf0 tx=0x7fe96c00e450 comp rx=0 tx=0).stop 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 -- 172.21.15.28:0/579754174 shutdown_connections 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f4d2777420 0x55f4d2779840 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d2700b90 0x55f4d2706340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f4d26f62d0 0x55f4d27090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 --2- 172.21.15.28:0/579754174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f4d26f59b0 0x55f4d2708bb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe968c00640 0 client.25390 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 -- 172.21.15.28:0/579754174 >> 172.21.15.28:0/579754174 conn(0x55f4d26f1ec0 msgr2=0x55f4d26fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 -- 172.21.15.28:0/579754174 shutdown_connections 2024-04-26T01:42:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.589+0000 7fe984dff740 1 -- 172.21.15.28:0/579754174 wait complete. 2024-04-26T01:42:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.590+0000 7fe984dff740 20 client.25390 trim_cache size 0 max 16384 2024-04-26T01:42:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:42:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:42:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:42:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:42:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.590+0000 7fe980a00640 1 -- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe9600a3ee0 msgr2=0x7fe9600a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.590+0000 7fe980a00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe9600a3ee0 0x7fe9600a63a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe974009e50 tx=0x7fe974005f90 comp rx=0 tx=0).stop 2024-04-26T01:42:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 -- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c1070b0 msgr2=0x7fe97c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c1070b0 0x7fe97c074790 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe96402a860 tx=0x7fe964032d50 comp rx=0 tx=0).stop 2024-04-26T01:42:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 -- 172.21.15.28:0/3737820089 shutdown_connections 2024-04-26T01:42:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe9600a3ee0 0x7fe9600a63a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe97c074cd0 0x7fe97c1a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c1070b0 0x7fe97c074790 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 --2- 172.21.15.28:0/3737820089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c101fe0 0x7fe97c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 -- 172.21.15.28:0/3737820089 >> 172.21.15.28:0/3737820089 conn(0x7fe97c0fd8a0 msgr2=0x7fe97c0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 -- 172.21.15.28:0/3737820089 shutdown_connections 2024-04-26T01:42:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:20.591+0000 7fe980a00640 1 -- 172.21.15.28:0/3737820089 wait complete. 2024-04-26T01:42:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 25.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:42:21.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:21 smithi169 ceph-mon[32851]: pgmap v44: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.2 MiB/s wr, 550 op/s 2024-04-26T01:42:21.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:21 smithi053 ceph-mon[32593]: pgmap v44: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.2 MiB/s wr, 550 op/s 2024-04-26T01:42:21.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:21 smithi028 ceph-mon[28247]: pgmap v44: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.2 MiB/s wr, 550 op/s 2024-04-26T01:42:23.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:23 smithi169 ceph-mon[32851]: pgmap v45: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-04-26T01:42:23.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:23 smithi053 ceph-mon[32593]: pgmap v45: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-04-26T01:42:23.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:23 smithi028 ceph-mon[28247]: pgmap v45: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-04-26T01:42:25.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:25 smithi169 ceph-mon[32851]: pgmap v46: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 2.0 MiB/s wr, 276 op/s 2024-04-26T01:42:25.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:25 smithi053 ceph-mon[32593]: pgmap v46: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 2.0 MiB/s wr, 276 op/s 2024-04-26T01:42:25.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:25 smithi028 ceph-mon[28247]: pgmap v46: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 2.0 MiB/s wr, 276 op/s 2024-04-26T01:42:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 30.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:42:27.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:27 smithi169 ceph-mon[32851]: pgmap v47: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-04-26T01:42:27.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:27 smithi053 ceph-mon[32593]: pgmap v47: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-04-26T01:42:27.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:27 smithi028 ceph-mon[28247]: pgmap v47: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-04-26T01:42:29.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:29 smithi169 ceph-mon[32851]: pgmap v48: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-04-26T01:42:29.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:29 smithi053 ceph-mon[32593]: pgmap v48: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-04-26T01:42:29.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:29 smithi028 ceph-mon[28247]: pgmap v48: 97 pgs: 97 active+clean; 998 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-04-26T01:42:31.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 35.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:42:31.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:31 smithi169 ceph-mon[32851]: pgmap v49: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.9 MiB/s wr, 274 op/s 2024-04-26T01:42:31.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:31 smithi169 ceph-mon[32851]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-26T01:42:31.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:31 smithi053 ceph-mon[32593]: pgmap v49: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.9 MiB/s wr, 274 op/s 2024-04-26T01:42:31.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:31 smithi053 ceph-mon[32593]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-26T01:42:31.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:31 smithi028 ceph-mon[28247]: pgmap v49: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.9 MiB/s wr, 274 op/s 2024-04-26T01:42:31.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:31 smithi028 ceph-mon[28247]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-26T01:42:33.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:33 smithi169 ceph-mon[32851]: pgmap v50: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 137 op/s 2024-04-26T01:42:33.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:33 smithi053 ceph-mon[32593]: pgmap v50: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 137 op/s 2024-04-26T01:42:33.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:33 smithi028 ceph-mon[28247]: pgmap v50: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 137 op/s 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 -- 172.21.15.28:0/3593832641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae4109490 msgr2=0x7fcae4111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 --2- 172.21.15.28:0/3593832641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae4109490 0x7fcae4111060 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcadc009900 tx=0x7fcadc02a410 comp rx=0 tx=0).stop 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 -- 172.21.15.28:0/3593832641 shutdown_connections 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 --2- 172.21.15.28:0/3593832641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae4109490 0x7fcae4111060 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 --2- 172.21.15.28:0/3593832641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae4104d70 0x7fcae4108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 --2- 172.21.15.28:0/3593832641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae41043c0 0x7fcae41047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 -- 172.21.15.28:0/3593832641 >> 172.21.15.28:0/3593832641 conn(0x7fcae40ffc20 msgr2=0x7fcae4102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 -- 172.21.15.28:0/3593832641 shutdown_connections 2024-04-26T01:42:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.742+0000 7fcaea200640 1 -- 172.21.15.28:0/3593832641 wait complete. 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.743+0000 7fcaea200640 1 Processor -- start 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.743+0000 7fcaea200640 1 -- start start 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcaea200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae41043c0 0x7fcae41a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcaea200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae4109490 0x7fcae41a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcaea200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae419a780 0x7fcae419ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcaea200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcae4114400 con 0x7fcae4109490 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcaea200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae4114280 con 0x7fcae419a780 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcaea200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcae4114580 con 0x7fcae41043c0 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcae2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae4109490 0x7fcae41a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcae2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae4109490 0x7fcae41a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45020/0 (socket says 172.21.15.28:45020) 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcae2a00640 1 -- 172.21.15.28:0/657502890 learned_addr learned my addr 172.21.15.28:0/657502890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcae3e00640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae419a780 0x7fcae419ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.744+0000 7fcae3400640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae41043c0 0x7fcae41a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcae2a00640 1 -- 172.21.15.28:0/657502890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae41043c0 msgr2=0x7fcae41a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcae2a00640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae41043c0 0x7fcae41a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcae2a00640 1 -- 172.21.15.28:0/657502890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae419a780 msgr2=0x7fcae419ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcae2a00640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae419a780 0x7fcae419ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcae2a00640 1 -- 172.21.15.28:0/657502890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae419b380 con 0x7fcae4109490 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcae2a00640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae4109490 0x7fcae41a0bb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcad000e9f0 tx=0x7fcad000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcadbe00640 1 -- 172.21.15.28:0/657502890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad000ccc0 con 0x7fcae4109490 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcadbe00640 1 -- 172.21.15.28:0/657502890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcad00054d0 con 0x7fcae4109490 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcadbe00640 1 -- 172.21.15.28:0/657502890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad0010430 con 0x7fcae4109490 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.745+0000 7fcaea200640 1 -- 172.21.15.28:0/657502890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae41a7410 con 0x7fcae4109490 2024-04-26T01:42:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.746+0000 7fcaea200640 1 -- 172.21.15.28:0/657502890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae41a77c0 con 0x7fcae4109490 2024-04-26T01:42:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.749+0000 7fcaed00b740 1 Processor -- start 2024-04-26T01:42:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.749+0000 7fcaed00b740 1 -- start start 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.749+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.749+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.749+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711b2a00 0x5595711b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.749+0000 7fcaed00b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55957110d900 con 0x5595711a8140 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.749+0000 7fcaed00b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5595710b3ef0 con 0x5595711a7820 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcaed00b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5595711098c0 con 0x5595711b2a00 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcadbe00640 1 -- 172.21.15.28:0/657502890 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcad00046c0 con 0x7fcae4109490 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711a7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:34288/0 (socket says 172.21.15.28:34288) 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3e00640 1 -- 172.21.15.28:0/354307768 learned_addr learned my addr 172.21.15.28:0/354307768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3e00640 1 -- 172.21.15.28:0/354307768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711b2a00 msgr2=0x5595711b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3e00640 1 --2- 172.21.15.28:0/354307768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711b2a00 0x5595711b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3e00640 1 -- 172.21.15.28:0/354307768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 msgr2=0x5595711b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3e00640 1 --2- 172.21.15.28:0/354307768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3e00640 1 -- 172.21.15.28:0/354307768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559570e20210 con 0x5595711a7820 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.750+0000 7fcae3400640 1 --2- 172.21.15.28:0/354307768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.751+0000 7fcae3e00640 1 --2- 172.21.15.28:0/354307768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711a7c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcadc02a930 tx=0x7fcadc032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae7ccfbb4e63bedf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.751+0000 7fcacfe00640 1 -- 172.21.15.28:0/354307768 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc040070 con 0x5595711a7820 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.751+0000 7fcacfe00640 1 -- 172.21.15.28:0/354307768 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcadc004030 con 0x5595711a7820 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.751+0000 7fcacfe00640 1 -- 172.21.15.28:0/354307768 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc035040 con 0x5595711a7820 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.751+0000 7fcadbe00640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcac80a3e40 0x7fcac80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.751+0000 7fcadbe00640 1 -- 172.21.15.28:0/657502890 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcad00d0a80 con 0x7fcae4109490 2024-04-26T01:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcae3400640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcac80a3e40 0x7fcac80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 -- 172.21.15.28:0/354307768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 msgr2=0x5595711a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 --2- 172.21.15.28:0/354307768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711a7c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcadc02a930 tx=0x7fcadc032b20 comp rx=0 tx=0).stop 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 -- 172.21.15.28:0/354307768 shutdown_connections 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 --2- 172.21.15.28:0/354307768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711b2a00 0x5595711b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 --2- 172.21.15.28:0/354307768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 --2- 172.21.15.28:0/354307768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711a7c00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 -- 172.21.15.28:0/354307768 >> 172.21.15.28:0/354307768 conn(0x5595711a3d30 msgr2=0x5595711a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 -- 172.21.15.28:0/354307768 shutdown_connections 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcaed00b740 1 -- 172.21.15.28:0/354307768 wait complete. 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.752+0000 7fcae3400640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcac80a3e40 0x7fcac80a6300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x5595711a8dc0 tx=0x7fcad4009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.753+0000 7fcaed00b740 1 -- start start 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.753+0000 7fcaed00b740 1 Processor -- start 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.753+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.753+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711aff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.753+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711b2a00 0x5595711b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.753+0000 7fcaed00b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595711b55d0 con 0x5595711a8140 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.753+0000 7fcaed00b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5595710c7310 con 0x5595711b2a00 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.753+0000 7fcaed00b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5595711b5750 con 0x5595711a7820 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcaed00b740 10 client.0 _ll_register_callbacks cb 0x559571095cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711b2a00 0x5595711b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54038/0 (socket says 172.21.15.28:54038) 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3400640 1 -- 172.21.15.28:0/3521421919 learned_addr learned my addr 172.21.15.28:0/3521421919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3e00640 1 -- 172.21.15.28:0/3521421919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 msgr2=0x5595711af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3e00640 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3e00640 1 -- 172.21.15.28:0/3521421919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 msgr2=0x5595711aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3e00640 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.754+0000 7fcae3e00640 1 -- 172.21.15.28:0/3521421919 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559570eec620 con 0x5595711b2a00 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.755+0000 7fcae3e00640 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711b2a00 0x5595711b0440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcadc005940 tx=0x7fcadc009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511d06224c88c746 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.755+0000 7fcaccc00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.755+0000 7fcaccc00640 1 -- 172.21.15.28:0/3521421919 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc052060 con 0x5595711b2a00 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.755+0000 7fcaed00b740 1 -- 172.21.15.28:0/3521421919 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcadc004030 con 0x5595711b2a00 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.755+0000 7fcaccc00640 1 -- 172.21.15.28:0/3521421919 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcadc005480 con 0x5595711b2a00 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.755+0000 7fcaccc00640 1 -- 172.21.15.28:0/3521421919 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcadc035070 con 0x5595711b2a00 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.755+0000 7fcaed00b740 10 client.25361 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.755+0000 7fcaed00b740 1 -- 172.21.15.28:0/3521421919 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595711c18a0 con 0x5595711b2a00 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcaccc00640 1 -- 172.21.15.28:0/3521421919 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcadc004030 con 0x5595711b2a00 2024-04-26T01:42:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcaccc00640 1 -- 172.21.15.28:0/3521421919 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fcadc004ba0 con 0x5595711b2a00 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcaed00b740 10 client.25361 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcaed00b740 10 client.25361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcaed00b740 20 client.25361 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcaed00b740 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5595711c2bf0 0x5595711c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcaed00b740 4 client.25361 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcaed00b740 1 -- 172.21.15.28:0/3521421919 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595711218b0 con 0x5595711c2bf0 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.756+0000 7fcae3400640 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5595711c2bf0 0x5595711c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.757+0000 7fcae3400640 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5595711c2bf0 0x5595711c4fe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=465f061ff79b7408 server_cookie=7423b38d57350ec7 in_seq=0 out_seq=0 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.757+0000 7fcaccc00640 10 client.25361 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaccc00640 1 -- 172.21.15.28:0/3521421919 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595711218b0 con 0x5595711c2bf0 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaccc00640 10 client.25361 handle_command_reply: tid=0 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 client.25361 shutdown 2024-04-26T01:42:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 -- 172.21.15.28:0/3521421919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711b2a00 msgr2=0x5595711b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711b2a00 0x5595711b0440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcadc005940 tx=0x7fcadc009520 comp rx=0 tx=0).stop 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 -- 172.21.15.28:0/3521421919 shutdown_connections 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5595711c2bf0 0x5595711c4fe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711b2a00 0x5595711b0440 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaccc00640 0 client.25361 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711a8140 0x5595711aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 --2- 172.21.15.28:0/3521421919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 -- 172.21.15.28:0/3521421919 >> 172.21.15.28:0/3521421919 conn(0x5595711a3d30 msgr2=0x5595711a9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 -- 172.21.15.28:0/3521421919 shutdown_connections 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.758+0000 7fcaed00b740 1 -- 172.21.15.28:0/3521421919 wait complete. 2024-04-26T01:42:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.759+0000 7fcaed00b740 20 client.25361 trim_cache size 0 max 16384 2024-04-26T01:42:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.775+0000 7fcaed00b740 1 Processor -- start 2024-04-26T01:42:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.775+0000 7fcaed00b740 1 -- start start 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.775+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.775+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a8140 0x5595711a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.775+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595711af980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.775+0000 7fcaed00b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41b2520 con 0x5595711b2a00 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.775+0000 7fcaed00b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41afdc0 con 0x5595711a8140 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.775+0000 7fcaed00b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41adc30 con 0x5595711a7820 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595711af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae3400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a8140 0x5595711a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae3400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54044/0 (socket says 172.21.15.28:54044) 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae3400640 1 -- 172.21.15.28:0/862021297 learned_addr learned my addr 172.21.15.28:0/862021297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595711af980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45044/0 (socket says 172.21.15.28:45044) 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae2a00640 1 -- 172.21.15.28:0/862021297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 msgr2=0x5595711aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae2a00640 1 --2- 172.21.15.28:0/862021297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae2a00640 1 -- 172.21.15.28:0/862021297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 msgr2=0x5595711af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae2a00640 1 --2- 172.21.15.28:0/862021297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595711af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae2a00640 1 -- 172.21.15.28:0/862021297 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595711adf10 con 0x5595711a8140 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae3e00640 1 --2- 172.21.15.28:0/862021297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595711af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.776+0000 7fcae2a00640 1 --2- 172.21.15.28:0/862021297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a8140 0x5595711a2d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcad00ce1a0 tx=0x7fcad0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34f03ffa3e306783 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcacf400640 1 -- 172.21.15.28:0/862021297 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad00bd070 con 0x5595711a8140 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcacf400640 1 -- 172.21.15.28:0/862021297 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcad009d0f0 con 0x5595711a8140 2024-04-26T01:42:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcacf400640 1 -- 172.21.15.28:0/862021297 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad00ce420 con 0x5595711a8140 2024-04-26T01:42:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 -- 172.21.15.28:0/862021297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a8140 msgr2=0x5595711a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 --2- 172.21.15.28:0/862021297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a8140 0x5595711a2d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcad00ce1a0 tx=0x7fcad0009310 comp rx=0 tx=0).stop 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 -- 172.21.15.28:0/862021297 shutdown_connections 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 --2- 172.21.15.28:0/862021297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595711af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 --2- 172.21.15.28:0/862021297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a8140 0x5595711a2d10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 --2- 172.21.15.28:0/862021297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a7820 0x5595711aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 -- 172.21.15.28:0/862021297 >> 172.21.15.28:0/862021297 conn(0x5595711a3d30 msgr2=0x5595711a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 -- 172.21.15.28:0/862021297 shutdown_connections 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.777+0000 7fcaed00b740 1 -- 172.21.15.28:0/862021297 wait complete. 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 1 -- start start 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 1 Processor -- start 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711c7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a8140 0x5595711c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595712200e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41ae280 con 0x5595711b2a00 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41af6f0 con 0x5595711a7820 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41ae950 con 0x5595711a8140 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.778+0000 7fcaed00b740 10 client.0 _ll_register_callbacks cb 0x7fcadc009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae2a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a8140 0x5595711c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595712200e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae2a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a8140 0x5595711c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54054/0 (socket says 172.21.15.28:54054) 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae2a00640 1 -- 172.21.15.28:0/566451168 learned_addr learned my addr 172.21.15.28:0/566451168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3400640 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3e00640 1 -- 172.21.15.28:0/566451168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a8140 msgr2=0x5595711c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3e00640 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a8140 0x5595711c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3e00640 1 -- 172.21.15.28:0/566451168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 msgr2=0x5595711c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3e00640 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3e00640 1 -- 172.21.15.28:0/566451168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559571220750 con 0x5595711b2a00 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae2a00640 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a8140 0x5595711c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3400640 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcae3e00640 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595712200e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcadc001eb0 tx=0x7fcadc005070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fbfb573479470 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.779+0000 7fcaccc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.780+0000 7fcaccc00640 1 -- 172.21.15.28:0/566451168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc052070 con 0x5595711b2a00 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.780+0000 7fcaccc00640 1 -- 172.21.15.28:0/566451168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcadc0314a0 con 0x5595711b2a00 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.780+0000 7fcaed00b740 1 -- 172.21.15.28:0/566451168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcad009d0f0 con 0x5595711b2a00 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.780+0000 7fcaccc00640 1 -- 172.21.15.28:0/566451168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcadc040040 con 0x5595711b2a00 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.780+0000 7fcaed00b740 10 client.15243 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.780+0000 7fcaed00b740 1 -- 172.21.15.28:0/566451168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559571228d00 con 0x5595711b2a00 2024-04-26T01:42:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.780+0000 7fcaccc00640 1 -- 172.21.15.28:0/566451168 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcad009d0f0 con 0x5595711b2a00 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.781+0000 7fcaccc00640 1 -- 172.21.15.28:0/566451168 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fcadc003ae0 con 0x5595711b2a00 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.781+0000 7fcaed00b740 10 client.15243 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.781+0000 7fcaed00b740 10 client.15243 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.781+0000 7fcaed00b740 20 client.15243 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.781+0000 7fcaed00b740 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559571229120 0x55957122b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.781+0000 7fcaed00b740 4 client.15243 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.781+0000 7fcaed00b740 1 -- 172.21.15.28:0/566451168 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55957111b320 con 0x559571229120 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.781+0000 7fcae3400640 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559571229120 0x55957122b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.782+0000 7fcae3400640 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559571229120 0x55957122b540 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1263661c2e3a7e server_cookie=62996398fe4fc29c in_seq=0 out_seq=0 2024-04-26T01:42:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.782+0000 7fcaccc00640 10 client.15243 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaccc00640 1 -- 172.21.15.28:0/566451168 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55957111b320 con 0x559571229120 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaccc00640 10 client.15243 handle_command_reply: tid=0 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 client.15243 shutdown 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 -- 172.21.15.28:0/566451168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 msgr2=0x5595712200e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595712200e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcadc001eb0 tx=0x7fcadc005070 comp rx=0 tx=0).stop 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 -- 172.21.15.28:0/566451168 shutdown_connections 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559571229120 0x55957122b540 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595711b2a00 0x5595712200e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5595711a8140 0x5595711c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaccc00640 0 client.15243 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 --2- 172.21.15.28:0/566451168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5595711a7820 0x5595711c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 -- 172.21.15.28:0/566451168 >> 172.21.15.28:0/566451168 conn(0x5595711a3d30 msgr2=0x5595711af960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 -- 172.21.15.28:0/566451168 shutdown_connections 2024-04-26T01:42:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 1 -- 172.21.15.28:0/566451168 wait complete. 2024-04-26T01:42:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.799+0000 7fcaed00b740 20 client.15243 trim_cache size 0 max 16384 2024-04-26T01:42:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:42:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:42:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:42:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:42:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 -- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcac80a3e40 msgr2=0x7fcac80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcac80a3e40 0x7fcac80a6300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x5595711a8dc0 tx=0x7fcad4009500 comp rx=0 tx=0).stop 2024-04-26T01:42:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 -- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae4109490 msgr2=0x7fcae41a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae4109490 0x7fcae41a0bb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcad000e9f0 tx=0x7fcad000eeb0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 -- 172.21.15.28:0/657502890 shutdown_connections 2024-04-26T01:42:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcac80a3e40 0x7fcac80a6300 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae419a780 0x7fcae419ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae4109490 0x7fcae41a0bb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 --2- 172.21.15.28:0/657502890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae41043c0 0x7fcae41a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.800+0000 7fcaea200640 1 -- 172.21.15.28:0/657502890 >> 172.21.15.28:0/657502890 conn(0x7fcae40ffc20 msgr2=0x7fcae4101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.801+0000 7fcaea200640 1 -- 172.21.15.28:0/657502890 shutdown_connections 2024-04-26T01:42:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:33.801+0000 7fcaea200640 1 -- 172.21.15.28:0/657502890 wait complete. 2024-04-26T01:42:35.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:35 smithi169 ceph-mon[32851]: pgmap v51: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.5 MiB/s wr, 143 op/s 2024-04-26T01:42:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:35 smithi053 ceph-mon[32593]: pgmap v51: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.5 MiB/s wr, 143 op/s 2024-04-26T01:42:35.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:35 smithi028 ceph-mon[28247]: pgmap v51: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.5 MiB/s wr, 143 op/s 2024-04-26T01:42:36.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 40.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:42:36.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:42:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:37.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:37 smithi028 ceph-mon[28247]: pgmap v52: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 MiB/s wr, 132 op/s 2024-04-26T01:42:37.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:37 smithi169 ceph-mon[32851]: pgmap v52: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 MiB/s wr, 132 op/s 2024-04-26T01:42:37.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:37 smithi053 ceph-mon[32593]: pgmap v52: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 MiB/s wr, 132 op/s 2024-04-26T01:42:39.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:39 smithi028 ceph-mon[28247]: pgmap v53: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 MiB/s wr, 132 op/s 2024-04-26T01:42:39.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:39 smithi053 ceph-mon[32593]: pgmap v53: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 MiB/s wr, 132 op/s 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 -- 172.21.15.28:0/4125154994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c102af0 msgr2=0x7ff68c102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 --2- 172.21.15.28:0/4125154994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c102af0 0x7ff68c102f70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff6780099c0 tx=0x7ff678009cd0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 -- 172.21.15.28:0/4125154994 shutdown_connections 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 --2- 172.21.15.28:0/4125154994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1034b0 0x7ff68c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 --2- 172.21.15.28:0/4125154994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff68c107940 0x7ff68c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 --2- 172.21.15.28:0/4125154994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c102af0 0x7ff68c102f70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 -- 172.21.15.28:0/4125154994 >> 172.21.15.28:0/4125154994 conn(0x7ff68c079ba0 msgr2=0x7ff68c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 -- 172.21.15.28:0/4125154994 shutdown_connections 2024-04-26T01:42:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.562+0000 7ff690a00700 1 -- 172.21.15.28:0/4125154994 wait complete. 2024-04-26T01:42:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.563+0000 7ff690a00700 1 Processor -- start 2024-04-26T01:42:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.563+0000 7ff690a00700 1 -- start start 2024-04-26T01:42:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff690a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff68c102af0 0x7ff68c1991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff690a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1034b0 0x7ff68c199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff690a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c107940 0x7ff68c19da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff690a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c112cb0 con 0x7ff68c1034b0 2024-04-26T01:42:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff690a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c112b30 con 0x7ff68c107940 2024-04-26T01:42:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff690a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c112e30 con 0x7ff68c102af0 2024-04-26T01:42:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff689400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1034b0 0x7ff68c199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff689e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff68c102af0 0x7ff68c1991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff68a800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c107940 0x7ff68c19da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff689400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1034b0 0x7ff68c199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46436/0 (socket says 172.21.15.28:46436) 2024-04-26T01:42:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.564+0000 7ff689400700 1 -- 172.21.15.28:0/2114968651 learned_addr learned my addr 172.21.15.28:0/2114968651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.565+0000 7ff689400700 1 -- 172.21.15.28:0/2114968651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff68c102af0 msgr2=0x7ff68c1991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.565+0000 7ff689400700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff68c102af0 0x7ff68c1991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.565+0000 7ff689400700 1 -- 172.21.15.28:0/2114968651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c107940 msgr2=0x7ff68c19da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.565+0000 7ff689400700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c107940 0x7ff68c19da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.565+0000 7ff689400700 1 -- 172.21.15.28:0/2114968651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68c19e1d0 con 0x7ff68c1034b0 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.565+0000 7ff68a800700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c107940 0x7ff68c19da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.565+0000 7ff689400700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1034b0 0x7ff68c199700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff678006010 tx=0x7ff6780055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.566+0000 7ff682a00700 1 -- 172.21.15.28:0/2114968651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678026070 con 0x7ff68c1034b0 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.566+0000 7ff682a00700 1 -- 172.21.15.28:0/2114968651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff67800f460 con 0x7ff68c1034b0 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.566+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68c19e460 con 0x7ff68c1034b0 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.566+0000 7ff682a00700 1 -- 172.21.15.28:0/2114968651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678009e60 con 0x7ff68c1034b0 2024-04-26T01:42:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.566+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68c074c70 con 0x7ff68c1034b0 2024-04-26T01:42:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.567+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff68c1041b0 con 0x7ff68c1034b0 2024-04-26T01:42:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.569+0000 7ff682a00700 1 -- 172.21.15.28:0/2114968651 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff678029070 con 0x7ff68c1034b0 2024-04-26T01:42:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.571+0000 7ff682a00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6700a3db0 0x7ff6700a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.571+0000 7ff682a00700 1 -- 172.21.15.28:0/2114968651 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7ff6780d0fe0 con 0x7ff68c1034b0 2024-04-26T01:42:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.571+0000 7ff689e00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6700a3db0 0x7ff6700a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.572+0000 7ff689e00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6700a3db0 0x7ff6700a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009be0 tx=0x7ff67c009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.576+0000 7ff682a00700 1 -- 172.21.15.28:0/2114968651 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff67805dff0 con 0x7ff68c1034b0 2024-04-26T01:42:39.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:39 smithi169 ceph-mon[32851]: pgmap v53: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 MiB/s wr, 132 op/s 2024-04-26T01:42:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.853+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff68c069730 con 0x7ff68c1034b0 2024-04-26T01:42:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.856+0000 7ff682a00700 1 -- 172.21.15.28:0/2114968651 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11865 (secure 0 0 0) 0x7ff678013070 con 0x7ff68c1034b0 2024-04-26T01:42:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:42:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:42:00.871153+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:42:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6700a3db0 msgr2=0x7ff6700a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6700a3db0 0x7ff6700a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009be0 tx=0x7ff67c009260 comp rx=0 tx=0).stop 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1034b0 msgr2=0x7ff68c199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1034b0 0x7ff68c199700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff678006010 tx=0x7ff6780055f0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 shutdown_connections 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff6700a3db0 0x7ff6700a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c107940 0x7ff68c19da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff68c102af0 0x7ff68c1991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 --2- 172.21.15.28:0/2114968651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1034b0 0x7ff68c199700 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 >> 172.21.15.28:0/2114968651 conn(0x7ff68c079ba0 msgr2=0x7ff68c100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 shutdown_connections 2024-04-26T01:42:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:39.859+0000 7ff690a00700 1 -- 172.21.15.28:0/2114968651 wait complete. 2024-04-26T01:42:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-26T01:42:40.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:40 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2114968651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:40.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:40 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2114968651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:40.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:40 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2114968651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:40.789 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id b141a1f --release --await' 2024-04-26T01:42:40.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id b141a1f --release --await 2024-04-26T01:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 45.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:42:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:41.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:41 smithi028 ceph-mon[28247]: pgmap v54: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 135 op/s 2024-04-26T01:42:41.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:41 smithi053 ceph-mon[32593]: pgmap v54: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 135 op/s 2024-04-26T01:42:41.561 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:41 smithi169 ceph-mon[32851]: pgmap v54: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 135 op/s 2024-04-26T01:42:43.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:43 smithi028 ceph-mon[28247]: pgmap v55: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.9 KiB/s wr, 9 op/s 2024-04-26T01:42:43.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:43 smithi053 ceph-mon[32593]: pgmap v55: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.9 KiB/s wr, 9 op/s 2024-04-26T01:42:43.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:43 smithi169 ceph-mon[32851]: pgmap v55: 97 pgs: 97 active+clean; 1015 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.9 KiB/s wr, 9 op/s 2024-04-26T01:42:43.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.807+0000 7f0e1f000700 1 -- 172.21.15.28:0/378898537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18103450 msgr2=0x7f0e1810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.807+0000 7f0e1f000700 1 --2- 172.21.15.28:0/378898537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18103450 0x7f0e1810f8b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000b0a0 tx=0x7f0e1000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.807+0000 7f0e1f000700 1 -- 172.21.15.28:0/378898537 shutdown_connections 2024-04-26T01:42:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.807+0000 7f0e1f000700 1 --2- 172.21.15.28:0/378898537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18103450 0x7f0e1810f8b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.807+0000 7f0e1f000700 1 --2- 172.21.15.28:0/378898537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e18108a50 0x7f0e18108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.807+0000 7f0e1f000700 1 --2- 172.21.15.28:0/378898537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e18102a90 0x7f0e18102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.807+0000 7f0e1f000700 1 -- 172.21.15.28:0/378898537 >> 172.21.15.28:0/378898537 conn(0x7f0e18079ba0 msgr2=0x7f0e18079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.808+0000 7f0e1f000700 1 -- 172.21.15.28:0/378898537 shutdown_connections 2024-04-26T01:42:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.808+0000 7f0e1f000700 1 -- 172.21.15.28:0/378898537 wait complete. 2024-04-26T01:42:43.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.809+0000 7f0e1f000700 1 Processor -- start 2024-04-26T01:42:43.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.809+0000 7f0e1f000700 1 -- start start 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e1f000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18102a90 0x7f0e18199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e1f000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e18103450 0x7f0e18199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e1f000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e18108a50 0x7f0e1819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e1f000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e18112c00 con 0x7f0e18102a90 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e1f000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e18112a80 con 0x7f0e18108a50 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e1f000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e18112d80 con 0x7f0e18103450 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18102a90 0x7f0e18199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18102a90 0x7f0e18199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46460/0 (socket says 172.21.15.28:46460) 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e17400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e18103450 0x7f0e18199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e17e00700 1 -- 172.21.15.28:0/705800181 learned_addr learned my addr 172.21.15.28:0/705800181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:43.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.810+0000 7f0e1d000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e18108a50 0x7f0e1819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.811+0000 7f0e17400700 1 -- 172.21.15.28:0/705800181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e18108a50 msgr2=0x7f0e1819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.811+0000 7f0e17400700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e18108a50 0x7f0e1819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.811+0000 7f0e17400700 1 -- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18102a90 msgr2=0x7f0e18199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.811+0000 7f0e17400700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18102a90 0x7f0e18199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.811+0000 7f0e17400700 1 -- 172.21.15.28:0/705800181 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1819e140 con 0x7f0e18103450 2024-04-26T01:42:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.811+0000 7f0e17400700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e18103450 0x7f0e18199670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400e9f0 tx=0x7f0e0400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.811+0000 7f0e14c00700 1 -- 172.21.15.28:0/705800181 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0400c9c0 con 0x7f0e18103450 2024-04-26T01:42:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.812+0000 7f0e14c00700 1 -- 172.21.15.28:0/705800181 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e0400cb60 con 0x7f0e18103450 2024-04-26T01:42:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.812+0000 7f0e1f000700 1 -- 172.21.15.28:0/705800181 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1819e430 con 0x7f0e18103450 2024-04-26T01:42:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.812+0000 7f0e14c00700 1 -- 172.21.15.28:0/705800181 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e04018680 con 0x7f0e18103450 2024-04-26T01:42:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.812+0000 7f0e1f000700 1 -- 172.21.15.28:0/705800181 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e181a5cb0 con 0x7f0e18103450 2024-04-26T01:42:43.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.815+0000 7f0e14c00700 1 -- 172.21.15.28:0/705800181 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0e04010430 con 0x7f0e18103450 2024-04-26T01:42:43.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.817+0000 7f0e298e0b80 1 Processor -- start 2024-04-26T01:42:43.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.817+0000 7f0e298e0b80 1 -- start start 2024-04-26T01:42:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e14c00700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0dfc0a3db0 0x7f0dfc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c53120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c608c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55da53ba5f90 con 0x55da53c53660 2024-04-26T01:42:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55da53b4c830 con 0x55da53c5e4d0 2024-04-26T01:42:43.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55da53ba1f50 con 0x55da53c52d40 2024-04-26T01:42:43.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e14c00700 1 -- 172.21.15.28:0/705800181 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f0e040d0990 con 0x7f0e18103450 2024-04-26T01:42:43.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e17400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e17400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46464/0 (socket says 172.21.15.28:46464) 2024-04-26T01:42:43.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c53120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e17400700 1 -- 172.21.15.28:0/3939442349 learned_addr learned my addr 172.21.15.28:0/3939442349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:43.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.818+0000 7f0e1d000700 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e17e00700 1 -- 172.21.15.28:0/3939442349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 msgr2=0x55da53c608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e17e00700 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c608c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e17e00700 1 -- 172.21.15.28:0/3939442349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 msgr2=0x55da53c5df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e17e00700 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e17e00700 1 -- 172.21.15.28:0/3939442349 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da53b42780 con 0x55da53c52d40 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e17400700 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e1d000700 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e1d000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0dfc0a3db0 0x7f0dfc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e17e00700 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c53120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009d00 tx=0x7f0e08011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faaf57e130ab9ef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e0d600700 1 -- 172.21.15.28:0/3939442349 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08004a30 con 0x55da53c52d40 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e0d600700 1 -- 172.21.15.28:0/3939442349 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e080054d0 con 0x55da53c52d40 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.819+0000 7f0e0d600700 1 -- 172.21.15.28:0/3939442349 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e080198d0 con 0x55da53c52d40 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e298e0b80 1 -- 172.21.15.28:0/3939442349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 msgr2=0x55da53c53120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c53120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009d00 tx=0x7f0e08011720 comp rx=0 tx=0).stop 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e1d000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0dfc0a3db0 0x7f0dfc0a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55da53c54710 tx=0x7f0e1000b3b0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e298e0b80 1 -- 172.21.15.28:0/3939442349 shutdown_connections 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c53120 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/3939442349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e298e0b80 1 -- 172.21.15.28:0/3939442349 >> 172.21.15.28:0/3939442349 conn(0x55da53b82050 msgr2=0x55da53c4f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.820+0000 7f0e298e0b80 1 -- 172.21.15.28:0/3939442349 shutdown_connections 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.821+0000 7f0e298e0b80 1 -- 172.21.15.28:0/3939442349 wait complete. 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.821+0000 7f0e298e0b80 1 -- start start 2024-04-26T01:42:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.821+0000 7f0e298e0b80 1 Processor -- start 2024-04-26T01:42:43.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c5db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c5b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55da53c60fc0 con 0x55da53c53660 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55da53b5f960 con 0x55da53c5e4d0 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55da53c61140 con 0x55da53c52d40 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e298e0b80 10 client.0 _ll_register_callbacks cb 0x55da53b3e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e17400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c5db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e17400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46468/0 (socket says 172.21.15.28:46468) 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c5db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51156/0 (socket says 172.21.15.28:51156) 2024-04-26T01:42:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e17400700 1 -- 172.21.15.28:0/4023724849 learned_addr learned my addr 172.21.15.28:0/4023724849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.822+0000 7f0e1d000700 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c5b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e17e00700 1 -- 172.21.15.28:0/4023724849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 msgr2=0x55da53c5b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e17e00700 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c5b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e17e00700 1 -- 172.21.15.28:0/4023724849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 msgr2=0x55da53c5aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e17e00700 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e17e00700 1 -- 172.21.15.28:0/4023724849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da53af39d0 con 0x55da53c52d40 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e17400700 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e17e00700 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c5db70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009cd0 tx=0x7f0e08019ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4252a60177025b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e02000700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.823+0000 7f0e02000700 1 -- 172.21.15.28:0/4023724849 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08029020 con 0x55da53c52d40 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.824+0000 7f0e02000700 1 -- 172.21.15.28:0/4023724849 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f0e08025680 con 0x55da53c52d40 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.824+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4023724849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e080054d0 con 0x55da53c52d40 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.824+0000 7f0e02000700 1 -- 172.21.15.28:0/4023724849 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0e08034050 con 0x55da53c52d40 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.824+0000 7f0e298e0b80 10 client.25423 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.824+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4023724849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da53b3cbf0 con 0x55da53c52d40 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.824+0000 7f0e02000700 1 -- 172.21.15.28:0/4023724849 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e080054d0 con 0x55da53c52d40 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.825+0000 7f0e02000700 1 -- 172.21.15.28:0/4023724849 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f0e080236f0 con 0x55da53c52d40 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.825+0000 7f0e298e0b80 10 client.25423 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.825+0000 7f0e298e0b80 10 client.25423 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.825+0000 7f0e298e0b80 20 client.25423 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.825+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55da53c6dc20 0x55da53c70010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.825+0000 7f0e298e0b80 4 client.25423 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.825+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4023724849 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da53bb3780 con 0x55da53c6dc20 2024-04-26T01:42:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.825+0000 7f0e17400700 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55da53c6dc20 0x55da53c70010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.826+0000 7f0e17400700 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55da53c6dc20 0x55da53c70010 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70bba70e9cbc7192 server_cookie=b19aeff11f947208 in_seq=0 out_seq=0 2024-04-26T01:42:43.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.826+0000 7f0e02000700 10 client.25423 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:43.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.826+0000 7f0e02000700 1 -- 172.21.15.28:0/4023724849 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da53bb3780 con 0x55da53c6dc20 2024-04-26T01:42:43.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.826+0000 7f0e02000700 10 client.25423 handle_command_reply: tid=0 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.826+0000 7f0e298e0b80 1 client.25423 shutdown 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4023724849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 msgr2=0x55da53c5db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c5db70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009cd0 tx=0x7f0e08019ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4023724849 shutdown_connections 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c5b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55da53c6dc20 0x55da53c70010 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c52d40 0x55da53c5db70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4023724849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c5aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e02000700 0 client.25423 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4023724849 >> 172.21.15.28:0/4023724849 conn(0x55da53b82050 msgr2=0x55da53c4ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4023724849 shutdown_connections 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4023724849 wait complete. 2024-04-26T01:42:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.827+0000 7f0e298e0b80 20 client.25423 trim_cache size 0 max 16384 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.849+0000 7f0e298e0b80 1 Processor -- start 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.849+0000 7f0e298e0b80 1 -- start start 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c52d40 0x55da53c69ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c4eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c5e4d0 0x55da53c4f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181afbf0 con 0x55da53c53660 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181ad450 con 0x55da53c52d40 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181ab2a0 con 0x55da53c5e4d0 2024-04-26T01:42:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c52d40 0x55da53c69ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e1d000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c5e4d0 0x55da53c4f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c52d40 0x55da53c69ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35726/0 (socket says 172.21.15.28:35726) 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e1d000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c5e4d0 0x55da53c4f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51160/0 (socket says 172.21.15.28:51160) 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e17e00700 1 -- 172.21.15.28:0/4049171816 learned_addr learned my addr 172.21.15.28:0/4049171816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e17400700 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c4eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e1d000700 1 -- 172.21.15.28:0/4049171816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c52d40 msgr2=0x55da53c69ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e1d000700 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c52d40 0x55da53c69ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e1d000700 1 -- 172.21.15.28:0/4049171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 msgr2=0x55da53c4eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.850+0000 7f0e1d000700 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c4eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.851+0000 7f0e1d000700 1 -- 172.21.15.28:0/4049171816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da53c54bb0 con 0x55da53c5e4d0 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.851+0000 7f0e17e00700 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c52d40 0x55da53c69ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.851+0000 7f0e17400700 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c4eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:42:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.851+0000 7f0e1d000700 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c5e4d0 0x55da53c4f050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009fd0 tx=0x7f0e1001f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9642857e798cb7bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.851+0000 7f0e0cc00700 1 -- 172.21.15.28:0/4049171816 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e100076e0 con 0x55da53c5e4d0 2024-04-26T01:42:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.851+0000 7f0e0cc00700 1 -- 172.21.15.28:0/4049171816 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e10007d30 con 0x55da53c5e4d0 2024-04-26T01:42:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.851+0000 7f0e0cc00700 1 -- 172.21.15.28:0/4049171816 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e100058c0 con 0x55da53c5e4d0 2024-04-26T01:42:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4049171816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c5e4d0 msgr2=0x55da53c4f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c5e4d0 0x55da53c4f050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009fd0 tx=0x7f0e1001f000 comp rx=0 tx=0).stop 2024-04-26T01:42:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4049171816 shutdown_connections 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c5e4d0 0x55da53c4f050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c52d40 0x55da53c69ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/4049171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c53660 0x55da53c4eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4049171816 >> 172.21.15.28:0/4049171816 conn(0x55da53b82050 msgr2=0x55da53c4e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4049171816 shutdown_connections 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.852+0000 7f0e298e0b80 1 -- 172.21.15.28:0/4049171816 wait complete. 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.853+0000 7f0e298e0b80 1 -- start start 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.853+0000 7f0e298e0b80 1 Processor -- start 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c52d40 0x55da53c633a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c53660 0x55da53c638e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e298e0b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c63e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181ab910 con 0x55da53c52d40 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181acd80 con 0x55da53c5e4d0 2024-04-26T01:42:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e298e0b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181abfe0 con 0x55da53c53660 2024-04-26T01:42:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e298e0b80 10 client.0 _ll_register_callbacks cb 0x55da53b3e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c52d40 0x55da53c633a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e1d000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c63e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e17e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c52d40 0x55da53c633a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46498/0 (socket says 172.21.15.28:46498) 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e17e00700 1 -- 172.21.15.28:0/682143822 learned_addr learned my addr 172.21.15.28:0/682143822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e1d000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c63e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35728/0 (socket says 172.21.15.28:35728) 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e17400700 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c53660 0x55da53c638e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e1d000700 1 -- 172.21.15.28:0/682143822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c53660 msgr2=0x55da53c638e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e1d000700 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c53660 0x55da53c638e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e1d000700 1 -- 172.21.15.28:0/682143822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c52d40 msgr2=0x55da53c633a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e1d000700 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c52d40 0x55da53c633a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.854+0000 7f0e1d000700 1 -- 172.21.15.28:0/682143822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da53c705a0 con 0x55da53c5e4d0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.855+0000 7f0e1d000700 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c63e20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000d010 tx=0x7f0e10005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b9ab6ab26619b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.855+0000 7f0e02000700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.855+0000 7f0e02000700 1 -- 172.21.15.28:0/682143822 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e10005360 con 0x55da53c5e4d0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.855+0000 7f0e298e0b80 1 -- 172.21.15.28:0/682143822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e10007d30 con 0x55da53c5e4d0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.855+0000 7f0e02000700 1 -- 172.21.15.28:0/682143822 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f0e100074a0 con 0x55da53c5e4d0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.855+0000 7f0e02000700 1 -- 172.21.15.28:0/682143822 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0e10005a10 con 0x55da53c5e4d0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e298e0b80 10 client.25391 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e298e0b80 1 -- 172.21.15.28:0/682143822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da53cddb30 con 0x55da53c5e4d0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e02000700 1 -- 172.21.15.28:0/682143822 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e10007d30 con 0x55da53c5e4d0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e02000700 1 -- 172.21.15.28:0/682143822 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f0e10011070 con 0x55da53c5e4d0 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e298e0b80 10 client.25391 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e298e0b80 10 client.25391 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e298e0b80 20 client.25391 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55da53cde220 0x55da53ce0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e298e0b80 4 client.25391 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b141a1f", "release": true, "await": true}] 2024-04-26T01:42:43.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.856+0000 7f0e298e0b80 1 -- 172.21.15.28:0/682143822 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b141a1f", "release": true, "await": true}) -- 0x55da53bad820 con 0x55da53cde220 2024-04-26T01:42:43.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.857+0000 7f0e17e00700 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55da53cde220 0x55da53ce0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.857+0000 7f0e17e00700 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55da53cde220 0x55da53ce0640 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cdadba015aaf6c server_cookie=13e411fd752241d5 in_seq=0 out_seq=0 2024-04-26T01:42:43.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.857+0000 7f0e02000700 10 client.25391 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:43.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.866+0000 7f0e02000700 1 -- 172.21.15.28:0/682143822 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+587 (crc 0 0 0) 0x55da53bad820 con 0x55da53cde220 2024-04-26T01:42:43.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.866+0000 7f0e02000700 10 client.25391 handle_command_reply: tid=0 2024-04-26T01:42:43.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.866+0000 7f0e298e0b80 1 client.25391 shutdown 2024-04-26T01:42:43.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 -- 172.21.15.28:0/682143822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 msgr2=0x55da53c63e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c63e20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000d010 tx=0x7f0e10005770 comp rx=0 tx=0).stop 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 -- 172.21.15.28:0/682143822 shutdown_connections 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55da53c5e4d0 0x55da53c63e20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da53c52d40 0x55da53c633a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55da53cde220 0x55da53ce0640 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 --2- 172.21.15.28:0/682143822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da53c53660 0x55da53c638e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 -- 172.21.15.28:0/682143822 >> 172.21.15.28:0/682143822 conn(0x55da53b82050 msgr2=0x55da53c4ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e02000700 0 client.25391 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 -- 172.21.15.28:0/682143822 shutdown_connections 2024-04-26T01:42:43.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.867+0000 7f0e298e0b80 1 -- 172.21.15.28:0/682143822 wait complete. 2024-04-26T01:42:43.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.868+0000 7f0e298e0b80 20 client.25391 trim_cache size 0 max 16384 2024-04-26T01:42:43.865 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:42:43.865 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 28, 2024-04-26T01:42:43.865 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:42:43.865 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 4, 2024-04-26T01:42:43.865 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-26T01:42:43.865 INFO:teuthology.orchestra.run.smithi028.stdout: "b141a1f": { 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 4, 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 60.0, 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 85.6, 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:42:43.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-26T01:42:43.867 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:42:43.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:43.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:43.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:43.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:43.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:42:43.867 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.868+0000 7f0e1f000700 1 -- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0dfc0a3db0 msgr2=0x7f0dfc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.868+0000 7f0e1f000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0dfc0a3db0 0x7f0dfc0a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55da53c54710 tx=0x7f0e1000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.868+0000 7f0e1f000700 1 -- 172.21.15.28:0/705800181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e18103450 msgr2=0x7f0e18199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.868+0000 7f0e1f000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e18103450 0x7f0e18199670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400e9f0 tx=0x7f0e0400ed00 comp rx=0 tx=0).stop 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.869+0000 7f0e1f000700 1 -- 172.21.15.28:0/705800181 shutdown_connections 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.869+0000 7f0e1f000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0dfc0a3db0 0x7f0dfc0a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.869+0000 7f0e1f000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e18108a50 0x7f0e1819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.869+0000 7f0e1f000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e18102a90 0x7f0e18199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.869+0000 7f0e1f000700 1 --2- 172.21.15.28:0/705800181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e18103450 0x7f0e18199670 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.869+0000 7f0e1f000700 1 -- 172.21.15.28:0/705800181 >> 172.21.15.28:0/705800181 conn(0x7f0e18079ba0 msgr2=0x7f0e18101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.869+0000 7f0e1f000700 1 -- 172.21.15.28:0/705800181 shutdown_connections 2024-04-26T01:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:42:43.869+0000 7f0e1f000700 1 -- 172.21.15.28:0/705800181 wait complete. 2024-04-26T01:42:45.050 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b141a1f', seconds elapsed: 44.8 2024-04-26T01:42:45.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:45 smithi028 ceph-mon[28247]: pgmap v56: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 24 op/s 2024-04-26T01:42:45.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:45 smithi053 ceph-mon[32593]: pgmap v56: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 24 op/s 2024-04-26T01:42:45.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:45 smithi169 ceph-mon[32851]: pgmap v56: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 24 op/s 2024-04-26T01:42:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 50.0 s, 1533.8 tps, lat 477.480 ms stddev 3891.735 2024-04-26T01:42:46.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.995+0000 7fa129400640 1 -- 172.21.15.28:0/1226797188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa124106100 msgr2=0x7fa124106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.995+0000 7fa129400640 1 --2- 172.21.15.28:0/1226797188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa124106100 0x7fa124106590 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009ca0 tx=0x7fa11c02a960 comp rx=0 tx=0).stop 2024-04-26T01:42:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.995+0000 7fa129400640 1 -- 172.21.15.28:0/1226797188 shutdown_connections 2024-04-26T01:42:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.995+0000 7fa129400640 1 --2- 172.21.15.28:0/1226797188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa124106100 0x7fa124106590 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.995+0000 7fa129400640 1 --2- 172.21.15.28:0/1226797188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa124101620 0x7fa124105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.995+0000 7fa129400640 1 --2- 172.21.15.28:0/1226797188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa124100d00 0x7fa1241010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.995+0000 7fa129400640 1 -- 172.21.15.28:0/1226797188 >> 172.21.15.28:0/1226797188 conn(0x7fa1240fc770 msgr2=0x7fa1240feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.995+0000 7fa129400640 1 -- 172.21.15.28:0/1226797188 shutdown_connections 2024-04-26T01:42:46.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.996+0000 7fa129400640 1 -- 172.21.15.28:0/1226797188 wait complete. 2024-04-26T01:42:46.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.996+0000 7fa129400640 1 Processor -- start 2024-04-26T01:42:46.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.997+0000 7fa129400640 1 -- start start 2024-04-26T01:42:46.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.997+0000 7fa129400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa124100d00 0x7fa1241a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.997+0000 7fa129400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa124101620 0x7fa1241a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.997+0000 7fa129400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa124106100 0x7fa12419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.997+0000 7fa129400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa124118640 con 0x7fa124101620 2024-04-26T01:42:46.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.997+0000 7fa129400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa1241184c0 con 0x7fa124100d00 2024-04-26T01:42:46.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.997+0000 7fa129400640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa1241187c0 con 0x7fa124106100 2024-04-26T01:42:46.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.998+0000 7fa123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa124101620 0x7fa1241a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.998+0000 7fa123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa124101620 0x7fa1241a4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46516/0 (socket says 172.21.15.28:46516) 2024-04-26T01:42:46.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.998+0000 7fa123400640 1 -- 172.21.15.28:0/1468517106 learned_addr learned my addr 172.21.15.28:0/1468517106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.998+0000 7fa123e00640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa124100d00 0x7fa1241a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.998+0000 7fa128a00640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa124106100 0x7fa12419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.999+0000 7fa128a00640 1 -- 172.21.15.28:0/1468517106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa124100d00 msgr2=0x7fa1241a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.999+0000 7fa128a00640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa124100d00 0x7fa1241a4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.999+0000 7fa128a00640 1 -- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa124101620 msgr2=0x7fa1241a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.999+0000 7fa128a00640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa124101620 0x7fa1241a4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.999+0000 7fa128a00640 1 -- 172.21.15.28:0/1468517106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12419f160 con 0x7fa124106100 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.999+0000 7fa128a00640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa124106100 0x7fa12419e9e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa11c02ae80 tx=0x7fa11c003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.999+0000 7fa120c00640 1 -- 172.21.15.28:0/1468517106 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c034070 con 0x7fa124106100 2024-04-26T01:42:46.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:46.999+0000 7fa129400640 1 -- 172.21.15.28:0/1468517106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12419f3f0 con 0x7fa124106100 2024-04-26T01:42:46.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.000+0000 7fa129400640 1 -- 172.21.15.28:0/1468517106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1241ab6f0 con 0x7fa124106100 2024-04-26T01:42:46.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.000+0000 7fa120c00640 1 -- 172.21.15.28:0/1468517106 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c009870 con 0x7fa124106100 2024-04-26T01:42:46.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.000+0000 7fa120c00640 1 -- 172.21.15.28:0/1468517106 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c004670 con 0x7fa124106100 2024-04-26T01:42:47.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.005+0000 7fa120c00640 1 -- 172.21.15.28:0/1468517106 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa11c037020 con 0x7fa124106100 2024-04-26T01:42:47.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.006+0000 7fa12d77d740 1 Processor -- start 2024-04-26T01:42:47.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.007+0000 7fa12d77d740 1 -- start start 2024-04-26T01:42:47.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.007+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8c9b0 0x5609ced8cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.007+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8d2d0 0x5609ced97650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.007+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced97b90 0x5609ced99f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.007+0000 7fa12d77d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5609cecf2900 con 0x5609ced8c9b0 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.007+0000 7fa12d77d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5609cec98ef0 con 0x5609ced8d2d0 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.007+0000 7fa12d77d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5609cecee8c0 con 0x5609ced97b90 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8c9b0 0x5609ced8cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8d2d0 0x5609ced97650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8d2d0 0x5609ced97650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35772/0 (socket says 172.21.15.28:35772) 2024-04-26T01:42:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123e00640 1 -- 172.21.15.28:0/481563001 learned_addr learned my addr 172.21.15.28:0/481563001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:47.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123400640 1 --2- 172.21.15.28:0/481563001 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced97b90 0x5609ced99f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123400640 1 -- 172.21.15.28:0/481563001 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced97b90 msgr2=0x5609ced99f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:47.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123400640 1 --2- 172.21.15.28:0/481563001 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced97b90 0x5609ced99f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123400640 1 -- 172.21.15.28:0/481563001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8d2d0 msgr2=0x5609ced97650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:47.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.008+0000 7fa123400640 1 --2- 172.21.15.28:0/481563001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8d2d0 0x5609ced97650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.009+0000 7fa123400640 1 -- 172.21.15.28:0/481563001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609cea05210 con 0x5609ced8c9b0 2024-04-26T01:42:47.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.009+0000 7fa123400640 1 --2- 172.21.15.28:0/481563001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8c9b0 0x5609ced8cd90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x5609ced8a050 tx=0x7fa110009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40501b9915488dcb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:47.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.009+0000 7fa119200640 1 -- 172.21.15.28:0/481563001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110015730 con 0x5609ced8c9b0 2024-04-26T01:42:47.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.010+0000 7fa119200640 1 -- 172.21.15.28:0/481563001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11001d070 con 0x5609ced8c9b0 2024-04-26T01:42:47.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.010+0000 7fa119200640 1 -- 172.21.15.28:0/481563001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110018990 con 0x5609ced8c9b0 2024-04-26T01:42:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.010+0000 7fa12d77d740 1 -- 172.21.15.28:0/481563001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8c9b0 msgr2=0x5609ced8cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.010+0000 7fa12d77d740 1 --2- 172.21.15.28:0/481563001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8c9b0 0x5609ced8cd90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x5609ced8a050 tx=0x7fa110009510 comp rx=0 tx=0).stop 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa120c00640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa1080a3d90 0x7fa1080a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa12d77d740 1 -- 172.21.15.28:0/481563001 shutdown_connections 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa12d77d740 1 --2- 172.21.15.28:0/481563001 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced97b90 0x5609ced99f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa12d77d740 1 --2- 172.21.15.28:0/481563001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8d2d0 0x5609ced97650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa12d77d740 1 --2- 172.21.15.28:0/481563001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8c9b0 0x5609ced8cd90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa12d77d740 1 -- 172.21.15.28:0/481563001 >> 172.21.15.28:0/481563001 conn(0x5609ced88ec0 msgr2=0x5609ced892a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa120c00640 1 -- 172.21.15.28:0/1468517106 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fa11c0f1700 con 0x7fa124106100 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa123400640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa1080a3d90 0x7fa1080a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa12d77d740 1 -- 172.21.15.28:0/481563001 shutdown_connections 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.011+0000 7fa12d77d740 1 -- 172.21.15.28:0/481563001 wait complete. 2024-04-26T01:42:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.012+0000 7fa12d77d740 1 -- start start 2024-04-26T01:42:47.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.012+0000 7fa12d77d740 1 Processor -- start 2024-04-26T01:42:47.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.012+0000 7fa123400640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa1080a3d90 0x7fa1080a6250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x5609ced8a050 tx=0x7fa110009fd0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8c9b0 0x5609ced94a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8d2d0 0x5609ced94f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced97b90 0x5609ced954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa12d77d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5609ced9a760 con 0x5609ced97b90 2024-04-26T01:42:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa12d77d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5609cecac310 con 0x5609ced8c9b0 2024-04-26T01:42:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa12d77d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5609ced9a8e0 con 0x5609ced8d2d0 2024-04-26T01:42:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa12d77d740 10 client.0 _ll_register_callbacks cb 0x5609cec7acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa123400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8d2d0 0x5609ced94f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa123400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8d2d0 0x5609ced94f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51194/0 (socket says 172.21.15.28:51194) 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.013+0000 7fa123400640 1 -- 172.21.15.28:0/480912566 learned_addr learned my addr 172.21.15.28:0/480912566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.014+0000 7fa128a00640 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced97b90 0x5609ced954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.014+0000 7fa123e00640 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8c9b0 0x5609ced94a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.015+0000 7fa128a00640 1 -- 172.21.15.28:0/480912566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8d2d0 msgr2=0x5609ced94f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.015+0000 7fa128a00640 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8d2d0 0x5609ced94f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.015+0000 7fa128a00640 1 -- 172.21.15.28:0/480912566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8c9b0 msgr2=0x5609ced94a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.015+0000 7fa128a00640 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8c9b0 0x5609ced94a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.015+0000 7fa128a00640 1 -- 172.21.15.28:0/480912566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609cead1620 con 0x5609ced97b90 2024-04-26T01:42:47.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.015+0000 7fa123400640 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8d2d0 0x5609ced94f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:47.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.015+0000 7fa128a00640 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced97b90 0x5609ced954b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0eee60 tx=0x7fa11c0be020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0035ae6187eba38 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:47.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.015+0000 7fa10d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:42:47.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.016+0000 7fa10d600640 1 -- 172.21.15.28:0/480912566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0c2a70 con 0x5609ced97b90 2024-04-26T01:42:47.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.016+0000 7fa12d77d740 1 -- 172.21.15.28:0/480912566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa11001d070 con 0x5609ced97b90 2024-04-26T01:42:47.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.016+0000 7fa10d600640 1 -- 172.21.15.28:0/480912566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fa11c0c6ea0 con 0x5609ced97b90 2024-04-26T01:42:47.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.016+0000 7fa10d600640 1 -- 172.21.15.28:0/480912566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa11c0c7330 con 0x5609ced97b90 2024-04-26T01:42:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.016+0000 7fa12d77d740 10 client.15273 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.016+0000 7fa12d77d740 1 -- 172.21.15.28:0/480912566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609ced9a760 con 0x5609ced97b90 2024-04-26T01:42:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.017+0000 7fa10d600640 1 -- 172.21.15.28:0/480912566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11001d070 con 0x5609ced97b90 2024-04-26T01:42:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.018+0000 7fa10d600640 1 -- 172.21.15.28:0/480912566 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fa11c04b070 con 0x5609ced97b90 2024-04-26T01:42:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.018+0000 7fa12d77d740 10 client.15273 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.018+0000 7fa12d77d740 10 client.15273 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:42:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.018+0000 7fa12d77d740 20 client.15273 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.018+0000 7fa12d77d740 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5609cedb8e50 0x5609cedbb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.018+0000 7fa12d77d740 4 client.15273 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:42:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.018+0000 7fa12d77d740 1 -- 172.21.15.28:0/480912566 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609ced068b0 con 0x5609cedb8e50 2024-04-26T01:42:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.019+0000 7fa123e00640 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5609cedb8e50 0x5609cedbb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.021+0000 7fa123e00640 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5609cedb8e50 0x5609cedbb2d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c621c3b68cec51fa server_cookie=440380b7f3e2a98d in_seq=0 out_seq=0 2024-04-26T01:42:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.021+0000 7fa10d600640 10 client.15273 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.022+0000 7fa10d600640 1 -- 172.21.15.28:0/480912566 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609ced068b0 con 0x5609cedb8e50 2024-04-26T01:42:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.022+0000 7fa10d600640 10 client.15273 handle_command_reply: tid=0 2024-04-26T01:42:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 client.15273 shutdown 2024-04-26T01:42:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 -- 172.21.15.28:0/480912566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced97b90 msgr2=0x5609ced954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced97b90 0x5609ced954b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0eee60 tx=0x7fa11c0be020 comp rx=0 tx=0).stop 2024-04-26T01:42:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 -- 172.21.15.28:0/480912566 shutdown_connections 2024-04-26T01:42:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5609cedb8e50 0x5609cedbb2d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced97b90 0x5609ced954b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8d2d0 0x5609ced94f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa10d600640 0 client.15273 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 --2- 172.21.15.28:0/480912566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced8c9b0 0x5609ced94a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.023+0000 7fa12d77d740 1 -- 172.21.15.28:0/480912566 >> 172.21.15.28:0/480912566 conn(0x5609ced88ec0 msgr2=0x5609ced8ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.024+0000 7fa12d77d740 1 -- 172.21.15.28:0/480912566 shutdown_connections 2024-04-26T01:42:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.024+0000 7fa12d77d740 1 -- 172.21.15.28:0/480912566 wait complete. 2024-04-26T01:42:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.024+0000 7fa12d77d740 20 client.15273 trim_cache size 0 max 16384 2024-04-26T01:42:47.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.051+0000 7fa12d77d740 1 Processor -- start 2024-04-26T01:42:47.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.051+0000 7fa12d77d740 1 -- start start 2024-04-26T01:42:47.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 0x5609ced892a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609ced952d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedc0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa12d77d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa104010f60 con 0x5609ced8d2d0 2024-04-26T01:42:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa12d77d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa10400e800 con 0x5609ced97b90 2024-04-26T01:42:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa12d77d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa10400c5f0 con 0x5609ced8c9b0 2024-04-26T01:42:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609ced952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609ced952d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46554/0 (socket says 172.21.15.28:46554) 2024-04-26T01:42:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa123400640 1 -- 172.21.15.28:0/1769562952 learned_addr learned my addr 172.21.15.28:0/1769562952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.052+0000 7fa128a00640 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedc0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.053+0000 7fa123e00640 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 0x5609ced892a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.053+0000 7fa128a00640 1 -- 172.21.15.28:0/1769562952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 msgr2=0x5609ced892a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.053+0000 7fa128a00640 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 0x5609ced892a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.053+0000 7fa128a00640 1 -- 172.21.15.28:0/1769562952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 msgr2=0x5609ced952d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.053+0000 7fa128a00640 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609ced952d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.053+0000 7fa128a00640 1 -- 172.21.15.28:0/1769562952 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609ced8f820 con 0x5609ced97b90 2024-04-26T01:42:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.054+0000 7fa128a00640 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedc0b50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0bc200 tx=0x7fa11c0bcbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e855a2f8fb2a476e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.054+0000 7fa10fe00640 1 -- 172.21.15.28:0/1769562952 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0a2080 con 0x5609ced97b90 2024-04-26T01:42:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.054+0000 7fa10fe00640 1 -- 172.21.15.28:0/1769562952 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c030e50 con 0x5609ced97b90 2024-04-26T01:42:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.054+0000 7fa10fe00640 1 -- 172.21.15.28:0/1769562952 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0c7330 con 0x5609ced97b90 2024-04-26T01:42:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.055+0000 7fa12d77d740 1 -- 172.21.15.28:0/1769562952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 msgr2=0x5609cedc0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.055+0000 7fa12d77d740 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedc0b50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0bc200 tx=0x7fa11c0bcbd0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.055+0000 7fa12d77d740 1 -- 172.21.15.28:0/1769562952 shutdown_connections 2024-04-26T01:42:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.055+0000 7fa12d77d740 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedc0b50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.055+0000 7fa12d77d740 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609ced952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.055+0000 7fa12d77d740 1 --2- 172.21.15.28:0/1769562952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 0x5609ced892a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.055+0000 7fa12d77d740 1 -- 172.21.15.28:0/1769562952 >> 172.21.15.28:0/1769562952 conn(0x5609ced88ec0 msgr2=0x5609ced88020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.056+0000 7fa12d77d740 1 -- 172.21.15.28:0/1769562952 shutdown_connections 2024-04-26T01:42:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.056+0000 7fa12d77d740 1 -- 172.21.15.28:0/1769562952 wait complete. 2024-04-26T01:42:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.056+0000 7fa12d77d740 1 -- start start 2024-04-26T01:42:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.056+0000 7fa12d77d740 1 Processor -- start 2024-04-26T01:42:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 0x5609cedadbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609cedae130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa12d77d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedae690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa12d77d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa10400ccc0 con 0x5609ced8d2d0 2024-04-26T01:42:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa12d77d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa10400e130 con 0x5609ced97b90 2024-04-26T01:42:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa128a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa128a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedae690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35800/0 (socket says 172.21.15.28:35800) 2024-04-26T01:42:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa128a00640 1 -- 172.21.15.28:0/2054545251 learned_addr learned my addr 172.21.15.28:0/2054545251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:42:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa123400640 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609cedae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa10400d390 con 0x5609ced8c9b0 2024-04-26T01:42:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa12d77d740 10 client.0 _ll_register_callbacks cb 0x7fa11c0be490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:42:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.058+0000 7fa123e00640 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 0x5609cedadbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa128a00640 1 -- 172.21.15.28:0/2054545251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 msgr2=0x5609cedadbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa128a00640 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 0x5609cedadbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa128a00640 1 -- 172.21.15.28:0/2054545251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 msgr2=0x5609cedae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa128a00640 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609cedae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa128a00640 1 -- 172.21.15.28:0/2054545251 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609cee1d630 con 0x5609ced97b90 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa123400640 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609cedae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa128a00640 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedae690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa11c004220 tx=0x7fa11c009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a015399922c0ff9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa10d600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.059+0000 7fa10d600640 1 -- 172.21.15.28:0/2054545251 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0a2070 con 0x5609ced97b90 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.060+0000 7fa10d600640 1 -- 172.21.15.28:0/2054545251 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fa11c0c6ea0 con 0x5609ced97b90 2024-04-26T01:42:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.060+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa11c030e50 con 0x5609ced97b90 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.061+0000 7fa10d600640 1 -- 172.21.15.28:0/2054545251 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa11c0dd880 con 0x5609ced97b90 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.061+0000 7fa12d77d740 10 client.25409 fetch_fsmap learned FSMap version 28 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.061+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609cee1d630 con 0x5609ced97b90 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.061+0000 7fa10d600640 1 -- 172.21.15.28:0/2054545251 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c030e50 con 0x5609ced97b90 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.061+0000 7fa10d600640 1 -- 172.21.15.28:0/2054545251 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fa11c04b070 con 0x5609ced97b90 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.062+0000 7fa12d77d740 10 client.25409 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.062+0000 7fa12d77d740 10 client.25409 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.062+0000 7fa12d77d740 20 client.25409 populate_metadata read hostname 'smithi028' 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.062+0000 7fa12d77d740 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5609cee1ec90 0x5609cee210b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.062+0000 7fa12d77d740 4 client.25409 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.062+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5609ced00320 con 0x5609cee1ec90 2024-04-26T01:42:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.062+0000 7fa123e00640 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5609cee1ec90 0x5609cee210b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.063+0000 7fa123e00640 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5609cee1ec90 0x5609cee210b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ced5948314bf5b84 server_cookie=79cf4f0c4b62044e in_seq=0 out_seq=0 2024-04-26T01:42:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.063+0000 7fa10d600640 10 client.25409 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:47.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.078+0000 7fa10d600640 1 -- 172.21.15.28:0/2054545251 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5609ced00320 con 0x5609cee1ec90 2024-04-26T01:42:47.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.078+0000 7fa10d600640 10 client.25409 handle_command_reply: tid=0 2024-04-26T01:42:47.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.078+0000 7fa12d77d740 1 client.25409 shutdown 2024-04-26T01:42:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.078+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 msgr2=0x5609cedae690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.078+0000 7fa12d77d740 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedae690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa11c004220 tx=0x7fa11c009a20 comp rx=0 tx=0).stop 2024-04-26T01:42:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.079+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 shutdown_connections 2024-04-26T01:42:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.079+0000 7fa12d77d740 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5609cee1ec90 0x5609cee210b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.079+0000 7fa12d77d740 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5609ced97b90 0x5609cedae690 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.079+0000 7fa12d77d740 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609ced8d2d0 0x5609cedae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.079+0000 7fa12d77d740 1 --2- 172.21.15.28:0/2054545251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609ced8c9b0 0x5609cedadbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.079+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 >> 172.21.15.28:0/2054545251 conn(0x5609ced88ec0 msgr2=0x5609ced88020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.079+0000 7fa10d600640 0 client.25409 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:42:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.079+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 shutdown_connections 2024-04-26T01:42:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.080+0000 7fa12d77d740 1 -- 172.21.15.28:0/2054545251 wait complete. 2024-04-26T01:42:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.080+0000 7fa12d77d740 20 client.25409 trim_cache size 0 max 16384 2024-04-26T01:42:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:42:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:42:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:42:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:42:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.081+0000 7fa129400640 1 -- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa1080a3d90 msgr2=0x7fa1080a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.081+0000 7fa129400640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa1080a3d90 0x7fa1080a6250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x5609ced8a050 tx=0x7fa110009fd0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.081+0000 7fa129400640 1 -- 172.21.15.28:0/1468517106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa124106100 msgr2=0x7fa12419e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.081+0000 7fa129400640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa124106100 0x7fa12419e9e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa11c02ae80 tx=0x7fa11c003ba0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.082+0000 7fa129400640 1 -- 172.21.15.28:0/1468517106 shutdown_connections 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.082+0000 7fa129400640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa1080a3d90 0x7fa1080a6250 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.082+0000 7fa129400640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa124106100 0x7fa12419e9e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.082+0000 7fa129400640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa124101620 0x7fa1241a4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.082+0000 7fa129400640 1 --2- 172.21.15.28:0/1468517106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa124100d00 0x7fa1241a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.082+0000 7fa129400640 1 -- 172.21.15.28:0/1468517106 >> 172.21.15.28:0/1468517106 conn(0x7fa1240fc770 msgr2=0x7fa1240fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.082+0000 7fa129400640 1 -- 172.21.15.28:0/1468517106 shutdown_connections 2024-04-26T01:42:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:42:47.082+0000 7fa129400640 1 -- 172.21.15.28:0/1468517106 wait complete. 2024-04-26T01:42:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:47 smithi053 ceph-mon[32593]: pgmap v57: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 18 op/s 2024-04-26T01:42:47.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:47 smithi028 ceph-mon[28247]: pgmap v57: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 18 op/s 2024-04-26T01:42:47.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:47 smithi169 ceph-mon[32851]: pgmap v57: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 18 op/s 2024-04-26T01:42:49.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:49 smithi053 ceph-mon[32593]: pgmap v58: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 18 op/s 2024-04-26T01:42:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:49 smithi028 ceph-mon[28247]: pgmap v58: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 18 op/s 2024-04-26T01:42:49.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:49 smithi169 ceph-mon[32851]: pgmap v58: 97 pgs: 97 active+clean; 1016 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 18 op/s 2024-04-26T01:42:51.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 55.0 s, 3188.4 tps, lat 31.283 ms stddev 31.200 2024-04-26T01:42:51.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:51 smithi053 ceph-mon[32593]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-04-26T01:42:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:51 smithi028 ceph-mon[28247]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-04-26T01:42:51.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:51 smithi169 ceph-mon[32851]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-04-26T01:42:53.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:53 smithi053 ceph-mon[32593]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 4.2 MiB/s wr, 414 op/s 2024-04-26T01:42:53.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:53 smithi028 ceph-mon[28247]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 4.2 MiB/s wr, 414 op/s 2024-04-26T01:42:53.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:53 smithi169 ceph-mon[32851]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 4.2 MiB/s wr, 414 op/s 2024-04-26T01:42:55.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:55 smithi053 ceph-mon[32593]: pgmap v61: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.4 MiB/s wr, 705 op/s 2024-04-26T01:42:55.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:55 smithi028 ceph-mon[28247]: pgmap v61: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.4 MiB/s wr, 705 op/s 2024-04-26T01:42:55.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:55 smithi169 ceph-mon[32851]: pgmap v61: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.4 MiB/s wr, 705 op/s 2024-04-26T01:42:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 60.0 s, 3101.4 tps, lat 32.263 ms stddev 36.986 2024-04-26T01:42:57.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:57 smithi028 ceph-mon[28247]: pgmap v62: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 8.3 MiB/s wr, 690 op/s 2024-04-26T01:42:57.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:57 smithi169 ceph-mon[32851]: pgmap v62: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 8.3 MiB/s wr, 690 op/s 2024-04-26T01:42:57.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:57 smithi053 ceph-mon[32593]: pgmap v62: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 8.3 MiB/s wr, 690 op/s 2024-04-26T01:42:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:42:59 smithi028 ceph-mon[28247]: pgmap v63: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 8.3 MiB/s wr, 690 op/s 2024-04-26T01:42:59.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:42:59 smithi169 ceph-mon[32851]: pgmap v63: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 8.3 MiB/s wr, 690 op/s 2024-04-26T01:42:59.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:42:59 smithi053 ceph-mon[32593]: pgmap v63: 97 pgs: 97 active+clean; 1.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 8.3 MiB/s wr, 690 op/s 2024-04-26T01:43:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.282+0000 7f854b800640 1 -- 172.21.15.28:0/1028442508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 msgr2=0x7f85441047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.282+0000 7f854b800640 1 --2- 172.21.15.28:0/1028442508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 0x7f85441047a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8538009900 tx=0x7f853802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.283+0000 7f854b800640 1 -- 172.21.15.28:0/1028442508 shutdown_connections 2024-04-26T01:43:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.283+0000 7f854b800640 1 --2- 172.21.15.28:0/1028442508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8544109490 0x7f8544111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.283+0000 7f854b800640 1 --2- 172.21.15.28:0/1028442508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8544104d70 0x7f8544108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.283+0000 7f854b800640 1 --2- 172.21.15.28:0/1028442508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 0x7f85441047a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.283+0000 7f854b800640 1 -- 172.21.15.28:0/1028442508 >> 172.21.15.28:0/1028442508 conn(0x7f85440ffc80 msgr2=0x7f85441020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.283+0000 7f854b800640 1 -- 172.21.15.28:0/1028442508 shutdown_connections 2024-04-26T01:43:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.283+0000 7f854b800640 1 -- 172.21.15.28:0/1028442508 wait complete. 2024-04-26T01:43:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.285+0000 7f854b800640 1 Processor -- start 2024-04-26T01:43:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.286+0000 7f854b800640 1 -- start start 2024-04-26T01:43:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.286+0000 7f854b800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 0x7f8544076460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.286+0000 7f854b800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8544109490 0x7f85440769a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.286+0000 7f854b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8544076ee0 0x7f8544072730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.286+0000 7f854b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8544114420 con 0x7f8544076ee0 2024-04-26T01:43:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.286+0000 7f854b800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85441142a0 con 0x7f85441043c0 2024-04-26T01:43:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.286+0000 7f854b800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f85441145a0 con 0x7f8544109490 2024-04-26T01:43:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.286+0000 7f8548e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 0x7f8544076460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8548e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 0x7f8544076460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:44962/0 (socket says 172.21.15.28:44962) 2024-04-26T01:43:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8548e00640 1 -- 172.21.15.28:0/3947894053 learned_addr learned my addr 172.21.15.28:0/3947894053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8543e00640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8544109490 0x7f85440769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8549800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8544076ee0 0x7f8544072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8548e00640 1 -- 172.21.15.28:0/3947894053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8544109490 msgr2=0x7f85440769a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8548e00640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8544109490 0x7f85440769a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8548e00640 1 -- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8544076ee0 msgr2=0x7f8544072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8548e00640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8544076ee0 0x7f8544072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8548e00640 1 -- 172.21.15.28:0/3947894053 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8544072c70 con 0x7f85441043c0 2024-04-26T01:43:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.287+0000 7f8549800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8544076ee0 0x7f8544072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:43:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.288+0000 7f8548e00640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 0x7f8544076460 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8538005ab0 tx=0x7f8538003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.289+0000 7f8541600640 1 -- 172.21.15.28:0/3947894053 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853803f070 con 0x7f85441043c0 2024-04-26T01:43:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.289+0000 7f8541600640 1 -- 172.21.15.28:0/3947894053 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f853803bd20 con 0x7f85441043c0 2024-04-26T01:43:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.289+0000 7f8541600640 1 -- 172.21.15.28:0/3947894053 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538035030 con 0x7f85441043c0 2024-04-26T01:43:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.289+0000 7f854b800640 1 -- 172.21.15.28:0/3947894053 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8544072f00 con 0x7f85441043c0 2024-04-26T01:43:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.289+0000 7f854b800640 1 -- 172.21.15.28:0/3947894053 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85440732e0 con 0x7f85441043c0 2024-04-26T01:43:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.295+0000 7f854e5af740 1 Processor -- start 2024-04-26T01:43:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.296+0000 7f854e5af740 1 -- start start 2024-04-26T01:43:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.296+0000 7f854e5af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d31c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.296+0000 7f854e5af740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 0x562563d3c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.296+0000 7f854e5af740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d3ca00 0x562563d3edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.296+0000 7f854e5af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562563c97900 con 0x562563d31820 2024-04-26T01:43:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.296+0000 7f854e5af740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562563c3def0 con 0x562563d32140 2024-04-26T01:43:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.296+0000 7f854e5af740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562563c938c0 con 0x562563d3ca00 2024-04-26T01:43:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8541600640 1 -- 172.21.15.28:0/3947894053 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8538042070 con 0x7f85441043c0 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8549800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d31c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8549800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d31c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43446/0 (socket says 172.21.15.28:43446) 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8549800640 1 -- 172.21.15.28:0/3747854440 learned_addr learned my addr 172.21.15.28:0/3747854440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8543e00640 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d3ca00 0x562563d3edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8543e00640 1 -- 172.21.15.28:0/3747854440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 msgr2=0x562563d3c4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8543e00640 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 0x562563d3c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8543e00640 1 -- 172.21.15.28:0/3747854440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 msgr2=0x562563d31c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8543e00640 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d31c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8543e00640 1 -- 172.21.15.28:0/3747854440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625639aa210 con 0x562563d3ca00 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.297+0000 7f8549800640 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d31c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.298+0000 7f8543e00640 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d3ca00 0x562563d3edf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8544077c60 tx=0x7f853000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3566c649872da7d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.298+0000 7f8541600640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f852c0a3f20 0x7f852c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.298+0000 7f8536000640 1 -- 172.21.15.28:0/3747854440 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530015070 con 0x562563d3ca00 2024-04-26T01:43:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.298+0000 7f8536000640 1 -- 172.21.15.28:0/3747854440 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85300035f0 con 0x562563d3ca00 2024-04-26T01:43:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.298+0000 7f8536000640 1 -- 172.21.15.28:0/3747854440 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85300116a0 con 0x562563d3ca00 2024-04-26T01:43:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.299+0000 7f8541600640 1 -- 172.21.15.28:0/3947894053 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f85380bac60 con 0x7f85441043c0 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.299+0000 7f854e5af740 1 -- 172.21.15.28:0/3747854440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d3ca00 msgr2=0x562563d3edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.299+0000 7f854e5af740 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d3ca00 0x562563d3edf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8544077c60 tx=0x7f853000f730 comp rx=0 tx=0).stop 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.300+0000 7f8549800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f852c0a3f20 0x7f852c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.300+0000 7f854e5af740 1 -- 172.21.15.28:0/3747854440 shutdown_connections 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.300+0000 7f854e5af740 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d3ca00 0x562563d3edf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.300+0000 7f854e5af740 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 0x562563d3c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.300+0000 7f854e5af740 1 --2- 172.21.15.28:0/3747854440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d31c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.300+0000 7f854e5af740 1 -- 172.21.15.28:0/3747854440 >> 172.21.15.28:0/3747854440 conn(0x562563d2dd30 msgr2=0x562563d2e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.300+0000 7f854e5af740 1 -- 172.21.15.28:0/3747854440 shutdown_connections 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.301+0000 7f854e5af740 1 -- 172.21.15.28:0/3747854440 wait complete. 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.301+0000 7f854e5af740 1 -- start start 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.301+0000 7f854e5af740 1 Processor -- start 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.301+0000 7f8549800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f852c0a3f20 0x7f852c0a63e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f853c009fd0 tx=0x7f853c0098c0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f854e5af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d39980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f854e5af740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d32140 0x562563d39ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f854e5af740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 0x562563d3a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f854e5af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562563d3f630 con 0x562563d31820 2024-04-26T01:43:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f854e5af740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562563c51310 con 0x562563d3ca00 2024-04-26T01:43:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f854e5af740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562563d3f7b0 con 0x562563d32140 2024-04-26T01:43:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f8543e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d32140 0x562563d39ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f8543e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d32140 0x562563d39ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:48740/0 (socket says 172.21.15.28:48740) 2024-04-26T01:43:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f8543e00640 1 -- 172.21.15.28:0/2930514780 learned_addr learned my addr 172.21.15.28:0/2930514780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f8549800640 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 0x562563d3a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.302+0000 7f8548e00640 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d39980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.303+0000 7f8543e00640 1 -- 172.21.15.28:0/2930514780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 msgr2=0x562563d3a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.303+0000 7f8543e00640 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 0x562563d3a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.303+0000 7f8543e00640 1 -- 172.21.15.28:0/2930514780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 msgr2=0x562563d39980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.303+0000 7f8543e00640 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d39980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.303+0000 7f8543e00640 1 -- 172.21.15.28:0/2930514780 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562563a76620 con 0x562563d32140 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.303+0000 7f854e5af740 10 client.0 _ll_register_callbacks cb 0x562563c1fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.303+0000 7f8543e00640 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d32140 0x562563d39ec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8530003f50 tx=0x7f853000bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4d677a8653d4b6d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.303+0000 7f852aa00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.304+0000 7f854e5af740 1 -- 172.21.15.28:0/2930514780 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85300035f0 con 0x562563d32140 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.304+0000 7f852aa00640 1 -- 172.21.15.28:0/2930514780 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530015070 con 0x562563d32140 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.304+0000 7f852aa00640 1 -- 172.21.15.28:0/2930514780 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f85300039b0 con 0x562563d32140 2024-04-26T01:43:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.305+0000 7f852aa00640 1 -- 172.21.15.28:0/2930514780 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f853000b7d0 con 0x562563d32140 2024-04-26T01:43:00.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.305+0000 7f854e5af740 10 client.25453 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:00.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.305+0000 7f854e5af740 1 -- 172.21.15.28:0/2930514780 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({fsmap=28}) -- 0x562563d4b750 con 0x562563d32140 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.306+0000 7f852aa00640 1 -- 172.21.15.28:0/2930514780 <== mon.1 v2:172.21.15.169:3300/0 4 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f8530003fb0 con 0x562563d32140 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.306+0000 7f854e5af740 10 client.25453 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.306+0000 7f854e5af740 10 client.25453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.306+0000 7f854e5af740 20 client.25453 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.306+0000 7f854e5af740 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562563d4caa0 0x562563d4ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.306+0000 7f854e5af740 4 client.25453 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.306+0000 7f854e5af740 1 -- 172.21.15.28:0/2930514780 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562563cab8b0 con 0x562563d4caa0 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.306+0000 7f8548e00640 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562563d4caa0 0x562563d4ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.307+0000 7f8548e00640 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562563d4caa0 0x562563d4ee90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb1259715716b292 server_cookie=2fcb086435fc4d3a in_seq=0 out_seq=0 2024-04-26T01:43:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.307+0000 7f852aa00640 10 client.25453 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.308+0000 7f852aa00640 1 -- 172.21.15.28:0/2930514780 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562563cab8b0 con 0x562563d4caa0 2024-04-26T01:43:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.308+0000 7f852aa00640 10 client.25453 handle_command_reply: tid=0 2024-04-26T01:43:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.308+0000 7f854e5af740 1 client.25453 shutdown 2024-04-26T01:43:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.308+0000 7f854e5af740 1 -- 172.21.15.28:0/2930514780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d32140 msgr2=0x562563d39ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.308+0000 7f854e5af740 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d32140 0x562563d39ec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8530003f50 tx=0x7f853000bea0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.309+0000 7f854e5af740 1 -- 172.21.15.28:0/2930514780 shutdown_connections 2024-04-26T01:43:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.309+0000 7f854e5af740 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562563d4caa0 0x562563d4ee90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.309+0000 7f854e5af740 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 0x562563d3a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.309+0000 7f854e5af740 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d32140 0x562563d39ec0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.309+0000 7f854e5af740 1 --2- 172.21.15.28:0/2930514780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d31820 0x562563d39980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.309+0000 7f854e5af740 1 -- 172.21.15.28:0/2930514780 >> 172.21.15.28:0/2930514780 conn(0x562563d2dd30 msgr2=0x562563d33ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:00.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.309+0000 7f852aa00640 0 client.25453 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:00.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.310+0000 7f854e5af740 1 -- 172.21.15.28:0/2930514780 shutdown_connections 2024-04-26T01:43:00.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.310+0000 7f854e5af740 1 -- 172.21.15.28:0/2930514780 wait complete. 2024-04-26T01:43:00.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.310+0000 7f854e5af740 20 client.25453 trim_cache size 0 max 16384 2024-04-26T01:43:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.337+0000 7f854e5af740 1 Processor -- start 2024-04-26T01:43:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.337+0000 7f854e5af740 1 -- start start 2024-04-26T01:43:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f854e5af740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d484f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f854e5af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d32140 0x562563d341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f854e5af740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 0x562563d54600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f854e5af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85441bf0e0 con 0x562563d32140 2024-04-26T01:43:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f854e5af740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85441bc980 con 0x562563d3ca00 2024-04-26T01:43:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f854e5af740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f85441ba7d0 con 0x562563d31820 2024-04-26T01:43:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f8548e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f8548e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:48744/0 (socket says 172.21.15.28:48744) 2024-04-26T01:43:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f8548e00640 1 -- 172.21.15.28:0/2684289414 learned_addr learned my addr 172.21.15.28:0/2684289414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.338+0000 7f8549800640 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 0x562563d54600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.339+0000 7f8543e00640 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d32140 0x562563d341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.339+0000 7f8548e00640 1 -- 172.21.15.28:0/2684289414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 msgr2=0x562563d54600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.339+0000 7f8548e00640 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 0x562563d54600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.339+0000 7f8548e00640 1 -- 172.21.15.28:0/2684289414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d32140 msgr2=0x562563d341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.340+0000 7f8548e00640 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d32140 0x562563d341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.340+0000 7f8548e00640 1 -- 172.21.15.28:0/2684289414 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562563d2e6b0 con 0x562563d31820 2024-04-26T01:43:00.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.340+0000 7f8543e00640 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d32140 0x562563d341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:00.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.341+0000 7f8548e00640 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d484f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f85380b9e40 tx=0x7f85380c2090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4261e85aca34c459 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.341+0000 7f8535600640 1 -- 172.21.15.28:0/2684289414 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85380b95f0 con 0x562563d31820 2024-04-26T01:43:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.342+0000 7f8535600640 1 -- 172.21.15.28:0/2684289414 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8538049e50 con 0x562563d31820 2024-04-26T01:43:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.342+0000 7f8535600640 1 -- 172.21.15.28:0/2684289414 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85380b9790 con 0x562563d31820 2024-04-26T01:43:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 -- 172.21.15.28:0/2684289414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 msgr2=0x562563d484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d484f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f85380b9e40 tx=0x7f85380c2090 comp rx=0 tx=0).stop 2024-04-26T01:43:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 -- 172.21.15.28:0/2684289414 shutdown_connections 2024-04-26T01:43:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d3ca00 0x562563d54600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d32140 0x562563d341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 --2- 172.21.15.28:0/2684289414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d484f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 -- 172.21.15.28:0/2684289414 >> 172.21.15.28:0/2684289414 conn(0x562563d2dd30 msgr2=0x562563d4eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 -- 172.21.15.28:0/2684289414 shutdown_connections 2024-04-26T01:43:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.343+0000 7f854e5af740 1 -- 172.21.15.28:0/2684289414 wait complete. 2024-04-26T01:43:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.344+0000 7f854e5af740 1 -- start start 2024-04-26T01:43:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.344+0000 7f854e5af740 1 Processor -- start 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f854e5af740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d44a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f854e5af740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 0x562563d44f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f854e5af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d3ca00 0x562563d41fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f854e5af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85441bae40 con 0x562563d3ca00 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f854e5af740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85441bc2b0 con 0x562563d32140 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f854e5af740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f85441bb510 con 0x562563d31820 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f854e5af740 10 client.0 _ll_register_callbacks cb 0x7f8530003d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f8548e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d44a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.346+0000 7f8548e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d44a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:48746/0 (socket says 172.21.15.28:48746) 2024-04-26T01:43:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8543e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 0x562563d44f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8548e00640 1 -- 172.21.15.28:0/469286159 learned_addr learned my addr 172.21.15.28:0/469286159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8549800640 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d3ca00 0x562563d41fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8543e00640 1 -- 172.21.15.28:0/469286159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 msgr2=0x562563d44a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8543e00640 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d44a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8543e00640 1 -- 172.21.15.28:0/469286159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d3ca00 msgr2=0x562563d41fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8543e00640 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d3ca00 0x562563d41fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8543e00640 1 -- 172.21.15.28:0/469286159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562563d427e0 con 0x562563d32140 2024-04-26T01:43:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f8543e00640 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 0x562563d44f90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f853000b970 tx=0x7f853000b4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8e325f44e183314 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.347+0000 7f852aa00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f852aa00640 1 -- 172.21.15.28:0/469286159 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853000be80 con 0x562563d32140 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f852aa00640 1 -- 172.21.15.28:0/469286159 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f853002d450 con 0x562563d32140 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 1 -- 172.21.15.28:0/469286159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8538049e50 con 0x562563d32140 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f852aa00640 1 -- 172.21.15.28:0/469286159 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f853002d650 con 0x562563d32140 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 10 client.25424 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 1 -- 172.21.15.28:0/469286159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562563db2b20 con 0x562563d32140 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f852aa00640 1 -- 172.21.15.28:0/469286159 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8538049e50 con 0x562563d32140 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f852aa00640 1 -- 172.21.15.28:0/469286159 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f853003f450 con 0x562563d32140 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 10 client.25424 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 10 client.25424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 20 client.25424 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562563db32a0 0x562563db56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 4 client.25424 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:43:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.348+0000 7f854e5af740 1 -- 172.21.15.28:0/469286159 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562563ca5320 con 0x562563db32a0 2024-04-26T01:43:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.349+0000 7f8548e00640 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562563db32a0 0x562563db56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.349+0000 7f8548e00640 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562563db32a0 0x562563db56c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=291edcbcade0f177 server_cookie=710dd4b541da192d in_seq=0 out_seq=0 2024-04-26T01:43:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.349+0000 7f852aa00640 10 client.25424 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:00.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.364+0000 7f852aa00640 1 -- 172.21.15.28:0/469286159 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562563ca5320 con 0x562563db32a0 2024-04-26T01:43:00.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.364+0000 7f852aa00640 10 client.25424 handle_command_reply: tid=0 2024-04-26T01:43:00.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 client.25424 shutdown 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 -- 172.21.15.28:0/469286159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 msgr2=0x562563d44f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 0x562563d44f90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f853000b970 tx=0x7f853000b4d0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 -- 172.21.15.28:0/469286159 shutdown_connections 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562563db32a0 0x562563db56c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562563d3ca00 0x562563d41fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562563d32140 0x562563d44f90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 --2- 172.21.15.28:0/469286159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562563d31820 0x562563d44a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f854e5af740 1 -- 172.21.15.28:0/469286159 >> 172.21.15.28:0/469286159 conn(0x562563d2dd30 msgr2=0x562563d2c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.365+0000 7f852aa00640 0 client.25424 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.366+0000 7f854e5af740 1 -- 172.21.15.28:0/469286159 shutdown_connections 2024-04-26T01:43:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.366+0000 7f854e5af740 1 -- 172.21.15.28:0/469286159 wait complete. 2024-04-26T01:43:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.366+0000 7f854e5af740 20 client.25424 trim_cache size 0 max 16384 2024-04-26T01:43:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:43:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:43:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:43:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:43:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 -- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f852c0a3f20 msgr2=0x7f852c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f852c0a3f20 0x7f852c0a63e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f853c009fd0 tx=0x7f853c0098c0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 -- 172.21.15.28:0/3947894053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 msgr2=0x7f8544076460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 0x7f8544076460 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8538005ab0 tx=0x7f8538003d50 comp rx=0 tx=0).stop 2024-04-26T01:43:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 -- 172.21.15.28:0/3947894053 shutdown_connections 2024-04-26T01:43:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f852c0a3f20 0x7f852c0a63e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8544076ee0 0x7f8544072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8544109490 0x7f85440769a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 --2- 172.21.15.28:0/3947894053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85441043c0 0x7f8544076460 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.367+0000 7f854b800640 1 -- 172.21.15.28:0/3947894053 >> 172.21.15.28:0/3947894053 conn(0x7f85440ffc80 msgr2=0x7f8544101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.368+0000 7f854b800640 1 -- 172.21.15.28:0/3947894053 shutdown_connections 2024-04-26T01:43:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:00.368+0000 7f854b800640 1 -- 172.21.15.28:0/3947894053 wait complete. 2024-04-26T01:43:01.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 65.0 s, 3431.2 tps, lat 28.753 ms stddev 30.756 2024-04-26T01:43:01.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:01 smithi028 ceph-mon[28247]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-26T01:43:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:01 smithi169 ceph-mon[32851]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-26T01:43:01.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:01 smithi053 ceph-mon[32593]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-26T01:43:03.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:03 smithi028 ceph-mon[28247]: pgmap v65: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.1 MiB/s wr, 638 op/s 2024-04-26T01:43:03.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:03 smithi169 ceph-mon[32851]: pgmap v65: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.1 MiB/s wr, 638 op/s 2024-04-26T01:43:03.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:03 smithi053 ceph-mon[32593]: pgmap v65: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.1 MiB/s wr, 638 op/s 2024-04-26T01:43:05.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:05 smithi028 ceph-mon[28247]: pgmap v66: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 10 MiB/s wr, 942 op/s 2024-04-26T01:43:05.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:05 smithi169 ceph-mon[32851]: pgmap v66: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 10 MiB/s wr, 942 op/s 2024-04-26T01:43:05.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:05 smithi053 ceph-mon[32593]: pgmap v66: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 10 MiB/s wr, 942 op/s 2024-04-26T01:43:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 70.0 s, 379.0 tps, lat 261.654 ms stddev 291.003 2024-04-26T01:43:07.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:07 smithi028 ceph-mon[28247]: pgmap v67: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-04-26T01:43:07.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:07 smithi169 ceph-mon[32851]: pgmap v67: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-04-26T01:43:07.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:07 smithi053 ceph-mon[32593]: pgmap v67: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-04-26T01:43:09.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:09 smithi028 ceph-mon[28247]: pgmap v68: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-04-26T01:43:09.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:09 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:09.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:09 smithi169 ceph-mon[32851]: pgmap v68: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-04-26T01:43:09.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:09 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:09.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:09 smithi053 ceph-mon[32593]: pgmap v68: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-04-26T01:43:09.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:09 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 75.0 s, 3205.0 tps, lat 31.798 ms stddev 36.155 2024-04-26T01:43:11.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:11 smithi028 ceph-mon[28247]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-26T01:43:11.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:11 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:11.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:11 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:11.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:11 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:43:11.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:11 smithi169 ceph-mon[32851]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-26T01:43:11.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:11 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:11.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:11 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:11 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:43:11.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:11 smithi053 ceph-mon[32593]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-26T01:43:11.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:11 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:11.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:11 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:11 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:43:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:13 smithi028 ceph-mon[28247]: pgmap v70: 97 pgs: 97 active+clean; 1.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 30 MiB/s wr, 1.71k op/s 2024-04-26T01:43:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.549+0000 7f7e2d000640 1 -- 172.21.15.28:0/391956494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 msgr2=0x7f7e28114b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.549+0000 7f7e2d000640 1 --2- 172.21.15.28:0/391956494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 0x7f7e28114b60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009930 tx=0x7f7e1402a4d0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.549+0000 7f7e2d000640 1 -- 172.21.15.28:0/391956494 shutdown_connections 2024-04-26T01:43:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.550+0000 7f7e2d000640 1 --2- 172.21.15.28:0/391956494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e281150a0 0x7f7e28117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.550+0000 7f7e2d000640 1 --2- 172.21.15.28:0/391956494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 0x7f7e28114b60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.550+0000 7f7e2d000640 1 --2- 172.21.15.28:0/391956494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e28108f50 0x7f7e28109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.550+0000 7f7e2d000640 1 -- 172.21.15.28:0/391956494 >> 172.21.15.28:0/391956494 conn(0x7f7e2806d020 msgr2=0x7f7e2806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:13.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.550+0000 7f7e2d000640 1 -- 172.21.15.28:0/391956494 shutdown_connections 2024-04-26T01:43:13.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.550+0000 7f7e2d000640 1 -- 172.21.15.28:0/391956494 wait complete. 2024-04-26T01:43:13.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.552+0000 7f7e2d000640 1 Processor -- start 2024-04-26T01:43:13.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.552+0000 7f7e2d000640 1 -- start start 2024-04-26T01:43:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.553+0000 7f7e2d000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e28108f50 0x7f7e2810c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.553+0000 7f7e2d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 0x7f7e2810a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.553+0000 7f7e2d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e281150a0 0x7f7e2810b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.553+0000 7f7e2d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2811a9e0 con 0x7f7e281150a0 2024-04-26T01:43:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.553+0000 7f7e2d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2811a860 con 0x7f7e28109870 2024-04-26T01:43:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.553+0000 7f7e2d000640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2811ab60 con 0x7f7e28108f50 2024-04-26T01:43:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.554+0000 7f7e25400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 0x7f7e2810a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e25400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 0x7f7e2810a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46464/0 (socket says 172.21.15.28:46464) 2024-04-26T01:43:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e25400640 1 -- 172.21.15.28:0/2437088385 learned_addr learned my addr 172.21.15.28:0/2437088385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e26800640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e281150a0 0x7f7e2810b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e25400640 1 -- 172.21.15.28:0/2437088385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e28108f50 msgr2=0x7f7e2810c5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T01:43:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e25400640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e28108f50 0x7f7e2810c5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e25400640 1 -- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e281150a0 msgr2=0x7f7e2810b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e25400640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e281150a0 0x7f7e2810b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e25400640 1 -- 172.21.15.28:0/2437088385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e281b3900 con 0x7f7e28109870 2024-04-26T01:43:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e26800640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e281150a0 0x7f7e2810b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:43:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.555+0000 7f7e25400640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 0x7f7e2810a9c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7e14005ab0 tx=0x7f7e14003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.556+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2437088385 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1403f070 con 0x7f7e28109870 2024-04-26T01:43:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.556+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2437088385 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e14031b50 con 0x7f7e28109870 2024-04-26T01:43:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.556+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2437088385 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14035030 con 0x7f7e28109870 2024-04-26T01:43:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.556+0000 7f7e2d000640 1 -- 172.21.15.28:0/2437088385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e281b3ad0 con 0x7f7e28109870 2024-04-26T01:43:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.556+0000 7f7e2d000640 1 -- 172.21.15.28:0/2437088385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e281b3f40 con 0x7f7e28109870 2024-04-26T01:43:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.560+0000 7f7e2fd6f740 1 Processor -- start 2024-04-26T01:43:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.560+0000 7f7e2fd6f740 1 -- start start 2024-04-26T01:43:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.560+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958de820 0x556e958dec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.560+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958df140 0x556e958e94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.560+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958e9a00 0x556e958ebdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.560+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e95844900 con 0x556e958df140 2024-04-26T01:43:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.560+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556e957eaef0 con 0x556e958de820 2024-04-26T01:43:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.560+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x556e958408c0 con 0x556e958e9a00 2024-04-26T01:43:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e26800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958de820 0x556e958dec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e26800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958de820 0x556e958dec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46480/0 (socket says 172.21.15.28:46480) 2024-04-26T01:43:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e26800640 1 -- 172.21.15.28:0/3038659360 learned_addr learned my addr 172.21.15.28:0/3038659360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e26800640 1 -- 172.21.15.28:0/3038659360 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958e9a00 msgr2=0x556e958ebdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:43:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e26800640 1 --2- 172.21.15.28:0/3038659360 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958e9a00 0x556e958ebdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e26800640 1 -- 172.21.15.28:0/3038659360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958df140 msgr2=0x556e958e94c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:43:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e26800640 1 --2- 172.21.15.28:0/3038659360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958df140 0x556e958e94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e26800640 1 -- 172.21.15.28:0/3038659360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e95557210 con 0x556e958de820 2024-04-26T01:43:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.561+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2437088385 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7e14042070 con 0x7f7e28109870 2024-04-26T01:43:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.562+0000 7f7e26800640 1 --2- 172.21.15.28:0/3038659360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958de820 0x556e958dec00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009b50 tx=0x7f7e1800e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3521fff741d05dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.563+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7e0c0a3e40 0x7f7e0c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.563+0000 7f7e13400640 1 -- 172.21.15.28:0/3038659360 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18016070 con 0x556e958de820 2024-04-26T01:43:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.563+0000 7f7e13400640 1 -- 172.21.15.28:0/3038659360 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e18009e40 con 0x556e958de820 2024-04-26T01:43:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.563+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2437088385 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f7e140f0f80 con 0x7f7e28109870 2024-04-26T01:43:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.563+0000 7f7e25e00640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7e0c0a3e40 0x7f7e0c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.564+0000 7f7e13400640 1 -- 172.21.15.28:0/3038659360 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18010790 con 0x556e958de820 2024-04-26T01:43:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.564+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3038659360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958de820 msgr2=0x556e958dec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.564+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3038659360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958de820 0x556e958dec00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009b50 tx=0x7f7e1800e6c0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.564+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3038659360 shutdown_connections 2024-04-26T01:43:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.564+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3038659360 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958e9a00 0x556e958ebdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.564+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3038659360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958df140 0x556e958e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.564+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3038659360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958de820 0x556e958dec00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.564+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3038659360 >> 172.21.15.28:0/3038659360 conn(0x556e958dad30 msgr2=0x556e958db110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.565+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3038659360 shutdown_connections 2024-04-26T01:43:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.566+0000 7f7e25e00640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7e0c0a3e40 0x7f7e0c0a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e20005c40 tx=0x7f7e20005bd0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.566+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3038659360 wait complete. 2024-04-26T01:43:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.566+0000 7f7e2fd6f740 1 -- start start 2024-04-26T01:43:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.567+0000 7f7e2fd6f740 1 Processor -- start 2024-04-26T01:43:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.567+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958e6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.567+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.567+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e958e72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958e6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46488/0 (socket says 172.21.15.28:46488) 2024-04-26T01:43:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25400640 1 -- 172.21.15.28:0/3621708829 learned_addr learned my addr 172.21.15.28:0/3621708829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e958ec680 con 0x556e958e9a00 2024-04-26T01:43:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556e957fe310 con 0x556e958df140 2024-04-26T01:43:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x556e958ec800 con 0x556e958de820 2024-04-26T01:43:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e2fd6f740 10 client.0 _ll_register_callbacks cb 0x556e957cccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25400640 1 -- 172.21.15.28:0/3621708829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 msgr2=0x556e958e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25400640 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958e6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25400640 1 -- 172.21.15.28:0/3621708829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 msgr2=0x556e958e72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25400640 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e958e72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.568+0000 7f7e25400640 1 -- 172.21.15.28:0/3621708829 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e95623620 con 0x556e958df140 2024-04-26T01:43:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.569+0000 7f7e07e00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:43:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.569+0000 7f7e25400640 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958e6d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7e14031440 tx=0x7f7e140bca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae33dadd63ff5c68 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.570+0000 7f7e07e00640 1 -- 172.21.15.28:0/3621708829 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e140d6060 con 0x556e958df140 2024-04-26T01:43:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.570+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556e958588b0 con 0x556e958df140 2024-04-26T01:43:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.570+0000 7f7e07e00640 1 -- 172.21.15.28:0/3621708829 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f7e140bb1f0 con 0x556e958df140 2024-04-26T01:43:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.571+0000 7f7e07e00640 1 -- 172.21.15.28:0/3621708829 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e140d6060 con 0x556e958df140 2024-04-26T01:43:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.571+0000 7f7e2fd6f740 10 client.25436 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.571+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556e958f8820 con 0x556e958df140 2024-04-26T01:43:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e07e00640 1 -- 172.21.15.28:0/3621708829 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556e958588b0 con 0x556e958df140 2024-04-26T01:43:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e07e00640 1 -- 172.21.15.28:0/3621708829 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f7e140b9560 con 0x556e958df140 2024-04-26T01:43:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e2fd6f740 10 client.25436 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e2fd6f740 10 client.25436 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e2fd6f740 20 client.25436 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e958f9cd0 0x556e958fc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e2fd6f740 4 client.25436 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e95852320 con 0x556e958f9cd0 2024-04-26T01:43:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.572+0000 7f7e25e00640 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e958f9cd0 0x556e958fc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.573+0000 7f7e07e00640 10 client.25436 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.574+0000 7f7e25e00640 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e958f9cd0 0x556e958fc110 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c2a2a0c17dcdc7 server_cookie=c49989ece5f93e03 in_seq=0 out_seq=0 2024-04-26T01:43:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.574+0000 7f7e07e00640 1 -- 172.21.15.28:0/3621708829 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e95852320 con 0x556e958f9cd0 2024-04-26T01:43:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.574+0000 7f7e07e00640 10 client.25436 handle_command_reply: tid=0 2024-04-26T01:43:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 client.25436 shutdown 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 msgr2=0x556e958e6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958e6d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7e14031440 tx=0x7f7e140bca30 comp rx=0 tx=0).stop 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 shutdown_connections 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e958f9cd0 0x556e958fc110 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e958e72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958e6d90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/3621708829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958e6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 >> 172.21.15.28:0/3621708829 conn(0x556e958dad30 msgr2=0x556e958e0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.575+0000 7f7e07e00640 0 client.25436 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.576+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 shutdown_connections 2024-04-26T01:43:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.576+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/3621708829 wait complete. 2024-04-26T01:43:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.576+0000 7f7e2fd6f740 20 client.25436 trim_cache size 0 max 16384 2024-04-26T01:43:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.601+0000 7f7e2fd6f740 1 Processor -- start 2024-04-26T01:43:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.602+0000 7f7e2fd6f740 1 -- start start 2024-04-26T01:43:13.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.602+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958e1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.602+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958df140 0x556e958f5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.602+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958e9a00 0x556e95901a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.602+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e281bf1f0 con 0x556e958df140 2024-04-26T01:43:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.602+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7e281bca90 con 0x556e958e9a00 2024-04-26T01:43:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.602+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7e281ba8e0 con 0x556e958de820 2024-04-26T01:43:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958e9a00 0x556e95901a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958e9a00 0x556e95901a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46494/0 (socket says 172.21.15.28:46494) 2024-04-26T01:43:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 -- 172.21.15.28:0/786678099 learned_addr learned my addr 172.21.15.28:0/786678099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 -- 172.21.15.28:0/786678099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 msgr2=0x556e958e1230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:43:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 --2- 172.21.15.28:0/786678099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958e1230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 -- 172.21.15.28:0/786678099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958df140 msgr2=0x556e958f5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 --2- 172.21.15.28:0/786678099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958df140 0x556e958f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 -- 172.21.15.28:0/786678099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e958d6690 con 0x556e958e9a00 2024-04-26T01:43:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.603+0000 7f7e26800640 1 --2- 172.21.15.28:0/786678099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958e9a00 0x556e95901a20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7e18005d30 tx=0x7f7e18009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=affedf85cc10f074 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.604+0000 7f7e12a00640 1 -- 172.21.15.28:0/786678099 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18016050 con 0x556e958e9a00 2024-04-26T01:43:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.604+0000 7f7e12a00640 1 -- 172.21.15.28:0/786678099 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e18004620 con 0x556e958e9a00 2024-04-26T01:43:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.604+0000 7f7e12a00640 1 -- 172.21.15.28:0/786678099 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180116d0 con 0x556e958e9a00 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/786678099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958e9a00 msgr2=0x556e95901a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/786678099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958e9a00 0x556e95901a20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7e18005d30 tx=0x7f7e18009c40 comp rx=0 tx=0).stop 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/786678099 shutdown_connections 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/786678099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958e9a00 0x556e95901a20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/786678099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958df140 0x556e958f5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/786678099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/786678099 >> 172.21.15.28:0/786678099 conn(0x556e958dad30 msgr2=0x556e958e1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/786678099 shutdown_connections 2024-04-26T01:43:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.605+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/786678099 wait complete. 2024-04-26T01:43:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.606+0000 7f7e2fd6f740 1 -- start start 2024-04-26T01:43:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.606+0000 7f7e2fd6f740 1 Processor -- start 2024-04-26T01:43:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.606+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.606+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958ed1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.606+0000 7f7e2fd6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e9595e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.606+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e281baf50 con 0x556e958e9a00 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.606+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7e281bc3c0 con 0x556e958df140 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.606+0000 7f7e2fd6f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7e281bb620 con 0x556e958de820 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.607+0000 7f7e2fd6f740 10 client.0 _ll_register_callbacks cb 0x7f7e140b9260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.607+0000 7f7e26800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e9595e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.607+0000 7f7e26800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e9595e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49978/0 (socket says 172.21.15.28:49978) 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.607+0000 7f7e26800640 1 -- 172.21.15.28:0/1413523875 learned_addr learned my addr 172.21.15.28:0/1413523875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.607+0000 7f7e25e00640 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.608+0000 7f7e25400640 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958ed1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.608+0000 7f7e26800640 1 -- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 msgr2=0x556e958eccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.608+0000 7f7e26800640 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958eccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.608+0000 7f7e26800640 1 -- 172.21.15.28:0/1413523875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 msgr2=0x556e958ed1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.608+0000 7f7e26800640 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958ed1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.608+0000 7f7e26800640 1 -- 172.21.15.28:0/1413523875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e9595e8c0 con 0x556e958e9a00 2024-04-26T01:43:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.608+0000 7f7e25e00640 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958eccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.610+0000 7f7e26800640 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e9595e280 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7e18004ee0 tx=0x7f7e1800efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853f55756d0c28b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.610+0000 7f7e07e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:43:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.611+0000 7f7e07e00640 1 -- 172.21.15.28:0/1413523875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18005450 con 0x556e958e9a00 2024-04-26T01:43:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.611+0000 7f7e07e00640 1 -- 172.21.15.28:0/1413523875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f7e18011470 con 0x556e958e9a00 2024-04-26T01:43:13.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.611+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/1413523875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e18004620 con 0x556e958e9a00 2024-04-26T01:43:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.611+0000 7f7e07e00640 1 -- 172.21.15.28:0/1413523875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e18016020 con 0x556e958e9a00 2024-04-26T01:43:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.612+0000 7f7e2fd6f740 10 client.15303 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.612+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/1413523875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556e9595fbd0 con 0x556e958e9a00 2024-04-26T01:43:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.612+0000 7f7e07e00640 1 -- 172.21.15.28:0/1413523875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e18004620 con 0x556e958e9a00 2024-04-26T01:43:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.613+0000 7f7e07e00640 1 -- 172.21.15.28:0/1413523875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f7e18023020 con 0x556e958e9a00 2024-04-26T01:43:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.613+0000 7f7e2fd6f740 10 client.15303 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.613+0000 7f7e2fd6f740 10 client.15303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.613+0000 7f7e2fd6f740 20 client.15303 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.613+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e95960060 0x556e95962480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.613+0000 7f7e2fd6f740 4 client.15303 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:43:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.613+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/1413523875 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e9584c190 con 0x556e95960060 2024-04-26T01:43:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.614+0000 7f7e25e00640 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e95960060 0x556e95962480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.615+0000 7f7e07e00640 10 client.15303 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.615+0000 7f7e25e00640 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e95960060 0x556e95962480 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34c02f3197b4b237 server_cookie=8f03f425dd16633a in_seq=0 out_seq=0 2024-04-26T01:43:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.631+0000 7f7e07e00640 1 -- 172.21.15.28:0/1413523875 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e9584c190 con 0x556e95960060 2024-04-26T01:43:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.631+0000 7f7e07e00640 10 client.15303 handle_command_reply: tid=0 2024-04-26T01:43:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.631+0000 7f7e2fd6f740 1 client.15303 shutdown 2024-04-26T01:43:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.632+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/1413523875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 msgr2=0x556e9595e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.632+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e9595e280 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7e18004ee0 tx=0x7f7e1800efb0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.633+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/1413523875 shutdown_connections 2024-04-26T01:43:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.633+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e95960060 0x556e95962480 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.633+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e958e9a00 0x556e9595e280 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.633+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e958df140 0x556e958ed1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.633+0000 7f7e2fd6f740 1 --2- 172.21.15.28:0/1413523875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e958de820 0x556e958eccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.633+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/1413523875 >> 172.21.15.28:0/1413523875 conn(0x556e958dad30 msgr2=0x556e958e6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.633+0000 7f7e07e00640 0 client.15303 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.633+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/1413523875 shutdown_connections 2024-04-26T01:43:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.634+0000 7f7e2fd6f740 1 -- 172.21.15.28:0/1413523875 wait complete. 2024-04-26T01:43:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.634+0000 7f7e2fd6f740 20 client.15303 trim_cache size 0 max 16384 2024-04-26T01:43:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:43:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:43:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 -- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7e0c0a3e40 msgr2=0x7f7e0c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7e0c0a3e40 0x7f7e0c0a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e20005c40 tx=0x7f7e20005bd0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 -- 172.21.15.28:0/2437088385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 msgr2=0x7f7e2810a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 0x7f7e2810a9c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7e14005ab0 tx=0x7f7e14003b90 comp rx=0 tx=0).stop 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 -- 172.21.15.28:0/2437088385 shutdown_connections 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f7e0c0a3e40 0x7f7e0c0a6300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e281150a0 0x7f7e2810b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e28109870 0x7f7e2810a9c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 --2- 172.21.15.28:0/2437088385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e28108f50 0x7f7e2810c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.635+0000 7f7e2d000640 1 -- 172.21.15.28:0/2437088385 >> 172.21.15.28:0/2437088385 conn(0x7f7e2806d020 msgr2=0x7f7e28070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.636+0000 7f7e2d000640 1 -- 172.21.15.28:0/2437088385 shutdown_connections 2024-04-26T01:43:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:13.636+0000 7f7e2d000640 1 -- 172.21.15.28:0/2437088385 wait complete. 2024-04-26T01:43:13.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:13 smithi169 ceph-mon[32851]: pgmap v70: 97 pgs: 97 active+clean; 1.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 30 MiB/s wr, 1.71k op/s 2024-04-26T01:43:13.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:13 smithi053 ceph-mon[32593]: pgmap v70: 97 pgs: 97 active+clean; 1.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 30 MiB/s wr, 1.71k op/s 2024-04-26T01:43:15.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:15 smithi028 ceph-mon[28247]: pgmap v71: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 33 MiB/s wr, 2.02k op/s 2024-04-26T01:43:15.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:15 smithi169 ceph-mon[32851]: pgmap v71: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 33 MiB/s wr, 2.02k op/s 2024-04-26T01:43:15.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:15 smithi053 ceph-mon[32593]: pgmap v71: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 33 MiB/s wr, 2.02k op/s 2024-04-26T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 80.0 s, 2953.6 tps, lat 33.820 ms stddev 40.558 2024-04-26T01:43:17.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:17 smithi028 ceph-mon[28247]: pgmap v72: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 1.72k op/s 2024-04-26T01:43:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:17 smithi169 ceph-mon[32851]: pgmap v72: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 1.72k op/s 2024-04-26T01:43:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:17 smithi053 ceph-mon[32593]: pgmap v72: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 1.72k op/s 2024-04-26T01:43:19.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:19 smithi028 ceph-mon[28247]: pgmap v73: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 1.72k op/s 2024-04-26T01:43:19.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:19 smithi169 ceph-mon[32851]: pgmap v73: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 1.72k op/s 2024-04-26T01:43:19.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:19 smithi053 ceph-mon[32593]: pgmap v73: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 1.72k op/s 2024-04-26T01:43:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 85.0 s, 2978.4 tps, lat 33.515 ms stddev 37.192 2024-04-26T01:43:21.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:21 smithi169 ceph-mon[32851]: pgmap v74: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 2.06k op/s 2024-04-26T01:43:21.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:21 smithi053 ceph-mon[32593]: pgmap v74: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 2.06k op/s 2024-04-26T01:43:21.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:21 smithi028 ceph-mon[28247]: pgmap v74: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 2.06k op/s 2024-04-26T01:43:23.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:23 smithi169 ceph-mon[32851]: pgmap v75: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.1 MiB/s wr, 656 op/s 2024-04-26T01:43:23.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:23 smithi053 ceph-mon[32593]: pgmap v75: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.1 MiB/s wr, 656 op/s 2024-04-26T01:43:23.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:23 smithi028 ceph-mon[28247]: pgmap v75: 97 pgs: 97 active+clean; 1.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.1 MiB/s wr, 656 op/s 2024-04-26T01:43:25.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:25 smithi169 ceph-mon[32851]: pgmap v76: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 955 op/s 2024-04-26T01:43:25.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:25 smithi053 ceph-mon[32593]: pgmap v76: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 955 op/s 2024-04-26T01:43:25.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:25 smithi028 ceph-mon[28247]: pgmap v76: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 955 op/s 2024-04-26T01:43:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 90.0 s, 2913.6 tps, lat 34.272 ms stddev 42.320 2024-04-26T01:43:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.833+0000 7f754d400640 1 -- 172.21.15.28:0/1433528658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548112de0 msgr2=0x7f75481151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.833+0000 7f754d400640 1 --2- 172.21.15.28:0/1433528658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548112de0 0x7f75481151d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7540009cd0 tx=0x7f754002a8e0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.834+0000 7f754d400640 1 -- 172.21.15.28:0/1433528658 shutdown_connections 2024-04-26T01:43:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.834+0000 7f754d400640 1 --2- 172.21.15.28:0/1433528658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548112de0 0x7f75481151d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.834+0000 7f754d400640 1 --2- 172.21.15.28:0/1433528658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7548076da0 0x7f75481128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.834+0000 7f754d400640 1 --2- 172.21.15.28:0/1433528658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7548076480 0x7f7548076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.834+0000 7f754d400640 1 -- 172.21.15.28:0/1433528658 >> 172.21.15.28:0/1433528658 conn(0x7f75480ffc50 msgr2=0x7f7548102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.835+0000 7f754d400640 1 -- 172.21.15.28:0/1433528658 shutdown_connections 2024-04-26T01:43:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.835+0000 7f754d400640 1 -- 172.21.15.28:0/1433528658 wait complete. 2024-04-26T01:43:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.837+0000 7f754d400640 1 Processor -- start 2024-04-26T01:43:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.838+0000 7f754d400640 1 -- start start 2024-04-26T01:43:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.838+0000 7f754d400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548076480 0x7f75481a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.838+0000 7f754d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7548076da0 0x7f75481a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.838+0000 7f754d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7548112de0 0x7f754819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.838+0000 7f754d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7548118800 con 0x7f7548112de0 2024-04-26T01:43:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.838+0000 7f754d400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7548118680 con 0x7f7548076da0 2024-04-26T01:43:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.838+0000 7f754d400640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7548118980 con 0x7f7548076480 2024-04-26T01:43:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.839+0000 7f7546800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548076480 0x7f75481a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.839+0000 7f7546800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548076480 0x7f75481a4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:35820/0 (socket says 172.21.15.28:35820) 2024-04-26T01:43:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.839+0000 7f7546800640 1 -- 172.21.15.28:0/891405465 learned_addr learned my addr 172.21.15.28:0/891405465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.839+0000 7f7545e00640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7548076da0 0x7f75481a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.839+0000 7f7547200640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7548112de0 0x7f754819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.840+0000 7f7547200640 1 -- 172.21.15.28:0/891405465 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548076480 msgr2=0x7f75481a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.840+0000 7f7547200640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548076480 0x7f75481a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.840+0000 7f7547200640 1 -- 172.21.15.28:0/891405465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7548076da0 msgr2=0x7f75481a4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.840+0000 7f7547200640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7548076da0 0x7f75481a4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.840+0000 7f7547200640 1 -- 172.21.15.28:0/891405465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754819f260 con 0x7f7548112de0 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.840+0000 7f7545e00640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7548076da0 0x7f75481a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.840+0000 7f7547200640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7548112de0 0x7f754819eae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f754002ae00 tx=0x7f7540003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.840+0000 7f753f400640 1 -- 172.21.15.28:0/891405465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7540004040 con 0x7f7548112de0 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.841+0000 7f754d400640 1 -- 172.21.15.28:0/891405465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754819f4f0 con 0x7f7548112de0 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.841+0000 7f754d400640 1 -- 172.21.15.28:0/891405465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75481ab7f0 con 0x7f7548112de0 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.841+0000 7f753f400640 1 -- 172.21.15.28:0/891405465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7540004690 con 0x7f7548112de0 2024-04-26T01:43:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.841+0000 7f753f400640 1 -- 172.21.15.28:0/891405465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7540004040 con 0x7f7548112de0 2024-04-26T01:43:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.844+0000 7f755017d740 1 Processor -- start 2024-04-26T01:43:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.845+0000 7f753f400640 1 -- 172.21.15.28:0/891405465 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7540030be0 con 0x7f7548112de0 2024-04-26T01:43:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.847+0000 7f755017d740 1 -- start start 2024-04-26T01:43:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.849+0000 7f755017d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5609820 0x5632f5609c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.849+0000 7f755017d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f56144c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.849+0000 7f7546800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f56144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.850+0000 7f755017d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 0x5632f5616df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.850+0000 7f755017d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632f556f900 con 0x5632f5609820 2024-04-26T01:43:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.850+0000 7f755017d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5632f5515ef0 con 0x5632f560a140 2024-04-26T01:43:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.850+0000 7f755017d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5632f556b8c0 con 0x5632f5614a00 2024-04-26T01:43:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.850+0000 7f7546800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f56144c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59342/0 (socket says 172.21.15.28:59342) 2024-04-26T01:43:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.850+0000 7f7546800640 1 -- 172.21.15.28:0/241143743 learned_addr learned my addr 172.21.15.28:0/241143743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.850+0000 7f7545e00640 1 --2- 172.21.15.28:0/241143743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5609820 0x5632f5609c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.850+0000 7f753f400640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f752c0a4040 0x7f752c0a6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.851+0000 7f753f400640 1 -- 172.21.15.28:0/891405465 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f75400f1100 con 0x7f7548112de0 2024-04-26T01:43:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.852+0000 7f7546800640 1 -- 172.21.15.28:0/241143743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 msgr2=0x5632f5616df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:43:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.852+0000 7f7546800640 1 --2- 172.21.15.28:0/241143743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 0x5632f5616df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.852+0000 7f7546800640 1 -- 172.21.15.28:0/241143743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5609820 msgr2=0x5632f5609c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.852+0000 7f7546800640 1 --2- 172.21.15.28:0/241143743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5609820 0x5632f5609c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.852+0000 7f7546800640 1 -- 172.21.15.28:0/241143743 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f5282210 con 0x5632f560a140 2024-04-26T01:43:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.852+0000 7f7547200640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f752c0a4040 0x7f752c0a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.852+0000 7f7546800640 1 --2- 172.21.15.28:0/241143743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f56144c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7538009d30 tx=0x7f75380117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=646c03f8cd42fe95 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.853+0000 7f7533400640 1 -- 172.21.15.28:0/241143743 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7538012650 con 0x5632f560a140 2024-04-26T01:43:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.853+0000 7f7533400640 1 -- 172.21.15.28:0/241143743 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7538009e20 con 0x5632f560a140 2024-04-26T01:43:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.853+0000 7f7533400640 1 -- 172.21.15.28:0/241143743 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753800fc30 con 0x5632f560a140 2024-04-26T01:43:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.854+0000 7f7547200640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f752c0a4040 0x7f752c0a6500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75400b95e0 tx=0x7f75400095c0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.854+0000 7f755017d740 1 -- 172.21.15.28:0/241143743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 msgr2=0x5632f56144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.854+0000 7f755017d740 1 --2- 172.21.15.28:0/241143743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f56144c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7538009d30 tx=0x7f75380117b0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.854+0000 7f755017d740 1 -- 172.21.15.28:0/241143743 shutdown_connections 2024-04-26T01:43:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.854+0000 7f755017d740 1 --2- 172.21.15.28:0/241143743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 0x5632f5616df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.854+0000 7f755017d740 1 --2- 172.21.15.28:0/241143743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f56144c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.854+0000 7f755017d740 1 --2- 172.21.15.28:0/241143743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5609820 0x5632f5609c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.854+0000 7f755017d740 1 -- 172.21.15.28:0/241143743 >> 172.21.15.28:0/241143743 conn(0x5632f5605d30 msgr2=0x5632f5606110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.855+0000 7f755017d740 1 -- 172.21.15.28:0/241143743 shutdown_connections 2024-04-26T01:43:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.855+0000 7f755017d740 1 -- 172.21.15.28:0/241143743 wait complete. 2024-04-26T01:43:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.856+0000 7f755017d740 1 -- start start 2024-04-26T01:43:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.856+0000 7f755017d740 1 Processor -- start 2024-04-26T01:43:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.858+0000 7f755017d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5609820 0x5632f56118c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.859+0000 7f755017d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f5611e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.859+0000 7f7545e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5609820 0x5632f56118c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.859+0000 7f7545e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5609820 0x5632f56118c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59358/0 (socket says 172.21.15.28:59358) 2024-04-26T01:43:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.860+0000 7f7545e00640 1 -- 172.21.15.28:0/624384001 learned_addr learned my addr 172.21.15.28:0/624384001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.860+0000 7f755017d740 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614a00 0x5632f5612340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.860+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632f5617630 con 0x5632f5614a00 2024-04-26T01:43:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.860+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5632f5529310 con 0x5632f5609820 2024-04-26T01:43:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.860+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5632f56177b0 con 0x5632f560a140 2024-04-26T01:43:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.861+0000 7f755017d740 10 client.0 _ll_register_callbacks cb 0x5632f54f7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.861+0000 7f7546800640 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f5611e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.861+0000 7f7545e00640 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614a00 0x5632f5612340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.861+0000 7f7545e00640 1 -- 172.21.15.28:0/624384001 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 msgr2=0x5632f5611e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.861+0000 7f7545e00640 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f5611e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.861+0000 7f7545e00640 1 -- 172.21.15.28:0/624384001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614a00 msgr2=0x5632f5612340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.861+0000 7f7545e00640 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614a00 0x5632f5612340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.861+0000 7f7545e00640 1 -- 172.21.15.28:0/624384001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f534e620 con 0x5632f5609820 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.862+0000 7f7545e00640 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5609820 0x5632f56118c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7534012ea0 tx=0x7f7534009890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27df679a58986cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.862+0000 7f7527e00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:43:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.862+0000 7f7527e00640 1 -- 172.21.15.28:0/624384001 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7534011e90 con 0x5632f5609820 2024-04-26T01:43:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.862+0000 7f7527e00640 1 -- 172.21.15.28:0/624384001 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f753400f480 con 0x5632f5609820 2024-04-26T01:43:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.862+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7538009e20 con 0x5632f5609820 2024-04-26T01:43:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.863+0000 7f7527e00640 1 -- 172.21.15.28:0/624384001 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7534028050 con 0x5632f5609820 2024-04-26T01:43:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.863+0000 7f755017d740 10 client.25451 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.863+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5632f56238a0 con 0x5632f5609820 2024-04-26T01:43:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.863+0000 7f7527e00640 1 -- 172.21.15.28:0/624384001 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f753401c040 con 0x5632f5609820 2024-04-26T01:43:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.864+0000 7f7527e00640 1 -- 172.21.15.28:0/624384001 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f7534020610 con 0x5632f5609820 2024-04-26T01:43:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.864+0000 7f755017d740 10 client.25451 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.864+0000 7f755017d740 10 client.25451 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.864+0000 7f755017d740 20 client.25451 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.864+0000 7f755017d740 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5632f5624dd0 0x5632f5627220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.864+0000 7f755017d740 4 client.25451 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.864+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7538009e20 con 0x5632f5624dd0 2024-04-26T01:43:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.865+0000 7f7546800640 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5632f5624dd0 0x5632f5627220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.865+0000 7f7527e00640 10 client.25451 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.866+0000 7f7546800640 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5632f5624dd0 0x5632f5627220 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=768a20c572b44b6d server_cookie=682089113f8d04f7 in_seq=0 out_seq=0 2024-04-26T01:43:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f7527e00640 1 -- 172.21.15.28:0/624384001 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f7538009e20 con 0x5632f5624dd0 2024-04-26T01:43:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f7527e00640 10 client.25451 handle_command_reply: tid=0 2024-04-26T01:43:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 client.25451 shutdown 2024-04-26T01:43:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5609820 msgr2=0x5632f56118c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5609820 0x5632f56118c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7534012ea0 tx=0x7f7534009890 comp rx=0 tx=0).stop 2024-04-26T01:43:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 shutdown_connections 2024-04-26T01:43:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5632f5624dd0 0x5632f5627220 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614a00 0x5632f5612340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f5611e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 --2- 172.21.15.28:0/624384001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5609820 0x5632f56118c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 >> 172.21.15.28:0/624384001 conn(0x5632f5605d30 msgr2=0x5632f560bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.867+0000 7f7527e00640 0 client.25451 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.868+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 shutdown_connections 2024-04-26T01:43:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.868+0000 7f755017d740 1 -- 172.21.15.28:0/624384001 wait complete. 2024-04-26T01:43:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.868+0000 7f755017d740 20 client.25451 trim_cache size 0 max 16384 2024-04-26T01:43:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.894+0000 7f755017d740 1 Processor -- start 2024-04-26T01:43:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.894+0000 7f755017d740 1 -- start start 2024-04-26T01:43:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.894+0000 7f755017d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f560c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.894+0000 7f755017d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5614a00 0x5632f5610190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.894+0000 7f755017d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 0x5632f5612190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.894+0000 7f755017d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7528010fc0 con 0x5632f5614100 2024-04-26T01:43:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.894+0000 7f755017d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f752800e860 con 0x5632f5614a00 2024-04-26T01:43:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.894+0000 7f755017d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f752800c6b0 con 0x5632f560a140 2024-04-26T01:43:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f560c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f560c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:35846/0 (socket says 172.21.15.28:35846) 2024-04-26T01:43:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 -- 172.21.15.28:0/1359981890 learned_addr learned my addr 172.21.15.28:0/1359981890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7546800640 1 --2- 172.21.15.28:0/1359981890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5614a00 0x5632f5610190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 -- 172.21.15.28:0/1359981890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5614a00 msgr2=0x5632f5610190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 --2- 172.21.15.28:0/1359981890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5614a00 0x5632f5610190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 -- 172.21.15.28:0/1359981890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 msgr2=0x5632f5612190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 --2- 172.21.15.28:0/1359981890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 0x5632f5612190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 -- 172.21.15.28:0/1359981890 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f5610870 con 0x5632f560a140 2024-04-26T01:43:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.895+0000 7f7545e00640 1 --2- 172.21.15.28:0/1359981890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f560c230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7534020a60 tx=0x7f7534012f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b75e63ae4e56def server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.896+0000 7f7532a00640 1 -- 172.21.15.28:0/1359981890 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7534025030 con 0x5632f560a140 2024-04-26T01:43:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.896+0000 7f7532a00640 1 -- 172.21.15.28:0/1359981890 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7534031020 con 0x5632f560a140 2024-04-26T01:43:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.896+0000 7f7532a00640 1 -- 172.21.15.28:0/1359981890 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753400f480 con 0x5632f560a140 2024-04-26T01:43:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.897+0000 7f755017d740 1 -- 172.21.15.28:0/1359981890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 msgr2=0x5632f560c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.897+0000 7f755017d740 1 --2- 172.21.15.28:0/1359981890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f560c230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7534020a60 tx=0x7f7534012f90 comp rx=0 tx=0).stop 2024-04-26T01:43:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.897+0000 7f755017d740 1 -- 172.21.15.28:0/1359981890 shutdown_connections 2024-04-26T01:43:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.897+0000 7f755017d740 1 --2- 172.21.15.28:0/1359981890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 0x5632f5612190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.898+0000 7f755017d740 1 --2- 172.21.15.28:0/1359981890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f5614a00 0x5632f5610190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.898+0000 7f755017d740 1 --2- 172.21.15.28:0/1359981890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f560a140 0x5632f560c230 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.898+0000 7f755017d740 1 -- 172.21.15.28:0/1359981890 >> 172.21.15.28:0/1359981890 conn(0x5632f5605d30 msgr2=0x5632f5604e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.898+0000 7f755017d740 1 -- 172.21.15.28:0/1359981890 shutdown_connections 2024-04-26T01:43:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.898+0000 7f755017d740 1 -- 172.21.15.28:0/1359981890 wait complete. 2024-04-26T01:43:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.899+0000 7f755017d740 1 -- start start 2024-04-26T01:43:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.899+0000 7f755017d740 1 Processor -- start 2024-04-26T01:43:26.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f755017d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f5683ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f755017d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 0x5632f5684430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f755017d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 0x5632f5684970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f755017d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f752800cd20 con 0x5632f5614100 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f755017d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f752800e190 con 0x5632f560a140 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f755017d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f752800d3f0 con 0x5632f5614a00 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f755017d740 10 client.0 _ll_register_callbacks cb 0x7f7534012a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f7545e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f5683ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f7545e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f5683ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59384/0 (socket says 172.21.15.28:59384) 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f7545e00640 1 -- 172.21.15.28:0/1420881693 learned_addr learned my addr 172.21.15.28:0/1420881693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.900+0000 7f7545e00640 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 0x5632f5684970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.901+0000 7f7546800640 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 0x5632f5684430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.901+0000 7f7545e00640 1 -- 172.21.15.28:0/1420881693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 msgr2=0x5632f5684970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.901+0000 7f7545e00640 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 0x5632f5684970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.901+0000 7f7545e00640 1 -- 172.21.15.28:0/1420881693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 msgr2=0x5632f5684430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.901+0000 7f7545e00640 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 0x5632f5684430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.901+0000 7f7545e00640 1 -- 172.21.15.28:0/1420881693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f561ae90 con 0x5632f560a140 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.901+0000 7f7545e00640 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f5683ef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7534012ea0 tx=0x7f7534012d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae935d69df68fee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.901+0000 7f7527e00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.902+0000 7f7527e00640 1 -- 172.21.15.28:0/1420881693 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7534020e60 con 0x5632f560a140 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.902+0000 7f7527e00640 1 -- 172.21.15.28:0/1420881693 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f7534020390 con 0x5632f560a140 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.902+0000 7f755017d740 1 -- 172.21.15.28:0/1420881693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7534031020 con 0x5632f560a140 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.902+0000 7f7527e00640 1 -- 172.21.15.28:0/1420881693 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7534020590 con 0x5632f560a140 2024-04-26T01:43:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.902+0000 7f755017d740 10 client.25457 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.902+0000 7f755017d740 1 -- 172.21.15.28:0/1420881693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5632f569c820 con 0x5632f560a140 2024-04-26T01:43:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.903+0000 7f7527e00640 1 -- 172.21.15.28:0/1420881693 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7534031020 con 0x5632f560a140 2024-04-26T01:43:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.903+0000 7f7527e00640 1 -- 172.21.15.28:0/1420881693 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f7534035070 con 0x5632f560a140 2024-04-26T01:43:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.903+0000 7f755017d740 10 client.25457 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.903+0000 7f755017d740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.903+0000 7f755017d740 20 client.25457 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.903+0000 7f755017d740 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5632f569cdb0 0x5632f569f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.903+0000 7f755017d740 4 client.25457 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:43:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.903+0000 7f755017d740 1 -- 172.21.15.28:0/1420881693 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632f55838b0 con 0x5632f569cdb0 2024-04-26T01:43:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.904+0000 7f7546800640 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5632f569cdb0 0x5632f569f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.906+0000 7f7546800640 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5632f569cdb0 0x5632f569f1d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91a5dba7795a95dc server_cookie=3b4b9d75556f6496 in_seq=0 out_seq=0 2024-04-26T01:43:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.907+0000 7f7527e00640 10 client.25457 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.919+0000 7f7527e00640 1 -- 172.21.15.28:0/1420881693 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632f55838b0 con 0x5632f569cdb0 2024-04-26T01:43:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.919+0000 7f7527e00640 10 client.25457 handle_command_reply: tid=0 2024-04-26T01:43:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.919+0000 7f755017d740 1 client.25457 shutdown 2024-04-26T01:43:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 -- 172.21.15.28:0/1420881693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 msgr2=0x5632f5683ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f5683ef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7534012ea0 tx=0x7f7534012d50 comp rx=0 tx=0).stop 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 -- 172.21.15.28:0/1420881693 shutdown_connections 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5632f569cdb0 0x5632f569f1d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632f5614a00 0x5632f5684970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632f5614100 0x5632f5684430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 --2- 172.21.15.28:0/1420881693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5632f560a140 0x5632f5683ef0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 -- 172.21.15.28:0/1420881693 >> 172.21.15.28:0/1420881693 conn(0x5632f5605d30 msgr2=0x5632f5604e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f7527e00640 0 client.25457 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 -- 172.21.15.28:0/1420881693 shutdown_connections 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 1 -- 172.21.15.28:0/1420881693 wait complete. 2024-04-26T01:43:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.920+0000 7f755017d740 20 client.25457 trim_cache size 0 max 16384 2024-04-26T01:43:26.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:43:26.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:43:26.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:43:26.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:43:26.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 -- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f752c0a4040 msgr2=0x7f752c0a6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f752c0a4040 0x7f752c0a6500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75400b95e0 tx=0x7f75400095c0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 -- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7548112de0 msgr2=0x7f754819eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7548112de0 0x7f754819eae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f754002ae00 tx=0x7f7540003970 comp rx=0 tx=0).stop 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 -- 172.21.15.28:0/891405465 shutdown_connections 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f752c0a4040 0x7f752c0a6500 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7548112de0 0x7f754819eae0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7548076da0 0x7f75481a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 --2- 172.21.15.28:0/891405465 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7548076480 0x7f75481a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.921+0000 7f754d400640 1 -- 172.21.15.28:0/891405465 >> 172.21.15.28:0/891405465 conn(0x7f75480ffc50 msgr2=0x7f75481143c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.922+0000 7f754d400640 1 -- 172.21.15.28:0/891405465 shutdown_connections 2024-04-26T01:43:26.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:26.922+0000 7f754d400640 1 -- 172.21.15.28:0/891405465 wait complete. 2024-04-26T01:43:27.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:27 smithi169 ceph-mon[32851]: pgmap v77: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.9 MiB/s wr, 639 op/s 2024-04-26T01:43:27.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:27 smithi053 ceph-mon[32593]: pgmap v77: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.9 MiB/s wr, 639 op/s 2024-04-26T01:43:27.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:27 smithi028 ceph-mon[28247]: pgmap v77: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.9 MiB/s wr, 639 op/s 2024-04-26T01:43:29.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:29 smithi169 ceph-mon[32851]: pgmap v78: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.9 MiB/s wr, 639 op/s 2024-04-26T01:43:29.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:29 smithi053 ceph-mon[32593]: pgmap v78: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.9 MiB/s wr, 639 op/s 2024-04-26T01:43:29.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:29 smithi028 ceph-mon[28247]: pgmap v78: 97 pgs: 97 active+clean; 1.4 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.9 MiB/s wr, 639 op/s 2024-04-26T01:43:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 95.0 s, 2797.2 tps, lat 35.641 ms stddev 38.586 2024-04-26T01:43:31.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:31 smithi169 ceph-mon[32851]: pgmap v79: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 964 op/s 2024-04-26T01:43:31.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:31 smithi053 ceph-mon[32593]: pgmap v79: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 964 op/s 2024-04-26T01:43:31.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:31 smithi028 ceph-mon[28247]: pgmap v79: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 964 op/s 2024-04-26T01:43:32.594 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:43:32.594 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-26T01:43:32.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:43:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:33.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:33 smithi028 ceph-mon[28247]: pgmap v80: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 623 op/s 2024-04-26T01:43:33.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:33 smithi169 ceph-mon[32851]: pgmap v80: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 623 op/s 2024-04-26T01:43:33.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:33 smithi053 ceph-mon[32593]: pgmap v80: 97 pgs: 97 active+clean; 1.5 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 623 op/s 2024-04-26T01:43:35.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:35 smithi028 ceph-mon[28247]: pgmap v81: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-26T01:43:35.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:35 smithi169 ceph-mon[32851]: pgmap v81: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-26T01:43:35.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:35 smithi053 ceph-mon[32593]: pgmap v81: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.038+0000 7fd0cae00700 1 -- 172.21.15.28:0/2401319119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c4104e90 msgr2=0x7fd0c4105270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.038+0000 7fd0cae00700 1 --2- 172.21.15.28:0/2401319119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c4104e90 0x7fd0c4105270 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009920 tx=0x7fd0b4009c30 comp rx=0 tx=0).stop 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.038+0000 7fd0cae00700 1 -- 172.21.15.28:0/2401319119 shutdown_connections 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.038+0000 7fd0cae00700 1 --2- 172.21.15.28:0/2401319119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c4109f70 0x7fd0c4111b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.038+0000 7fd0cae00700 1 --2- 172.21.15.28:0/2401319119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c4104e90 0x7fd0c4105270 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.038+0000 7fd0cae00700 1 --2- 172.21.15.28:0/2401319119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0c4105840 0x7fd0c4109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.038+0000 7fd0cae00700 1 -- 172.21.15.28:0/2401319119 >> 172.21.15.28:0/2401319119 conn(0x7fd0c4100720 msgr2=0x7fd0c4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.038+0000 7fd0cae00700 1 -- 172.21.15.28:0/2401319119 shutdown_connections 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.039+0000 7fd0cae00700 1 -- 172.21.15.28:0/2401319119 wait complete. 2024-04-26T01:43:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.041+0000 7fd0cae00700 1 Processor -- start 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.041+0000 7fd0cae00700 1 -- start start 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.041+0000 7fd0cae00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c4104e90 0x7fd0c419b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.041+0000 7fd0cae00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0c4105840 0x7fd0c419b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.041+0000 7fd0cae00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c4109f70 0x7fd0c419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.041+0000 7fd0cae00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c4114f20 con 0x7fd0c4109f70 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.041+0000 7fd0cae00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c4114da0 con 0x7fd0c4105840 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.041+0000 7fd0cae00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c41150a0 con 0x7fd0c4104e90 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.042+0000 7fd0c3e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c4104e90 0x7fd0c419b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.042+0000 7fd0c8e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c4109f70 0x7fd0c419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.042+0000 7fd0c8e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c4109f70 0x7fd0c419fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43236/0 (socket says 172.21.15.28:43236) 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.042+0000 7fd0c8e00700 1 -- 172.21.15.28:0/726434394 learned_addr learned my addr 172.21.15.28:0/726434394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:36.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.042+0000 7fd0c3400700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0c4105840 0x7fd0c419b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.043+0000 7fd0c8e00700 1 -- 172.21.15.28:0/726434394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c4104e90 msgr2=0x7fd0c419b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.043+0000 7fd0c8e00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c4104e90 0x7fd0c419b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.043+0000 7fd0c8e00700 1 -- 172.21.15.28:0/726434394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0c4105840 msgr2=0x7fd0c419b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.043+0000 7fd0c8e00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0c4105840 0x7fd0c419b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.043+0000 7fd0c8e00700 1 -- 172.21.15.28:0/726434394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c41a0440 con 0x7fd0c4109f70 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.044+0000 7fd0c8e00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c4109f70 0x7fd0c419fd00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc007fd0 tx=0x7fd0bc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.044+0000 7fd0c0c00700 1 -- 172.21.15.28:0/726434394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00dbd0 con 0x7fd0c4109f70 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.044+0000 7fd0c0c00700 1 -- 172.21.15.28:0/726434394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0bc00dd70 con 0x7fd0c4109f70 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.044+0000 7fd0c0c00700 1 -- 172.21.15.28:0/726434394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc005990 con 0x7fd0c4109f70 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.045+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0c41a0730 con 0x7fd0c4109f70 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.045+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0c41a7fb0 con 0x7fd0c4109f70 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.046+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0c4069730 con 0x7fd0c4109f70 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.050+0000 7fd0c0c00700 1 -- 172.21.15.28:0/726434394 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd0bc01a070 con 0x7fd0c4109f70 2024-04-26T01:43:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.053+0000 7fd0c0c00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0a80a3e70 0x7fd0a80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.053+0000 7fd0c0c00700 1 -- 172.21.15.28:0/726434394 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fd0bc0cfb40 con 0x7fd0c4109f70 2024-04-26T01:43:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.054+0000 7fd0c3e00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0a80a3e70 0x7fd0a80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.056+0000 7fd0c3e00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0a80a3e70 0x7fd0a80a6330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40098f0 tx=0x7fd0b4019040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.059+0000 7fd0c0c00700 1 -- 172.21.15.28:0/726434394 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0bc020070 con 0x7fd0c4109f70 2024-04-26T01:43:36.222 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 100.0 s, 3018.8 tps, lat 33.093 ms stddev 42.969 2024-04-26T01:43:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.508+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0c419c790 con 0x7fd0c4109f70 2024-04-26T01:43:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.510+0000 7fd0c0c00700 1 -- 172.21.15.28:0/726434394 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11865 (secure 0 0 0) 0x7fd0bc010070 con 0x7fd0c4109f70 2024-04-26T01:43:36.508 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:43:36.508 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:42:00.871153+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0a80a3e70 msgr2=0x7fd0a80a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0a80a3e70 0x7fd0a80a6330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40098f0 tx=0x7fd0b4019040 comp rx=0 tx=0).stop 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c4109f70 msgr2=0x7fd0c419fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c4109f70 0x7fd0c419fd00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc007fd0 tx=0x7fd0bc00d3b0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 shutdown_connections 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0a80a3e70 0x7fd0a80a6330 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c4109f70 0x7fd0c419fd00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c4104e90 0x7fd0c419b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 --2- 172.21.15.28:0/726434394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0c4105840 0x7fd0c419b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 >> 172.21.15.28:0/726434394 conn(0x7fd0c4100720 msgr2=0x7fd0c4102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 shutdown_connections 2024-04-26T01:43:36.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:36.514+0000 7fd0cae00700 1 -- 172.21.15.28:0/726434394 wait complete. 2024-04-26T01:43:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-26T01:43:37.466 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-26T01:43:37.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-26T01:43:37.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:37 smithi169 ceph-mon[32851]: pgmap v82: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T01:43:37.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:37 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/726434394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:37.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:37 smithi053 ceph-mon[32593]: pgmap v82: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T01:43:37.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:37 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/726434394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:37 smithi028 ceph-mon[28247]: pgmap v82: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T01:43:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:37 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/726434394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:39.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:39 smithi169 ceph-mon[32851]: pgmap v83: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T01:43:39.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:39 smithi053 ceph-mon[32593]: pgmap v83: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T01:43:39.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:39 smithi028 ceph-mon[28247]: pgmap v83: 97 pgs: 97 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T01:43:40.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.140+0000 7f8d14200640 1 -- 172.21.15.28:0/1260244821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 msgr2=0x7f8d0c104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.140+0000 7f8d14200640 1 --2- 172.21.15.28:0/1260244821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 0x7f8d0c104780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009900 tx=0x7f8cf802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.141+0000 7f8d14200640 1 -- 172.21.15.28:0/1260244821 shutdown_connections 2024-04-26T01:43:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.141+0000 7f8d14200640 1 --2- 172.21.15.28:0/1260244821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c109470 0x7f8d0c111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.141+0000 7f8d14200640 1 --2- 172.21.15.28:0/1260244821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d0c104d50 0x7f8d0c108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.141+0000 7f8d14200640 1 --2- 172.21.15.28:0/1260244821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 0x7f8d0c104780 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.141+0000 7f8d14200640 1 -- 172.21.15.28:0/1260244821 >> 172.21.15.28:0/1260244821 conn(0x7f8d0c0ffc40 msgr2=0x7f8d0c102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.141+0000 7f8d14200640 1 -- 172.21.15.28:0/1260244821 shutdown_connections 2024-04-26T01:43:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.141+0000 7f8d14200640 1 -- 172.21.15.28:0/1260244821 wait complete. 2024-04-26T01:43:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.142+0000 7f8d14200640 1 Processor -- start 2024-04-26T01:43:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.142+0000 7f8d14200640 1 -- start start 2024-04-26T01:43:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d14200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 0x7f8d0c1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d14200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c109470 0x7f8d0c1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d14200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d0c19a680 0x7f8d0c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d14200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c114420 con 0x7f8d0c109470 2024-04-26T01:43:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d14200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1142a0 con 0x7f8d0c1043a0 2024-04-26T01:43:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d14200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1145a0 con 0x7f8d0c19a680 2024-04-26T01:43:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d11800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 0x7f8d0c1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d12200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d0c19a680 0x7f8d0c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d11800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 0x7f8d0c1a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47366/0 (socket says 172.21.15.28:47366) 2024-04-26T01:43:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.143+0000 7f8d11800640 1 -- 172.21.15.28:0/1465114599 learned_addr learned my addr 172.21.15.28:0/1465114599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.144+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c109470 0x7f8d0c1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.144+0000 7f8d12200640 1 -- 172.21.15.28:0/1465114599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 msgr2=0x7f8d0c1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.144+0000 7f8d12200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 0x7f8d0c1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.144+0000 7f8d12200640 1 -- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c109470 msgr2=0x7f8d0c1a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.144+0000 7f8d12200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c109470 0x7f8d0c1a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.144+0000 7f8d12200640 1 -- 172.21.15.28:0/1465114599 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0c19b280 con 0x7f8d0c19a680 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.145+0000 7f8d12200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d0c19a680 0x7f8d0c19ab00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c970 tx=0x7f8d0000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.145+0000 7f8d0a000640 1 -- 172.21.15.28:0/1465114599 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00007bb0 con 0x7f8d0c19a680 2024-04-26T01:43:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.145+0000 7f8d0a000640 1 -- 172.21.15.28:0/1465114599 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d00007d50 con 0x7f8d0c19a680 2024-04-26T01:43:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.145+0000 7f8d0a000640 1 -- 172.21.15.28:0/1465114599 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00018660 con 0x7f8d0c19a680 2024-04-26T01:43:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.145+0000 7f8d14200640 1 -- 172.21.15.28:0/1465114599 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0c1a7360 con 0x7f8d0c19a680 2024-04-26T01:43:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.146+0000 7f8d14200640 1 -- 172.21.15.28:0/1465114599 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0c1a7710 con 0x7f8d0c19a680 2024-04-26T01:43:40.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.151+0000 7f8d16ff6740 1 Processor -- start 2024-04-26T01:43:40.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.157+0000 7f8d0a000640 1 -- 172.21.15.28:0/1465114599 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8d0000f450 con 0x7f8d0c19a680 2024-04-26T01:43:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.158+0000 7f8d0a000640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8cf40a3ff0 0x7f8cf40a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.159+0000 7f8d0a000640 1 -- 172.21.15.28:0/1465114599 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f8d000d0660 con 0x7f8d0c19a680 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.159+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8cf40a3ff0 0x7f8cf40a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.159+0000 7f8d16ff6740 1 -- start start 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b776a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a950 0x5641b7774cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b7777600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d11800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b776a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d11800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b776a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:42404/0 (socket says 172.21.15.28:42404) 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641b76c7900 con 0x5641b776a950 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5641b766def0 con 0x5641b7775210 2024-04-26T01:43:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5641b76c38c0 con 0x5641b776a030 2024-04-26T01:43:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.160+0000 7f8d11800640 1 -- 172.21.15.28:0/1271053340 learned_addr learned my addr 172.21.15.28:0/1271053340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.161+0000 7f8d12200640 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b7777600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.161+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8cf40a3ff0 0x7f8cf40a64b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c19bed0 tx=0x7f8d04009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.161+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a950 0x5641b7774cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.161+0000 7f8d12200640 1 -- 172.21.15.28:0/1271053340 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 msgr2=0x5641b776a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.161+0000 7f8d12200640 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b776a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.161+0000 7f8d12200640 1 -- 172.21.15.28:0/1271053340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a950 msgr2=0x5641b7774cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.161+0000 7f8d12200640 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a950 0x5641b7774cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.161+0000 7f8d12200640 1 -- 172.21.15.28:0/1271053340 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641b73da210 con 0x5641b7775210 2024-04-26T01:43:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.162+0000 7f8d12200640 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b7777600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8d000cdd80 tx=0x7f8d0009c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2563f5f0f5fa8c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.162+0000 7f8cfea00640 1 -- 172.21.15.28:0/1271053340 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0009c5d0 con 0x5641b7775210 2024-04-26T01:43:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.162+0000 7f8cfea00640 1 -- 172.21.15.28:0/1271053340 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d000a0980 con 0x5641b7775210 2024-04-26T01:43:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.163+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1271053340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 msgr2=0x5641b7777600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.163+0000 7f8cfea00640 1 -- 172.21.15.28:0/1271053340 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0009c5d0 con 0x5641b7775210 2024-04-26T01:43:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.163+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b7777600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8d000cdd80 tx=0x7f8d0009c040 comp rx=0 tx=0).stop 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.164+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1271053340 shutdown_connections 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.164+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b7777600 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.164+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a950 0x5641b7774cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.164+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1271053340 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b776a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.164+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1271053340 >> 172.21.15.28:0/1271053340 conn(0x5641b7766540 msgr2=0x5641b7766920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.164+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1271053340 shutdown_connections 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.164+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1271053340 wait complete. 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.164+0000 7f8d16ff6740 1 -- start start 2024-04-26T01:43:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.165+0000 7f8d16ff6740 1 Processor -- start 2024-04-26T01:43:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b776a030 0x5641b77720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a950 0x5641b7772610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b7772b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d10e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a950 0x5641b7772610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d10e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a950 0x5641b7772610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:42408/0 (socket says 172.21.15.28:42408) 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d11800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b776a030 0x5641b77720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d11800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b776a030 0x5641b77720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47378/0 (socket says 172.21.15.28:47378) 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d10e00640 1 -- 172.21.15.28:0/1351489969 learned_addr learned my addr 172.21.15.28:0/1351489969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641b7777e40 con 0x5641b7775210 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5641b7681310 con 0x5641b776a030 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5641b7777fc0 con 0x5641b776a950 2024-04-26T01:43:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d16ff6740 10 client.0 _ll_register_callbacks cb 0x5641b764fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.166+0000 7f8d12200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b7772b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.167+0000 7f8d10e00640 1 -- 172.21.15.28:0/1351489969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b776a030 msgr2=0x5641b77720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.167+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b776a030 0x5641b77720d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.167+0000 7f8d10e00640 1 -- 172.21.15.28:0/1351489969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 msgr2=0x5641b7772b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.167+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b7772b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.167+0000 7f8d10e00640 1 -- 172.21.15.28:0/1351489969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641b74a6620 con 0x5641b776a950 2024-04-26T01:43:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.167+0000 7f8d11800640 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b776a030 0x5641b77720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:43:40.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.168+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a950 0x5641b7772610 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8d04011ce0 tx=0x7f8d04004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a35892158a6144f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.168+0000 7f8cf3400640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:43:40.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.168+0000 7f8cf3400640 1 -- 172.21.15.28:0/1351489969 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04028430 con 0x5641b776a950 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.168+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d000a0980 con 0x5641b776a950 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.169+0000 7f8cf3400640 1 -- 172.21.15.28:0/1351489969 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f8d0401f430 con 0x5641b776a950 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.169+0000 7f8cf3400640 1 -- 172.21.15.28:0/1351489969 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d0401f630 con 0x5641b776a950 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.169+0000 7f8d16ff6740 10 client.25501 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.169+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5641b7783f00 con 0x5641b776a950 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.169+0000 7f8cf3400640 1 -- 172.21.15.28:0/1351489969 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d000a0980 con 0x5641b776a950 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.170+0000 7f8cf3400640 1 -- 172.21.15.28:0/1351489969 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f8d04028a50 con 0x5641b776a950 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.170+0000 7f8d16ff6740 10 client.25501 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.170+0000 7f8d16ff6740 10 client.25501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.170+0000 7f8d16ff6740 20 client.25501 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.170+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5641b7785430 0x5641b7787840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.170+0000 7f8d16ff6740 4 client.25501 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.170+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641b76db8b0 con 0x5641b7785430 2024-04-26T01:43:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.170+0000 7f8d11800640 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5641b7785430 0x5641b7787840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.171+0000 7f8d11800640 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5641b7785430 0x5641b7787840 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a09b561ae9f92517 server_cookie=3d6763b727ea7bd4 in_seq=0 out_seq=0 2024-04-26T01:43:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.171+0000 7f8cf3400640 10 client.25501 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8cf3400640 1 -- 172.21.15.28:0/1351489969 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5641b76db8b0 con 0x5641b7785430 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8cf3400640 10 client.25501 handle_command_reply: tid=0 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 client.25501 shutdown 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a950 msgr2=0x5641b7772610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a950 0x5641b7772610 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8d04011ce0 tx=0x7f8d04004ab0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 shutdown_connections 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5641b7785430 0x5641b7787840 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b7772b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a950 0x5641b7772610 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8cf3400640 0 client.25501 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1351489969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b776a030 0x5641b77720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 >> 172.21.15.28:0/1351489969 conn(0x5641b7766540 msgr2=0x5641b775ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 shutdown_connections 2024-04-26T01:43:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.172+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1351489969 wait complete. 2024-04-26T01:43:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.173+0000 7f8d16ff6740 20 client.25501 trim_cache size 0 max 16384 2024-04-26T01:43:40.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.202+0000 7f8d16ff6740 1 Processor -- start 2024-04-26T01:43:40.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.203+0000 7f8d16ff6740 1 -- start start 2024-04-26T01:43:40.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.203+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a030 0x5641b7780d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.203+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b776c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.203+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b7774910 0x5641b778d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.203+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1ae100 con 0x5641b776a030 2024-04-26T01:43:40.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.203+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1ab9a0 con 0x5641b7775210 2024-04-26T01:43:40.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.204+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1a9810 con 0x5641b7774910 2024-04-26T01:43:40.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.204+0000 7f8d10e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b776c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.205+0000 7f8d10e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b776c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47384/0 (socket says 172.21.15.28:47384) 2024-04-26T01:43:40.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.205+0000 7f8d12200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b7774910 0x5641b778d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.205+0000 7f8d10e00640 1 -- 172.21.15.28:0/4205610133 learned_addr learned my addr 172.21.15.28:0/4205610133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:40.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.205+0000 7f8d11800640 1 --2- 172.21.15.28:0/4205610133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a030 0x5641b7780d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.206+0000 7f8d11800640 1 -- 172.21.15.28:0/4205610133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b7774910 msgr2=0x5641b778d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.206+0000 7f8d11800640 1 --2- 172.21.15.28:0/4205610133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b7774910 0x5641b778d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.206+0000 7f8d11800640 1 -- 172.21.15.28:0/4205610133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 msgr2=0x5641b776c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.207+0000 7f8d11800640 1 --2- 172.21.15.28:0/4205610133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b776c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.207+0000 7f8d11800640 1 -- 172.21.15.28:0/4205610133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641b7787ec0 con 0x5641b776a030 2024-04-26T01:43:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.207+0000 7f8d11800640 1 --2- 172.21.15.28:0/4205610133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a030 0x5641b7780d50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8005ab0 tx=0x7f8cf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e92d574b9ad564e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.208+0000 7f8cfe000640 1 -- 172.21.15.28:0/4205610133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8035040 con 0x5641b776a030 2024-04-26T01:43:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.208+0000 7f8cfe000640 1 -- 172.21.15.28:0/4205610133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cf8005020 con 0x5641b776a030 2024-04-26T01:43:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.208+0000 7f8cfe000640 1 -- 172.21.15.28:0/4205610133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf80446c0 con 0x5641b776a030 2024-04-26T01:43:40.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.209+0000 7f8d16ff6740 1 -- 172.21.15.28:0/4205610133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a030 msgr2=0x5641b7780d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.209+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/4205610133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a030 0x5641b7780d50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8005ab0 tx=0x7f8cf80046c0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.210+0000 7f8d16ff6740 1 -- 172.21.15.28:0/4205610133 shutdown_connections 2024-04-26T01:43:40.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.210+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/4205610133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b7774910 0x5641b778d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.210+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/4205610133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7775210 0x5641b776c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.210+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/4205610133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b776a030 0x5641b7780d50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.210+0000 7f8d16ff6740 1 -- 172.21.15.28:0/4205610133 >> 172.21.15.28:0/4205610133 conn(0x5641b7766540 msgr2=0x5641b7766920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.211+0000 7f8d16ff6740 1 -- 172.21.15.28:0/4205610133 shutdown_connections 2024-04-26T01:43:40.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.212+0000 7f8d16ff6740 1 -- 172.21.15.28:0/4205610133 wait complete. 2024-04-26T01:43:40.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.212+0000 7f8d16ff6740 1 -- start start 2024-04-26T01:43:40.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.213+0000 7f8d16ff6740 1 Processor -- start 2024-04-26T01:43:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.219+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b77e4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.219+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7774910 0x5641b77e4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.220+0000 7f8d16ff6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b777b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.220+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1a9e60 con 0x5641b7775210 2024-04-26T01:43:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.220+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1ab2d0 con 0x5641b7774910 2024-04-26T01:43:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.220+0000 7f8d16ff6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1aa530 con 0x5641b776a030 2024-04-26T01:43:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.220+0000 7f8d16ff6740 10 client.0 _ll_register_callbacks cb 0x7f8d04011990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:40.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.220+0000 7f8d11800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b77e4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.220+0000 7f8d10e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7774910 0x5641b77e4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.221+0000 7f8d10e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7774910 0x5641b77e4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47390/0 (socket says 172.21.15.28:47390) 2024-04-26T01:43:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.221+0000 7f8d10e00640 1 -- 172.21.15.28:0/1084259509 learned_addr learned my addr 172.21.15.28:0/1084259509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.221+0000 7f8d12200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b777b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.221+0000 7f8d10e00640 1 -- 172.21.15.28:0/1084259509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 msgr2=0x5641b77e4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.222+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b77e4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.223+0000 7f8d10e00640 1 -- 172.21.15.28:0/1084259509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 msgr2=0x5641b777b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.223+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b777b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.223+0000 7f8d10e00640 1 -- 172.21.15.28:0/1084259509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641b777be60 con 0x5641b7774910 2024-04-26T01:43:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.225+0000 7f8d12200640 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b777b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.224+0000 7f8d11800640 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b77e4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:43:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.226+0000 7f8d10e00640 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7774910 0x5641b77e4ff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009a70 tx=0x7f8d0401eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a258848c39a721 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.226+0000 7f8cf3400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:43:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.226+0000 7f8cf3400640 1 -- 172.21.15.28:0/1084259509 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04040060 con 0x5641b7774910 2024-04-26T01:43:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.227+0000 7f8cf3400640 1 -- 172.21.15.28:0/1084259509 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f8d0401f9d0 con 0x5641b7774910 2024-04-26T01:43:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.227+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1084259509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cf8005020 con 0x5641b7774910 2024-04-26T01:43:40.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.228+0000 7f8cf3400640 1 -- 172.21.15.28:0/1084259509 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d04035040 con 0x5641b7774910 2024-04-26T01:43:40.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.228+0000 7f8d16ff6740 10 client.25478 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:40.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.228+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1084259509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5641b7763f60 con 0x5641b7774910 2024-04-26T01:43:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.229+0000 7f8cf3400640 1 -- 172.21.15.28:0/1084259509 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cf8005020 con 0x5641b7774910 2024-04-26T01:43:40.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.229+0000 7f8cf3400640 1 -- 172.21.15.28:0/1084259509 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f8d04038030 con 0x5641b7774910 2024-04-26T01:43:40.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.229+0000 7f8d16ff6740 10 client.25478 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.229+0000 7f8d16ff6740 10 client.25478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.229+0000 7f8d16ff6740 20 client.25478 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.229+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5641b77646a0 0x5641b7764ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.229+0000 7f8d16ff6740 4 client.25478 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:43:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.229+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1084259509 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5641b76d5320 con 0x5641b77646a0 2024-04-26T01:43:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.230+0000 7f8d11800640 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5641b77646a0 0x5641b7764ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.233+0000 7f8d11800640 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5641b77646a0 0x5641b7764ab0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38842389f462a129 server_cookie=eaeca4e380cbd75f in_seq=0 out_seq=0 2024-04-26T01:43:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.233+0000 7f8cf3400640 10 client.25478 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:40.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.245+0000 7f8cf3400640 1 -- 172.21.15.28:0/1084259509 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5641b76d5320 con 0x5641b77646a0 2024-04-26T01:43:40.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.245+0000 7f8cf3400640 10 client.25478 handle_command_reply: tid=0 2024-04-26T01:43:40.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.245+0000 7f8d16ff6740 1 client.25478 shutdown 2024-04-26T01:43:40.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1084259509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7774910 msgr2=0x5641b77e4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7774910 0x5641b77e4ff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009a70 tx=0x7f8d0401eed0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1084259509 shutdown_connections 2024-04-26T01:43:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5641b77646a0 0x5641b7764ab0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641b7775210 0x5641b777b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5641b7774910 0x5641b77e4ff0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8d16ff6740 1 --2- 172.21.15.28:0/1084259509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5641b776a030 0x5641b77e4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8cf3400640 0 client.25478 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.246+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1084259509 >> 172.21.15.28:0/1084259509 conn(0x5641b7766540 msgr2=0x5641b775c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.247+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1084259509 shutdown_connections 2024-04-26T01:43:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.247+0000 7f8d16ff6740 1 -- 172.21.15.28:0/1084259509 wait complete. 2024-04-26T01:43:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.247+0000 7f8d16ff6740 20 client.25478 trim_cache size 0 max 16384 2024-04-26T01:43:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:43:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:43:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:43:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:43:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 -- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8cf40a3ff0 msgr2=0x7f8cf40a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8cf40a3ff0 0x7f8cf40a64b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c19bed0 tx=0x7f8d04009500 comp rx=0 tx=0).stop 2024-04-26T01:43:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 -- 172.21.15.28:0/1465114599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d0c19a680 msgr2=0x7f8d0c19ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d0c19a680 0x7f8d0c19ab00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c970 tx=0x7f8d0000ce30 comp rx=0 tx=0).stop 2024-04-26T01:43:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 -- 172.21.15.28:0/1465114599 shutdown_connections 2024-04-26T01:43:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8cf40a3ff0 0x7f8cf40a64b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d0c19a680 0x7f8d0c19ab00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c109470 0x7f8d0c1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 --2- 172.21.15.28:0/1465114599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d0c1043a0 0x7f8d0c1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 -- 172.21.15.28:0/1465114599 >> 172.21.15.28:0/1465114599 conn(0x7f8d0c0ffc40 msgr2=0x7f8d0c100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.248+0000 7f8d14200640 1 -- 172.21.15.28:0/1465114599 shutdown_connections 2024-04-26T01:43:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:40.249+0000 7f8d14200640 1 -- 172.21.15.28:0/1465114599 wait complete. 2024-04-26T01:43:41.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.125+0000 7ff322c00700 1 -- 172.21.15.28:0/1095623244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff31c1105a0 msgr2=0x7ff31c118170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.125+0000 7ff322c00700 1 --2- 172.21.15.28:0/1095623244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff31c1105a0 0x7ff31c118170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fc0 tx=0x7ff31400b330 comp rx=0 tx=0).stop 2024-04-26T01:43:41.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.126+0000 7ff322c00700 1 -- 172.21.15.28:0/1095623244 shutdown_connections 2024-04-26T01:43:41.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.126+0000 7ff322c00700 1 --2- 172.21.15.28:0/1095623244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff31c1105a0 0x7ff31c118170 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.126+0000 7ff322c00700 1 --2- 172.21.15.28:0/1095623244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff31c10b510 0x7ff31c10b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.126+0000 7ff322c00700 1 --2- 172.21.15.28:0/1095623244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff31c10bec0 0x7ff31c10fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.126+0000 7ff322c00700 1 -- 172.21.15.28:0/1095623244 >> 172.21.15.28:0/1095623244 conn(0x7ff31c06fbe0 msgr2=0x7ff31c109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.126+0000 7ff322c00700 1 -- 172.21.15.28:0/1095623244 shutdown_connections 2024-04-26T01:43:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.126+0000 7ff322c00700 1 -- 172.21.15.28:0/1095623244 wait complete. 2024-04-26T01:43:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.128+0000 7ff322c00700 1 Processor -- start 2024-04-26T01:43:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.128+0000 7ff322c00700 1 -- start start 2024-04-26T01:43:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.128+0000 7ff322c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff31c10b510 0x7ff31c1a7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.128+0000 7ff322c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff31c10bec0 0x7ff31c1a8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.128+0000 7ff322c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff31c1105a0 0x7ff31c1ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.129+0000 7ff31be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff31c10b510 0x7ff31c1a7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.129+0000 7ff322c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c121a80 con 0x7ff31c10b510 2024-04-26T01:43:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.129+0000 7ff31b400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff31c10bec0 0x7ff31c1a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.129+0000 7ff322c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c121900 con 0x7ff31c10bec0 2024-04-26T01:43:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.129+0000 7ff322c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c121c00 con 0x7ff31c1105a0 2024-04-26T01:43:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.129+0000 7ff31b400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff31c10bec0 0x7ff31c1a8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47406/0 (socket says 172.21.15.28:47406) 2024-04-26T01:43:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.129+0000 7ff31b400700 1 -- 172.21.15.28:0/1044835848 learned_addr learned my addr 172.21.15.28:0/1044835848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.130+0000 7ff320c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff31c1105a0 0x7ff31c1ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.130+0000 7ff31b400700 1 -- 172.21.15.28:0/1044835848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff31c1105a0 msgr2=0x7ff31c1ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.130+0000 7ff31b400700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff31c1105a0 0x7ff31c1ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.130+0000 7ff31b400700 1 -- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff31c10b510 msgr2=0x7ff31c1a7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.131+0000 7ff31b400700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff31c10b510 0x7ff31c1a7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.131+0000 7ff31b400700 1 -- 172.21.15.28:0/1044835848 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31c1acdf0 con 0x7ff31c10bec0 2024-04-26T01:43:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.131+0000 7ff31be00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff31c10b510 0x7ff31c1a7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.132+0000 7ff31b400700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff31c10bec0 0x7ff31c1a8370 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff30800ec70 tx=0x7ff30800ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.133+0000 7ff318c00700 1 -- 172.21.15.28:0/1044835848 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30800cd40 con 0x7ff31c10bec0 2024-04-26T01:43:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.133+0000 7ff318c00700 1 -- 172.21.15.28:0/1044835848 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3080044d0 con 0x7ff31c10bec0 2024-04-26T01:43:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.133+0000 7ff318c00700 1 -- 172.21.15.28:0/1044835848 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308010690 con 0x7ff31c10bec0 2024-04-26T01:43:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.133+0000 7ff322c00700 1 -- 172.21.15.28:0/1044835848 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff31c1ad0e0 con 0x7ff31c10bec0 2024-04-26T01:43:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.133+0000 7ff322c00700 1 -- 172.21.15.28:0/1044835848 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff31c1b4960 con 0x7ff31c10bec0 2024-04-26T01:43:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.140+0000 7ff318c00700 1 -- 172.21.15.28:0/1044835848 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff3080037a0 con 0x7ff31c10bec0 2024-04-26T01:43:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.142+0000 7ff32d550b80 1 Processor -- start 2024-04-26T01:43:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.145+0000 7ff32d550b80 1 -- start start 2024-04-26T01:43:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.146+0000 7ff318c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff3000a3ed0 0x7ff3000a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.147+0000 7ff318c00700 1 -- 172.21.15.28:0/1044835848 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7ff3080d0a10 con 0x7ff31c10bec0 2024-04-26T01:43:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.147+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3e9b0 0x55932ab3ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.147+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab49c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.147+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab4c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.147+0000 7ff32d550b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55932aa92020 con 0x55932ab3f2d0 2024-04-26T01:43:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.147+0000 7ff32d550b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55932aa388c0 con 0x55932ab4a180 2024-04-26T01:43:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.147+0000 7ff32d550b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55932aa8dfe0 con 0x55932ab3e9b0 2024-04-26T01:43:41.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.148+0000 7ff31b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab49c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.149+0000 7ff31be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3e9b0 0x55932ab3ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.149+0000 7ff31be00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3e9b0 0x55932ab3ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:42480/0 (socket says 172.21.15.28:42480) 2024-04-26T01:43:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.149+0000 7ff31be00700 1 -- 172.21.15.28:0/1381960065 learned_addr learned my addr 172.21.15.28:0/1381960065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.149+0000 7ff320c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff3000a3ed0 0x7ff3000a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.149+0000 7ff320c00700 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab4c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.150+0000 7ff31b400700 1 -- 172.21.15.28:0/1381960065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3e9b0 msgr2=0x55932ab3ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.150+0000 7ff31b400700 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3e9b0 0x55932ab3ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.150+0000 7ff31b400700 1 -- 172.21.15.28:0/1381960065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 msgr2=0x55932ab4c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.150+0000 7ff31b400700 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab4c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.150+0000 7ff31b400700 1 -- 172.21.15.28:0/1381960065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55932aa2e810 con 0x55932ab3f2d0 2024-04-26T01:43:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.150+0000 7ff320c00700 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab4c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:43:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.150+0000 7ff31be00700 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3e9b0 0x55932ab3ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.151+0000 7ff31b400700 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab49c40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff3080ce190 tx=0x7ff308098c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a09a913b851e135 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.151+0000 7ff320c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff3000a3ed0 0x7ff3000a6390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff3140060b0 tx=0x7ff314006020 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.152+0000 7ff311600700 1 -- 172.21.15.28:0/1381960065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30809c340 con 0x55932ab3f2d0 2024-04-26T01:43:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.152+0000 7ff311600700 1 -- 172.21.15.28:0/1381960065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff30809d0e0 con 0x55932ab3f2d0 2024-04-26T01:43:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.152+0000 7ff311600700 1 -- 172.21.15.28:0/1381960065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30809cc40 con 0x55932ab3f2d0 2024-04-26T01:43:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.153+0000 7ff32d550b80 1 -- 172.21.15.28:0/1381960065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 msgr2=0x55932ab49c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.153+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab49c40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff3080ce190 tx=0x7ff308098c80 comp rx=0 tx=0).stop 2024-04-26T01:43:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.153+0000 7ff32d550b80 1 -- 172.21.15.28:0/1381960065 shutdown_connections 2024-04-26T01:43:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.153+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab4c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.153+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3e9b0 0x55932ab3ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.153+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1381960065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab49c40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.153+0000 7ff32d550b80 1 -- 172.21.15.28:0/1381960065 >> 172.21.15.28:0/1381960065 conn(0x55932aa6d730 msgr2=0x55932ab3ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.154+0000 7ff32d550b80 1 -- 172.21.15.28:0/1381960065 shutdown_connections 2024-04-26T01:43:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.154+0000 7ff32d550b80 1 -- 172.21.15.28:0/1381960065 wait complete. 2024-04-26T01:43:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.155+0000 7ff32d550b80 1 -- start start 2024-04-26T01:43:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.155+0000 7ff32d550b80 1 Processor -- start 2024-04-26T01:43:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.157+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.157+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3f2d0 0x55932ab46bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.157+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab470f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.157+0000 7ff31be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff320c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff31be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35198/0 (socket says 172.21.15.28:35198) 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff320c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:42488/0 (socket says 172.21.15.28:42488) 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff31b400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3f2d0 0x55932ab46bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff31b400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3f2d0 0x55932ab46bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47432/0 (socket says 172.21.15.28:47432) 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff32d550b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55932ab4cd00 con 0x55932ab3e9b0 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff32d550b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55932aa4b9f0 con 0x55932ab3f2d0 2024-04-26T01:43:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff32d550b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55932ab4ce80 con 0x55932ab4a180 2024-04-26T01:43:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.158+0000 7ff31be00700 1 -- 172.21.15.28:0/1805849559 learned_addr learned my addr 172.21.15.28:0/1805849559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.159+0000 7ff32d550b80 10 client.0 _ll_register_callbacks cb 0x55932aa2a8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.159+0000 7ff320c00700 1 -- 172.21.15.28:0/1805849559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3f2d0 msgr2=0x55932ab46bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.159+0000 7ff320c00700 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3f2d0 0x55932ab46bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.159+0000 7ff320c00700 1 -- 172.21.15.28:0/1805849559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 msgr2=0x55932ab498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.159+0000 7ff320c00700 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.159+0000 7ff320c00700 1 -- 172.21.15.28:0/1805849559 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55932a9dfa60 con 0x55932ab4a180 2024-04-26T01:43:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.160+0000 7ff31b400700 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3f2d0 0x55932ab46bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.160+0000 7ff31be00700 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:43:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.161+0000 7ff320c00700 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab470f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff31400b6c0 tx=0x7ff314009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56119ef3656febc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.161+0000 7ff306000700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:43:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.162+0000 7ff306000700 1 -- 172.21.15.28:0/1805849559 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31402e030 con 0x55932ab4a180 2024-04-26T01:43:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.162+0000 7ff306000700 1 -- 172.21.15.28:0/1805849559 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7ff314004e00 con 0x55932ab4a180 2024-04-26T01:43:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.162+0000 7ff32d550b80 1 -- 172.21.15.28:0/1805849559 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff30809d0e0 con 0x55932ab4a180 2024-04-26T01:43:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.163+0000 7ff306000700 1 -- 172.21.15.28:0/1805849559 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff314023070 con 0x55932ab4a180 2024-04-26T01:43:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.163+0000 7ff32d550b80 10 client.25516 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.163+0000 7ff32d550b80 1 -- 172.21.15.28:0/1805849559 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55932aa28c80 con 0x55932ab4a180 2024-04-26T01:43:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.163+0000 7ff306000700 1 -- 172.21.15.28:0/1805849559 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff30809d0e0 con 0x55932ab4a180 2024-04-26T01:43:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.163+0000 7ff306000700 1 -- 172.21.15.28:0/1805849559 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7ff314011070 con 0x55932ab4a180 2024-04-26T01:43:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.164+0000 7ff32d550b80 10 client.25516 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.164+0000 7ff32d550b80 10 client.25516 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:43:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.164+0000 7ff32d550b80 20 client.25516 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.164+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55932ab59a30 0x55932ab5be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.164+0000 7ff32d550b80 4 client.25516 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.164+0000 7ff32d550b80 1 -- 172.21.15.28:0/1805849559 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55932aa9f810 con 0x55932ab59a30 2024-04-26T01:43:41.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.164+0000 7ff31be00700 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55932ab59a30 0x55932ab5be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.165+0000 7ff31be00700 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55932ab59a30 0x55932ab5be80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7830981c9be45144 server_cookie=5575ecbe8c3d48d3 in_seq=0 out_seq=0 2024-04-26T01:43:41.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.165+0000 7ff306000700 10 client.25516 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff306000700 1 -- 172.21.15.28:0/1805849559 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55932aa9f810 con 0x55932ab59a30 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff306000700 10 client.25516 handle_command_reply: tid=0 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 client.25516 shutdown 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 -- 172.21.15.28:0/1805849559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 msgr2=0x55932ab470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab470f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff31400b6c0 tx=0x7ff314009f10 comp rx=0 tx=0).stop 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 -- 172.21.15.28:0/1805849559 shutdown_connections 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55932ab59a30 0x55932ab5be80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab470f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 --2- 172.21.15.28:0/1805849559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3f2d0 0x55932ab46bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff32d550b80 1 -- 172.21.15.28:0/1805849559 >> 172.21.15.28:0/1805849559 conn(0x55932aa6d730 msgr2=0x55932ab3aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:41.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.166+0000 7ff306000700 0 client.25516 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:41.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.167+0000 7ff32d550b80 1 -- 172.21.15.28:0/1805849559 shutdown_connections 2024-04-26T01:43:41.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.167+0000 7ff32d550b80 1 -- 172.21.15.28:0/1805849559 wait complete. 2024-04-26T01:43:41.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.167+0000 7ff32d550b80 20 client.25516 trim_cache size 0 max 16384 2024-04-26T01:43:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.198+0000 7ff32d550b80 1 Processor -- start 2024-04-26T01:43:41.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.198+0000 7ff32d550b80 1 -- start start 2024-04-26T01:43:41.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.198+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab55c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.198+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3f2d0 0x55932ab48af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.198+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab49030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff32d550b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c1be920 con 0x55932ab3e9b0 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff32d550b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c1bc180 con 0x55932ab4a180 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff32d550b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c1b9fd0 con 0x55932ab3f2d0 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31b400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3f2d0 0x55932ab48af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31b400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3f2d0 0x55932ab48af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:42494/0 (socket says 172.21.15.28:42494) 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31b400700 1 -- 172.21.15.28:0/2332010593 learned_addr learned my addr 172.21.15.28:0/2332010593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31be00700 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab55c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff320c00700 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab49030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31b400700 1 -- 172.21.15.28:0/2332010593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 msgr2=0x55932ab49030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31b400700 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab49030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31b400700 1 -- 172.21.15.28:0/2332010593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 msgr2=0x55932ab55c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31b400700 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab55c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31b400700 1 -- 172.21.15.28:0/2332010593 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55932ab48150 con 0x55932ab3f2d0 2024-04-26T01:43:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.199+0000 7ff31be00700 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab55c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:43:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.200+0000 7ff31b400700 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3f2d0 0x55932ab48af0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff308099360 tx=0x7ff308000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=775dff391ac7b429 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.200+0000 7ff310c00700 1 -- 172.21.15.28:0/2332010593 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308080070 con 0x55932ab3f2d0 2024-04-26T01:43:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.200+0000 7ff310c00700 1 -- 172.21.15.28:0/2332010593 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff30809d0e0 con 0x55932ab3f2d0 2024-04-26T01:43:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.201+0000 7ff310c00700 1 -- 172.21.15.28:0/2332010593 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30809cc40 con 0x55932ab3f2d0 2024-04-26T01:43:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.202+0000 7ff32d550b80 1 -- 172.21.15.28:0/2332010593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3f2d0 msgr2=0x55932ab48af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.202+0000 7ff32d550b80 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3f2d0 0x55932ab48af0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff308099360 tx=0x7ff308000bc0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.202+0000 7ff32d550b80 1 -- 172.21.15.28:0/2332010593 shutdown_connections 2024-04-26T01:43:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.203+0000 7ff32d550b80 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab4a180 0x55932ab49030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.203+0000 7ff32d550b80 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3e9b0 0x55932ab55c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.203+0000 7ff32d550b80 1 --2- 172.21.15.28:0/2332010593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab3f2d0 0x55932ab48af0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.203+0000 7ff32d550b80 1 -- 172.21.15.28:0/2332010593 >> 172.21.15.28:0/2332010593 conn(0x55932aa6d730 msgr2=0x55932ab483e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.204+0000 7ff32d550b80 1 -- 172.21.15.28:0/2332010593 shutdown_connections 2024-04-26T01:43:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.204+0000 7ff32d550b80 1 -- 172.21.15.28:0/2332010593 wait complete. 2024-04-26T01:43:41.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.205+0000 7ff32d550b80 1 -- start start 2024-04-26T01:43:41.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.206+0000 7ff32d550b80 1 Processor -- start 2024-04-26T01:43:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3e9b0 0x55932ab52520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab4e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff32d550b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab4edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff32d550b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c1ba640 con 0x55932ab3f2d0 2024-04-26T01:43:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff32d550b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c1bbab0 con 0x55932ab3e9b0 2024-04-26T01:43:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff32d550b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c1bad10 con 0x55932ab4a180 2024-04-26T01:43:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff32d550b80 10 client.0 _ll_register_callbacks cb 0x55932aa2a8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff31be00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3e9b0 0x55932ab52520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff31be00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3e9b0 0x55932ab52520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47454/0 (socket says 172.21.15.28:47454) 2024-04-26T01:43:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff31be00700 1 -- 172.21.15.28:0/3949681592 learned_addr learned my addr 172.21.15.28:0/3949681592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff31b400700 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab4e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff31be00700 1 -- 172.21.15.28:0/3949681592 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 msgr2=0x55932ab4edf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:43:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff31be00700 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab4edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.210+0000 7ff31be00700 1 -- 172.21.15.28:0/3949681592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 msgr2=0x55932ab4e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.211+0000 7ff31be00700 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab4e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.211+0000 7ff31be00700 1 -- 172.21.15.28:0/3949681592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55932ab4f600 con 0x55932ab3e9b0 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.211+0000 7ff31b400700 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab4e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.211+0000 7ff31be00700 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3e9b0 0x55932ab52520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff30c007b70 tx=0x7ff30c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=968c4f2fd116d9d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.211+0000 7ff306000700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.211+0000 7ff306000700 1 -- 172.21.15.28:0/3949681592 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c004520 con 0x55932ab3e9b0 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.211+0000 7ff306000700 1 -- 172.21.15.28:0/3949681592 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7ff30c003680 con 0x55932ab3e9b0 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.212+0000 7ff32d550b80 1 -- 172.21.15.28:0/3949681592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff30809d0e0 con 0x55932ab3e9b0 2024-04-26T01:43:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.212+0000 7ff306000700 1 -- 172.21.15.28:0/3949681592 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff30c02d050 con 0x55932ab3e9b0 2024-04-26T01:43:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.212+0000 7ff32d550b80 10 client.25496 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.212+0000 7ff32d550b80 1 -- 172.21.15.28:0/3949681592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55932abc0fc0 con 0x55932ab3e9b0 2024-04-26T01:43:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.213+0000 7ff306000700 1 -- 172.21.15.28:0/3949681592 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff30809d0e0 con 0x55932ab3e9b0 2024-04-26T01:43:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.213+0000 7ff306000700 1 -- 172.21.15.28:0/3949681592 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7ff30c010d20 con 0x55932ab3e9b0 2024-04-26T01:43:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.213+0000 7ff32d550b80 10 client.25496 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.213+0000 7ff32d550b80 10 client.25496 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:43:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.213+0000 7ff32d550b80 20 client.25496 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.213+0000 7ff32d550b80 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55932abc1770 0x55932abc3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.213+0000 7ff32d550b80 4 client.25496 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-26T01:43:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.213+0000 7ff32d550b80 1 -- 172.21.15.28:0/3949681592 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x55932aa998b0 con 0x55932abc1770 2024-04-26T01:43:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.215+0000 7ff31b400700 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55932abc1770 0x55932abc3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.217+0000 7ff306000700 10 client.25496 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.217+0000 7ff31b400700 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55932abc1770 0x55932abc3bb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b7e9380b8120425 server_cookie=da3598b680a6f892 in_seq=0 out_seq=0 2024-04-26T01:43:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 105.0 s, 3098.1 tps, lat 32.426 ms stddev 34.977 2024-04-26T01:43:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff306000700 1 -- 172.21.15.28:0/3949681592 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55932aa998b0 con 0x55932abc1770 2024-04-26T01:43:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff306000700 10 client.25496 handle_command_reply: tid=0 2024-04-26T01:43:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 client.25496 shutdown 2024-04-26T01:43:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:41 smithi028 ceph-mon[28247]: pgmap v84: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 943 op/s 2024-04-26T01:43:41.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 -- 172.21.15.28:0/3949681592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3e9b0 msgr2=0x55932ab52520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3e9b0 0x55932ab52520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff30c007b70 tx=0x7ff30c0117b0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 -- 172.21.15.28:0/3949681592 shutdown_connections 2024-04-26T01:43:41.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55932abc1770 0x55932abc3bb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55932ab4a180 0x55932ab4edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55932ab3e9b0 0x55932ab52520 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 --2- 172.21.15.28:0/3949681592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55932ab3f2d0 0x55932ab4e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff306000700 0 client.25496 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:41.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.378+0000 7ff32d550b80 1 -- 172.21.15.28:0/3949681592 >> 172.21.15.28:0/3949681592 conn(0x55932aa6d730 msgr2=0x55932ab483e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:41.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.379+0000 7ff32d550b80 1 -- 172.21.15.28:0/3949681592 shutdown_connections 2024-04-26T01:43:41.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.379+0000 7ff32d550b80 1 -- 172.21.15.28:0/3949681592 wait complete. 2024-04-26T01:43:41.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.379+0000 7ff32d550b80 20 client.25496 trim_cache size 0 max 16384 2024-04-26T01:43:41.378 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 28, 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 6, 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "b7020b43": { 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 6, 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 60.0, 2024-04-26T01:43:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 90.0, 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:43:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:43:41.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 -- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff3000a3ed0 msgr2=0x7ff3000a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff3000a3ed0 0x7ff3000a6390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff3140060b0 tx=0x7ff314006020 comp rx=0 tx=0).stop 2024-04-26T01:43:41.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 -- 172.21.15.28:0/1044835848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff31c10bec0 msgr2=0x7ff31c1a8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:41.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff31c10bec0 0x7ff31c1a8370 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff30800ec70 tx=0x7ff30800ef80 comp rx=0 tx=0).stop 2024-04-26T01:43:41.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 -- 172.21.15.28:0/1044835848 shutdown_connections 2024-04-26T01:43:41.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff3000a3ed0 0x7ff3000a6390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff31c1105a0 0x7ff31c1ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff31c10b510 0x7ff31c1a7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 --2- 172.21.15.28:0/1044835848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff31c10bec0 0x7ff31c1a8370 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.380+0000 7ff322c00700 1 -- 172.21.15.28:0/1044835848 >> 172.21.15.28:0/1044835848 conn(0x7ff31c06fbe0 msgr2=0x7ff31c070060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.381+0000 7ff322c00700 1 -- 172.21.15.28:0/1044835848 shutdown_connections 2024-04-26T01:43:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:43:41.381+0000 7ff322c00700 1 -- 172.21.15.28:0/1044835848 wait complete. 2024-04-26T01:43:41.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:41 smithi169 ceph-mon[32851]: pgmap v84: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 943 op/s 2024-04-26T01:43:41.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:41 smithi053 ceph-mon[32593]: pgmap v84: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 943 op/s 2024-04-26T01:43:42.351 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b7020b43', quiesce took 9.8 seconds. Will release after: 30.0 2024-04-26T01:43:43.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:43 smithi169 ceph-mon[32851]: pgmap v85: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 619 op/s 2024-04-26T01:43:43.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:43 smithi053 ceph-mon[32593]: pgmap v85: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 619 op/s 2024-04-26T01:43:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:43 smithi028 ceph-mon[28247]: pgmap v85: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 619 op/s 2024-04-26T01:43:45.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:45 smithi169 ceph-mon[32851]: pgmap v86: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.5 MiB/s wr, 758 op/s 2024-04-26T01:43:45.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:45 smithi053 ceph-mon[32593]: pgmap v86: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.5 MiB/s wr, 758 op/s 2024-04-26T01:43:45.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:45 smithi028 ceph-mon[28247]: pgmap v86: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.5 MiB/s wr, 758 op/s 2024-04-26T01:43:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 110.0 s, 11.6 tps, lat 31.558 ms stddev 29.160 2024-04-26T01:43:47.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:47 smithi169 ceph-mon[32851]: pgmap v87: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 441 op/s 2024-04-26T01:43:47.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:47 smithi053 ceph-mon[32593]: pgmap v87: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 441 op/s 2024-04-26T01:43:47.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:47 smithi028 ceph-mon[28247]: pgmap v87: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 441 op/s 2024-04-26T01:43:49.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:49 smithi169 ceph-mon[32851]: pgmap v88: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 441 op/s 2024-04-26T01:43:49.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:49 smithi053 ceph-mon[32593]: pgmap v88: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 441 op/s 2024-04-26T01:43:49.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:49 smithi028 ceph-mon[28247]: pgmap v88: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 441 op/s 2024-04-26T01:43:51.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 115.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:43:51.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:51 smithi169 ceph-mon[32851]: pgmap v89: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.9 MiB/s wr, 458 op/s 2024-04-26T01:43:51.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:51 smithi053 ceph-mon[32593]: pgmap v89: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.9 MiB/s wr, 458 op/s 2024-04-26T01:43:51.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:51 smithi028 ceph-mon[28247]: pgmap v89: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.9 MiB/s wr, 458 op/s 2024-04-26T01:43:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.401+0000 7f21c8800640 1 -- 172.21.15.28:0/3724873496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c010e620 msgr2=0x7f21c010ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.401+0000 7f21c8800640 1 --2- 172.21.15.28:0/3724873496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c010e620 0x7f21c010ea00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009930 tx=0x7f21b402a420 comp rx=0 tx=0).stop 2024-04-26T01:43:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.401+0000 7f21c8800640 1 -- 172.21.15.28:0/3724873496 shutdown_connections 2024-04-26T01:43:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.401+0000 7f21c8800640 1 --2- 172.21.15.28:0/3724873496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21c0108f20 0x7f21c01153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.401+0000 7f21c8800640 1 --2- 172.21.15.28:0/3724873496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c0108560 0x7f21c01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.401+0000 7f21c8800640 1 --2- 172.21.15.28:0/3724873496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c010e620 0x7f21c010ea00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.402+0000 7f21c8800640 1 -- 172.21.15.28:0/3724873496 >> 172.21.15.28:0/3724873496 conn(0x7f21c00764e0 msgr2=0x7f21c0078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.402+0000 7f21c8800640 1 -- 172.21.15.28:0/3724873496 shutdown_connections 2024-04-26T01:43:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.402+0000 7f21c8800640 1 -- 172.21.15.28:0/3724873496 wait complete. 2024-04-26T01:43:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.403+0000 7f21c8800640 1 Processor -- start 2024-04-26T01:43:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.403+0000 7f21c8800640 1 -- start start 2024-04-26T01:43:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c0108560 0x7f21c01a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c8800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c0108f20 0x7f21c01a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c8800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21c010e620 0x7f21c019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01187e0 con 0x7f21c0108560 2024-04-26T01:43:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c8800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f21c0118660 con 0x7f21c0108f20 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c8800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f21c0118960 con 0x7f21c010e620 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c0108560 0x7f21c01a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c0108560 0x7f21c01a4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58238/0 (socket says 172.21.15.28:58238) 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5e00640 1 -- 172.21.15.28:0/1699910604 learned_addr learned my addr 172.21.15.28:0/1699910604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c6800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21c010e620 0x7f21c019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5400640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c0108f20 0x7f21c01a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5e00640 1 -- 172.21.15.28:0/1699910604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21c010e620 msgr2=0x7f21c019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5e00640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21c010e620 0x7f21c019eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5e00640 1 -- 172.21.15.28:0/1699910604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c0108f20 msgr2=0x7f21c01a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5e00640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c0108f20 0x7f21c01a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5e00640 1 -- 172.21.15.28:0/1699910604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c019f2a0 con 0x7f21c0108560 2024-04-26T01:43:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.404+0000 7f21c5400640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c0108f20 0x7f21c01a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:43:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.405+0000 7f21c5e00640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c0108560 0x7f21c01a4920 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f21b4005ab0 tx=0x7f21b4003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.405+0000 7f21baa00640 1 -- 172.21.15.28:0/1699910604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b403f070 con 0x7f21c0108560 2024-04-26T01:43:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.405+0000 7f21baa00640 1 -- 172.21.15.28:0/1699910604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b4031b50 con 0x7f21c0108560 2024-04-26T01:43:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.405+0000 7f21c8800640 1 -- 172.21.15.28:0/1699910604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c019f530 con 0x7f21c0108560 2024-04-26T01:43:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.405+0000 7f21baa00640 1 -- 172.21.15.28:0/1699910604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b4035030 con 0x7f21c0108560 2024-04-26T01:43:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.405+0000 7f21c8800640 1 -- 172.21.15.28:0/1699910604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c01ab7e0 con 0x7f21c0108560 2024-04-26T01:43:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.408+0000 7f21cb588740 1 Processor -- start 2024-04-26T01:43:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.408+0000 7f21baa00640 1 -- 172.21.15.28:0/1699910604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f21b4042070 con 0x7f21c0108560 2024-04-26T01:43:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21cb588740 1 -- start start 2024-04-26T01:43:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21cb588740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ad1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21cb588740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4adc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21cb588740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4adca00 0x55dda4adedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21cb588740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dda4a37900 con 0x55dda4adca00 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21cb588740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55dda49ddef0 con 0x55dda4ad2140 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21cb588740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55dda4a338c0 con 0x55dda4ad1820 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21c5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4adc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4adca00 0x55dda4adedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21c6800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ad1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21c5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4adc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35908/0 (socket says 172.21.15.28:35908) 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.409+0000 7f21c5e00640 1 -- 172.21.15.28:0/1620177028 learned_addr learned my addr 172.21.15.28:0/1620177028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.410+0000 7f21c5e00640 1 -- 172.21.15.28:0/1620177028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 msgr2=0x55dda4ad1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.410+0000 7f21c5e00640 1 --2- 172.21.15.28:0/1620177028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ad1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.410+0000 7f21c5e00640 1 -- 172.21.15.28:0/1620177028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4adca00 msgr2=0x55dda4adedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.410+0000 7f21c5e00640 1 --2- 172.21.15.28:0/1620177028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4adca00 0x55dda4adedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.410+0000 7f21c5e00640 1 -- 172.21.15.28:0/1620177028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dda474a210 con 0x55dda4ad2140 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.410+0000 7f21c5400640 1 --2- 172.21.15.28:0/1620177028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4adca00 0x55dda4adedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.410+0000 7f21c5e00640 1 --2- 172.21.15.28:0/1620177028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4adc4c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f21b40ee5b0 tx=0x7f21b40f9220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372f8ef0c9e56092 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.410+0000 7f21baa00640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f21a80a3dc0 0x7f21a80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21baa00640 1 -- 172.21.15.28:0/1699910604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f21b40f0f50 con 0x7f21c0108560 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21af000640 1 -- 172.21.15.28:0/1620177028 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b40b9160 con 0x55dda4ad2140 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21af000640 1 -- 172.21.15.28:0/1620177028 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b40fe070 con 0x55dda4ad2140 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21af000640 1 -- 172.21.15.28:0/1620177028 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b40bc820 con 0x55dda4ad2140 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21c6800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f21a80a3dc0 0x7f21a80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21cb588740 1 -- 172.21.15.28:0/1620177028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 msgr2=0x55dda4adc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21cb588740 1 --2- 172.21.15.28:0/1620177028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4adc4c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f21b40ee5b0 tx=0x7f21b40f9220 comp rx=0 tx=0).stop 2024-04-26T01:43:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21c6800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f21a80a3dc0 0x7f21a80a6280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f21bc005fd0 tx=0x7f21bc0096a0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21cb588740 1 -- 172.21.15.28:0/1620177028 shutdown_connections 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21cb588740 1 --2- 172.21.15.28:0/1620177028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4adca00 0x55dda4adedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21cb588740 1 --2- 172.21.15.28:0/1620177028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4adc4c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21cb588740 1 --2- 172.21.15.28:0/1620177028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ad1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.411+0000 7f21cb588740 1 -- 172.21.15.28:0/1620177028 >> 172.21.15.28:0/1620177028 conn(0x55dda4acdd30 msgr2=0x55dda4ace110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.412+0000 7f21cb588740 1 -- 172.21.15.28:0/1620177028 shutdown_connections 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.412+0000 7f21cb588740 1 -- 172.21.15.28:0/1620177028 wait complete. 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.412+0000 7f21cb588740 1 -- start start 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.412+0000 7f21cb588740 1 Processor -- start 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21cb588740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad1820 0x55dda4ad9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21cb588740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4ad9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21cb588740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4ada3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21cb588740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dda4adf3f0 con 0x55dda4ad1820 2024-04-26T01:43:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21cb588740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55dda49f1310 con 0x55dda4ad2140 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21cb588740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55dda4adf570 con 0x55dda4adca00 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21cb588740 10 client.0 _ll_register_callbacks cb 0x55dda49bfcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad1820 0x55dda4ad9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21c6800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4ada3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21c5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4ad9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21c6800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4ada3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:34038/0 (socket says 172.21.15.28:34038) 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.413+0000 7f21c6800640 1 -- 172.21.15.28:0/4124997277 learned_addr learned my addr 172.21.15.28:0/4124997277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21c5e00640 1 -- 172.21.15.28:0/4124997277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 msgr2=0x55dda4ada3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21c5e00640 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4ada3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21c5e00640 1 -- 172.21.15.28:0/4124997277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 msgr2=0x55dda4ad9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21c5e00640 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4ad9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21c5e00640 1 -- 172.21.15.28:0/4124997277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dda4816620 con 0x55dda4ad1820 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21c5e00640 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad1820 0x55dda4ad9960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21b4031f20 tx=0x7f21b402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa31ce16aee7d21 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21a3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21a3400640 1 -- 172.21.15.28:0/4124997277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b4049070 con 0x55dda4ad1820 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21a3400640 1 -- 172.21.15.28:0/4124997277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f21b40f4b70 con 0x55dda4ad1820 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21cb588740 1 -- 172.21.15.28:0/4124997277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21b40fe070 con 0x55dda4ad1820 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.414+0000 7f21a3400640 1 -- 172.21.15.28:0/4124997277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21b40bca00 con 0x55dda4ad1820 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21cb588740 10 client.15366 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21cb588740 1 -- 172.21.15.28:0/4124997277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dda4adf3f0 con 0x55dda4ad1820 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21a3400640 1 -- 172.21.15.28:0/4124997277 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b40fe070 con 0x55dda4ad1820 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21a3400640 1 -- 172.21.15.28:0/4124997277 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f21b40b9530 con 0x55dda4ad1820 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21cb588740 10 client.15366 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21cb588740 10 client.15366 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21cb588740 20 client.15366 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21cb588740 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55dda4aec7e0 0x55dda4aeebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21cb588740 4 client.15366 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.415+0000 7f21cb588740 1 -- 172.21.15.28:0/4124997277 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dda4a4b8b0 con 0x55dda4aec7e0 2024-04-26T01:43:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.416+0000 7f21c5400640 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55dda4aec7e0 0x55dda4aeebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.416+0000 7f21c5400640 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55dda4aec7e0 0x55dda4aeebd0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c6719b617f64174 server_cookie=5cf642c4e35b81e6 in_seq=0 out_seq=0 2024-04-26T01:43:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.416+0000 7f21a3400640 10 client.15366 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21a3400640 1 -- 172.21.15.28:0/4124997277 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dda4a4b8b0 con 0x55dda4aec7e0 2024-04-26T01:43:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21a3400640 10 client.15366 handle_command_reply: tid=0 2024-04-26T01:43:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21cb588740 1 client.15366 shutdown 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21cb588740 1 -- 172.21.15.28:0/4124997277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad1820 msgr2=0x55dda4ad9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21cb588740 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad1820 0x55dda4ad9960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21b4031f20 tx=0x7f21b402a420 comp rx=0 tx=0).stop 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21cb588740 1 -- 172.21.15.28:0/4124997277 shutdown_connections 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21cb588740 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55dda4aec7e0 0x55dda4aeebd0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21cb588740 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4ada3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21cb588740 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad2140 0x55dda4ad9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21a3400640 0 client.15366 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.417+0000 7f21cb588740 1 --2- 172.21.15.28:0/4124997277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad1820 0x55dda4ad9960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.418+0000 7f21cb588740 1 -- 172.21.15.28:0/4124997277 >> 172.21.15.28:0/4124997277 conn(0x55dda4acdd30 msgr2=0x55dda4ad3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.418+0000 7f21cb588740 1 -- 172.21.15.28:0/4124997277 shutdown_connections 2024-04-26T01:43:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.418+0000 7f21cb588740 1 -- 172.21.15.28:0/4124997277 wait complete. 2024-04-26T01:43:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.418+0000 7f21cb588740 20 client.15366 trim_cache size 0 max 16384 2024-04-26T01:43:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.434+0000 7f21cb588740 1 Processor -- start 2024-04-26T01:43:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.434+0000 7f21cb588740 1 -- start start 2024-04-26T01:43:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21cb588740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad1820 0x55dda4ae8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21cb588740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad2140 0x55dda4ad41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21cb588740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4accd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21cb588740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01b69f0 con 0x55dda4ad2140 2024-04-26T01:43:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21cb588740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01b4290 con 0x55dda4ad1820 2024-04-26T01:43:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21cb588740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01b20e0 con 0x55dda4adca00 2024-04-26T01:43:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad2140 0x55dda4ad41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad2140 0x55dda4ad41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58266/0 (socket says 172.21.15.28:58266) 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5400640 1 -- 172.21.15.28:0/3165296106 learned_addr learned my addr 172.21.15.28:0/3165296106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5e00640 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad1820 0x55dda4ae8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c6800640 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4accd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5400640 1 -- 172.21.15.28:0/3165296106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 msgr2=0x55dda4accd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5400640 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4accd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5400640 1 -- 172.21.15.28:0/3165296106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad1820 msgr2=0x55dda4ae8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5400640 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad1820 0x55dda4ae8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.435+0000 7f21c5400640 1 -- 172.21.15.28:0/3165296106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dda4ad1f70 con 0x55dda4ad2140 2024-04-26T01:43:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.436+0000 7f21c5400640 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad2140 0x55dda4ad41e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000c00 tx=0x7f21b0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a474db4d56028123 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.436+0000 7f21ae600640 1 -- 172.21.15.28:0/3165296106 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0024020 con 0x55dda4ad2140 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.436+0000 7f21ae600640 1 -- 172.21.15.28:0/3165296106 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b0003d60 con 0x55dda4ad2140 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.436+0000 7f21ae600640 1 -- 172.21.15.28:0/3165296106 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b00127d0 con 0x55dda4ad2140 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 -- 172.21.15.28:0/3165296106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad2140 msgr2=0x55dda4ad41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad2140 0x55dda4ad41e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000c00 tx=0x7f21b0005f90 comp rx=0 tx=0).stop 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 -- 172.21.15.28:0/3165296106 shutdown_connections 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4adca00 0x55dda4accd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ad2140 0x55dda4ad41e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 --2- 172.21.15.28:0/3165296106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4ad1820 0x55dda4ae8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 -- 172.21.15.28:0/3165296106 >> 172.21.15.28:0/3165296106 conn(0x55dda4acdd30 msgr2=0x55dda4acc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 -- 172.21.15.28:0/3165296106 shutdown_connections 2024-04-26T01:43:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 -- 172.21.15.28:0/3165296106 wait complete. 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 -- start start 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.437+0000 7f21cb588740 1 Processor -- start 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21cb588740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ae2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21cb588740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4adca00 0x55dda4ae29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21cb588740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ae3040 0x55dda4b51d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21cb588740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01b2750 con 0x55dda4ae3040 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21cb588740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01b3bc0 con 0x55dda4adca00 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21cb588740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01b2e20 con 0x55dda4ad1820 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21cb588740 10 client.0 _ll_register_callbacks cb 0x7f21b40f4eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21c5e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ae2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21c5e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ae2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:34062/0 (socket says 172.21.15.28:34062) 2024-04-26T01:43:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21c5e00640 1 -- 172.21.15.28:0/3758092262 learned_addr learned my addr 172.21.15.28:0/3758092262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21c6800640 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ae3040 0x55dda4b51d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.438+0000 7f21c5400640 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4adca00 0x55dda4ae29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21c5e00640 1 -- 172.21.15.28:0/3758092262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4adca00 msgr2=0x55dda4ae29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21c5e00640 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4adca00 0x55dda4ae29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21c5e00640 1 -- 172.21.15.28:0/3758092262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ae3040 msgr2=0x55dda4b51d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21c5e00640 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ae3040 0x55dda4b51d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21c5e00640 1 -- 172.21.15.28:0/3758092262 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dda4b524b0 con 0x55dda4ad1820 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21c5e00640 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ae2460 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f21b40bcbc0 tx=0x7f21b40f9220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f45a3056c8ebae2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21a3400640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21a3400640 1 -- 172.21.15.28:0/3758092262 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b40d5060 con 0x55dda4ad1820 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21a3400640 1 -- 172.21.15.28:0/3758092262 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f21b40f47d0 con 0x55dda4ad1820 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21cb588740 1 -- 172.21.15.28:0/3758092262 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21b0003d60 con 0x55dda4ad1820 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21a3400640 1 -- 172.21.15.28:0/3758092262 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21b4049070 con 0x55dda4ad1820 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21cb588740 10 client.25537 fetch_fsmap learned FSMap version 28 2024-04-26T01:43:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.439+0000 7f21cb588740 1 -- 172.21.15.28:0/3758092262 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dda4b524b0 con 0x55dda4ad1820 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21a3400640 1 -- 172.21.15.28:0/3758092262 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b0003d60 con 0x55dda4ad1820 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21a3400640 1 -- 172.21.15.28:0/3758092262 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f21b40d6250 con 0x55dda4ad1820 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21cb588740 10 client.25537 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21cb588740 10 client.25537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21cb588740 20 client.25537 populate_metadata read hostname 'smithi028' 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21cb588740 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55dda4b53b80 0x55dda4b55fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21cb588740 4 client.25537 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21cb588740 1 -- 172.21.15.28:0/3758092262 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dda4a45320 con 0x55dda4b53b80 2024-04-26T01:43:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.440+0000 7f21c5400640 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55dda4b53b80 0x55dda4b55fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.441+0000 7f21c5400640 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55dda4b53b80 0x55dda4b55fa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d92f54ba000d9e6 server_cookie=a41b665b754f1e79 in_seq=0 out_seq=0 2024-04-26T01:43:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.441+0000 7f21a3400640 10 client.25537 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.455+0000 7f21a3400640 1 -- 172.21.15.28:0/3758092262 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dda4a45320 con 0x55dda4b53b80 2024-04-26T01:43:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.455+0000 7f21a3400640 10 client.25537 handle_command_reply: tid=0 2024-04-26T01:43:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.455+0000 7f21cb588740 1 client.25537 shutdown 2024-04-26T01:43:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 -- 172.21.15.28:0/3758092262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 msgr2=0x55dda4ae2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ae2460 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f21b40bcbc0 tx=0x7f21b40f9220 comp rx=0 tx=0).stop 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 -- 172.21.15.28:0/3758092262 shutdown_connections 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55dda4b53b80 0x55dda4b55fa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda4ae3040 0x55dda4b51d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dda4adca00 0x55dda4ae29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 --2- 172.21.15.28:0/3758092262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dda4ad1820 0x55dda4ae2460 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21a3400640 0 client.25537 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 -- 172.21.15.28:0/3758092262 >> 172.21.15.28:0/3758092262 conn(0x55dda4acdd30 msgr2=0x55dda4ada190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 -- 172.21.15.28:0/3758092262 shutdown_connections 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 1 -- 172.21.15.28:0/3758092262 wait complete. 2024-04-26T01:43:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.456+0000 7f21cb588740 20 client.25537 trim_cache size 0 max 16384 2024-04-26T01:43:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:43:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:43:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:43:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:43:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 -- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f21a80a3dc0 msgr2=0x7f21a80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f21a80a3dc0 0x7f21a80a6280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f21bc005fd0 tx=0x7f21bc0096a0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 -- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c0108560 msgr2=0x7f21c01a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c0108560 0x7f21c01a4920 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f21b4005ab0 tx=0x7f21b4003b90 comp rx=0 tx=0).stop 2024-04-26T01:43:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 -- 172.21.15.28:0/1699910604 shutdown_connections 2024-04-26T01:43:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f21a80a3dc0 0x7f21a80a6280 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21c010e620 0x7f21c019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21c0108f20 0x7f21c01a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 --2- 172.21.15.28:0/1699910604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c0108560 0x7f21c01a4920 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.457+0000 7f21c8800640 1 -- 172.21.15.28:0/1699910604 >> 172.21.15.28:0/1699910604 conn(0x7f21c00764e0 msgr2=0x7f21c0077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.458+0000 7f21c8800640 1 -- 172.21.15.28:0/1699910604 shutdown_connections 2024-04-26T01:43:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:43:53.458+0000 7f21c8800640 1 -- 172.21.15.28:0/1699910604 wait complete. 2024-04-26T01:43:53.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:53 smithi169 ceph-mon[32851]: pgmap v90: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-04-26T01:43:53.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:53 smithi053 ceph-mon[32593]: pgmap v90: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-04-26T01:43:53.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:53 smithi028 ceph-mon[28247]: pgmap v90: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-04-26T01:43:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:55 smithi169 ceph-mon[32851]: pgmap v91: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-04-26T01:43:55.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:55 smithi053 ceph-mon[32593]: pgmap v91: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-04-26T01:43:55.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:55 smithi028 ceph-mon[28247]: pgmap v91: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-04-26T01:43:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 120.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:43:57.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:57 smithi169 ceph-mon[32851]: pgmap v92: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 772 KiB/s wr, 24 op/s 2024-04-26T01:43:57.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:57 smithi053 ceph-mon[32593]: pgmap v92: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 772 KiB/s wr, 24 op/s 2024-04-26T01:43:57.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:57 smithi028 ceph-mon[28247]: pgmap v92: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 772 KiB/s wr, 24 op/s 2024-04-26T01:43:59.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:43:59 smithi169 ceph-mon[32851]: pgmap v93: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 772 KiB/s wr, 24 op/s 2024-04-26T01:43:59.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:43:59 smithi053 ceph-mon[32593]: pgmap v93: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 772 KiB/s wr, 24 op/s 2024-04-26T01:43:59.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:43:59 smithi028 ceph-mon[28247]: pgmap v93: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 772 KiB/s wr, 24 op/s 2024-04-26T01:44:01.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 125.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:01 smithi169 ceph-mon[32851]: pgmap v94: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-04-26T01:44:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:01 smithi053 ceph-mon[32593]: pgmap v94: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-04-26T01:44:01.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:01 smithi028 ceph-mon[28247]: pgmap v94: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-04-26T01:44:02.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:02 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:44:02.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:02 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:44:02.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:02 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:44:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:03 smithi169 ceph-mon[32851]: pgmap v95: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 305 KiB/s wr, 21 op/s 2024-04-26T01:44:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:03 smithi053 ceph-mon[32593]: pgmap v95: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 305 KiB/s wr, 21 op/s 2024-04-26T01:44:03.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:03 smithi028 ceph-mon[28247]: pgmap v95: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 305 KiB/s wr, 21 op/s 2024-04-26T01:44:05.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:05 smithi053 ceph-mon[32593]: pgmap v96: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 421 KiB/s wr, 42 op/s 2024-04-26T01:44:05.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:05 smithi028 ceph-mon[28247]: pgmap v96: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 421 KiB/s wr, 42 op/s 2024-04-26T01:44:05.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:05 smithi169 ceph-mon[32851]: pgmap v96: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 421 KiB/s wr, 42 op/s 2024-04-26T01:44:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 130.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 -- 172.21.15.28:0/967932667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec1043a0 msgr2=0x7faeec104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 --2- 172.21.15.28:0/967932667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec1043a0 0x7faeec104780 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faedc009900 tx=0x7faedc02a410 comp rx=0 tx=0).stop 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 -- 172.21.15.28:0/967932667 shutdown_connections 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 --2- 172.21.15.28:0/967932667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faeec109470 0x7faeec111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 --2- 172.21.15.28:0/967932667 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeec104d50 0x7faeec108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 --2- 172.21.15.28:0/967932667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec1043a0 0x7faeec104780 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 -- 172.21.15.28:0/967932667 >> 172.21.15.28:0/967932667 conn(0x7faeec0ffc40 msgr2=0x7faeec102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 -- 172.21.15.28:0/967932667 shutdown_connections 2024-04-26T01:44:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.604+0000 7faef2200640 1 -- 172.21.15.28:0/967932667 wait complete. 2024-04-26T01:44:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.605+0000 7faef2200640 1 Processor -- start 2024-04-26T01:44:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faef2200640 1 -- start start 2024-04-26T01:44:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faef2200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faeec1043a0 0x7faeec1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faef2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec109470 0x7faeec1a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faef2200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeec19a700 0x7faeec19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faef2200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faeec114400 con 0x7faeec109470 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faef2200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faeec114280 con 0x7faeec1043a0 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faef2200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7faeec114580 con 0x7faeec19a700 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faeeaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec109470 0x7faeec1a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faeeaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec109470 0x7faeec1a0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58984/0 (socket says 172.21.15.28:58984) 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.606+0000 7faeeaa00640 1 -- 172.21.15.28:0/955236898 learned_addr learned my addr 172.21.15.28:0/955236898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeebe00640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeec19a700 0x7faeec19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeeb400640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faeec1043a0 0x7faeec1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeeaa00640 1 -- 172.21.15.28:0/955236898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeec19a700 msgr2=0x7faeec19ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeeaa00640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeec19a700 0x7faeec19ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeeaa00640 1 -- 172.21.15.28:0/955236898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faeec1043a0 msgr2=0x7faeec1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeeaa00640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faeec1043a0 0x7faeec1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeeaa00640 1 -- 172.21.15.28:0/955236898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeec19b300 con 0x7faeec109470 2024-04-26T01:44:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeebe00640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeec19a700 0x7faeec19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.607+0000 7faeeaa00640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec109470 0x7faeec1a0b30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faed800e9a0 tx=0x7faed800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.608+0000 7faee3e00640 1 -- 172.21.15.28:0/955236898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed800cd80 con 0x7faeec109470 2024-04-26T01:44:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.608+0000 7faee3e00640 1 -- 172.21.15.28:0/955236898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faed80035f0 con 0x7faeec109470 2024-04-26T01:44:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.608+0000 7faef2200640 1 -- 172.21.15.28:0/955236898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeec1a7390 con 0x7faeec109470 2024-04-26T01:44:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.608+0000 7faee3e00640 1 -- 172.21.15.28:0/955236898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed8010690 con 0x7faeec109470 2024-04-26T01:44:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.608+0000 7faef2200640 1 -- 172.21.15.28:0/955236898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeec1a7790 con 0x7faeec109470 2024-04-26T01:44:06.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.611+0000 7faee3e00640 1 -- 172.21.15.28:0/955236898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faed8003aa0 con 0x7faeec109470 2024-04-26T01:44:06.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.611+0000 7faef4fad740 1 Processor -- start 2024-04-26T01:44:06.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faef4fad740 1 -- start start 2024-04-26T01:44:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faef4fad740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177ae820 0x55d3177aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faef4fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177b94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faef4fad740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177bbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faef4fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d317714900 con 0x55d3177af140 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faef4fad740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d3176baef0 con 0x55d3177ae820 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faef4fad740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d3177108c0 con 0x55d3177b9a00 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faeeb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177b94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faeeaa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177bbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faeeb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177b94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58988/0 (socket says 172.21.15.28:58988) 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faeeb400640 1 -- 172.21.15.28:0/3727195370 learned_addr learned my addr 172.21.15.28:0/3727195370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.612+0000 7faeebe00640 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177ae820 0x55d3177aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.613+0000 7faeeb400640 1 -- 172.21.15.28:0/3727195370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 msgr2=0x55d3177bbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.613+0000 7faeeb400640 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177bbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.613+0000 7faeeb400640 1 -- 172.21.15.28:0/3727195370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177ae820 msgr2=0x55d3177aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.613+0000 7faeeb400640 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177ae820 0x55d3177aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.613+0000 7faeeb400640 1 -- 172.21.15.28:0/3727195370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d317427210 con 0x55d3177af140 2024-04-26T01:44:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.613+0000 7faeeaa00640 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177bbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.613+0000 7faeebe00640 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177ae820 0x55d3177aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.613+0000 7faee3e00640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7faed00a3dc0 0x7faed00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.614+0000 7faeeb400640 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177b94c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faedc000c00 tx=0x7faedc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4533ddec8b8f7972 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.614+0000 7faee3e00640 1 -- 172.21.15.28:0/955236898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7faed80d0af0 con 0x7faeec109470 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.614+0000 7faed7e00640 1 -- 172.21.15.28:0/3727195370 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faedc035040 con 0x55d3177af140 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.614+0000 7faed7e00640 1 -- 172.21.15.28:0/3727195370 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faedc0054f0 con 0x55d3177af140 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.614+0000 7faed7e00640 1 -- 172.21.15.28:0/3727195370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faedc0035f0 con 0x55d3177af140 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.614+0000 7faeebe00640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7faed00a3dc0 0x7faed00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 -- 172.21.15.28:0/3727195370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 msgr2=0x55d3177b94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177b94c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faedc000c00 tx=0x7faedc004040 comp rx=0 tx=0).stop 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faeebe00640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7faed00a3dc0 0x7faed00a6280 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faee4005fd0 tx=0x7faee4009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 -- 172.21.15.28:0/3727195370 shutdown_connections 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177bbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177b94c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 --2- 172.21.15.28:0/3727195370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177ae820 0x55d3177aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 -- 172.21.15.28:0/3727195370 >> 172.21.15.28:0/3727195370 conn(0x55d3177aad30 msgr2=0x55d3177ab110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 -- 172.21.15.28:0/3727195370 shutdown_connections 2024-04-26T01:44:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.615+0000 7faef4fad740 1 -- 172.21.15.28:0/3727195370 wait complete. 2024-04-26T01:44:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.616+0000 7faef4fad740 1 -- start start 2024-04-26T01:44:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.616+0000 7faef4fad740 1 Processor -- start 2024-04-26T01:44:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faef4fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177b6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faef4fad740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177af140 0x55d3177b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faef4fad740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 0x55d3177b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faef4fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d3177bc5d0 con 0x55d3177ae820 2024-04-26T01:44:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faef4fad740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d3176ce310 con 0x55d3177b9a00 2024-04-26T01:44:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faef4fad740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d3177bc750 con 0x55d3177af140 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faef4fad740 10 client.0 _ll_register_callbacks cb 0x55d31769ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faeeb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faeeb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177b6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58996/0 (socket says 172.21.15.28:58996) 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faeeaa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177af140 0x55d3177b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faeeb400640 1 -- 172.21.15.28:0/881036938 learned_addr learned my addr 172.21.15.28:0/881036938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.617+0000 7faeebe00640 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 0x55d3177b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faeeaa00640 1 -- 172.21.15.28:0/881036938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 msgr2=0x55d3177b73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faeeaa00640 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 0x55d3177b73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faeeaa00640 1 -- 172.21.15.28:0/881036938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 msgr2=0x55d3177b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faeeaa00640 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faeeaa00640 1 -- 172.21.15.28:0/881036938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3174f3620 con 0x55d3177af140 2024-04-26T01:44:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faeeaa00640 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177af140 0x55d3177b6e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faed80ce210 tx=0x7faed8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af40529192db1383 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faed4c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faed4c00640 1 -- 172.21.15.28:0/881036938 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed809c5a0 con 0x55d3177af140 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faed4c00640 1 -- 172.21.15.28:0/881036938 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7faed809ad60 con 0x55d3177af140 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faef4fad740 1 -- 172.21.15.28:0/881036938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faedc0054f0 con 0x55d3177af140 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.618+0000 7faed4c00640 1 -- 172.21.15.28:0/881036938 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faed809c3c0 con 0x55d3177af140 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faef4fad740 10 client.25549 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faef4fad740 1 -- 172.21.15.28:0/881036938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3177c8820 con 0x55d3177af140 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faed4c00640 1 -- 172.21.15.28:0/881036938 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faedc0054f0 con 0x55d3177af140 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faed4c00640 1 -- 172.21.15.28:0/881036938 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7faed80ce490 con 0x55d3177af140 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faef4fad740 10 client.25549 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faef4fad740 10 client.25549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faef4fad740 20 client.25549 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faef4fad740 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d3177c9c90 0x55d3177cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faef4fad740 4 client.25549 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:06.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.619+0000 7faef4fad740 1 -- 172.21.15.28:0/881036938 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3177288b0 con 0x55d3177c9c90 2024-04-26T01:44:06.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.620+0000 7faeeb400640 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d3177c9c90 0x55d3177cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.620+0000 7faeeb400640 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d3177c9c90 0x55d3177cc0e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4408890608cd57a server_cookie=7e467a7f9110b2c6 in_seq=0 out_seq=0 2024-04-26T01:44:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.620+0000 7faed4c00640 10 client.25549 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.621+0000 7faed4c00640 1 -- 172.21.15.28:0/881036938 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3177288b0 con 0x55d3177c9c90 2024-04-26T01:44:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.621+0000 7faed4c00640 10 client.25549 handle_command_reply: tid=0 2024-04-26T01:44:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.621+0000 7faef4fad740 1 client.25549 shutdown 2024-04-26T01:44:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.621+0000 7faef4fad740 1 -- 172.21.15.28:0/881036938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177af140 msgr2=0x55d3177b6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.621+0000 7faef4fad740 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177af140 0x55d3177b6e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faed80ce210 tx=0x7faed8000bc0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 1 -- 172.21.15.28:0/881036938 shutdown_connections 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d3177c9c90 0x55d3177cc0e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 0x55d3177b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177af140 0x55d3177b6e60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 1 --2- 172.21.15.28:0/881036938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faed4c00640 0 client.25549 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 1 -- 172.21.15.28:0/881036938 >> 172.21.15.28:0/881036938 conn(0x55d3177aad30 msgr2=0x55d3177b0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 1 -- 172.21.15.28:0/881036938 shutdown_connections 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 1 -- 172.21.15.28:0/881036938 wait complete. 2024-04-26T01:44:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.622+0000 7faef4fad740 20 client.25549 trim_cache size 0 max 16384 2024-04-26T01:44:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.639+0000 7faef4fad740 1 Processor -- start 2024-04-26T01:44:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.639+0000 7faef4fad740 1 -- start start 2024-04-26T01:44:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faef4fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faef4fad740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177af140 0x55d3177a9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faef4fad740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faef4fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faeec1b2580 con 0x55d3177ae820 2024-04-26T01:44:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faef4fad740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faeec1afe50 con 0x55d3177af140 2024-04-26T01:44:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faef4fad740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7faeec1adca0 con 0x55d3177b9a00 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faeeb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faeeb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59008/0 (socket says 172.21.15.28:59008) 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faeeb400640 1 -- 172.21.15.28:0/1342582949 learned_addr learned my addr 172.21.15.28:0/1342582949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faeebe00640 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faeeaa00640 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177af140 0x55d3177a9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faeebe00640 1 -- 172.21.15.28:0/1342582949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177af140 msgr2=0x55d3177a9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.640+0000 7faeebe00640 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177af140 0x55d3177a9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.641+0000 7faeebe00640 1 -- 172.21.15.28:0/1342582949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 msgr2=0x55d3177c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.641+0000 7faeebe00640 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.641+0000 7faeebe00640 1 -- 172.21.15.28:0/1342582949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3177cc760 con 0x55d3177b9a00 2024-04-26T01:44:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.641+0000 7faeeb400640 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.641+0000 7faeebe00640 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177d19f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faee400f790 tx=0x7faee401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d589c63d3a4831ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.641+0000 7faed7400640 1 -- 172.21.15.28:0/1342582949 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee40209c0 con 0x55d3177b9a00 2024-04-26T01:44:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.641+0000 7faed7400640 1 -- 172.21.15.28:0/1342582949 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faee4003e20 con 0x55d3177b9a00 2024-04-26T01:44:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.641+0000 7faed7400640 1 -- 172.21.15.28:0/1342582949 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4004510 con 0x55d3177b9a00 2024-04-26T01:44:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 -- 172.21.15.28:0/1342582949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 msgr2=0x55d3177d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177d19f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faee400f790 tx=0x7faee401b000 comp rx=0 tx=0).stop 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 -- 172.21.15.28:0/1342582949 shutdown_connections 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177b9a00 0x55d3177d19f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177af140 0x55d3177a9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 --2- 172.21.15.28:0/1342582949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177ae820 0x55d3177c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 -- 172.21.15.28:0/1342582949 >> 172.21.15.28:0/1342582949 conn(0x55d3177aad30 msgr2=0x55d3177a9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 -- 172.21.15.28:0/1342582949 shutdown_connections 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.642+0000 7faef4fad740 1 -- 172.21.15.28:0/1342582949 wait complete. 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.643+0000 7faef4fad740 1 -- start start 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.643+0000 7faef4fad740 1 Processor -- start 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.643+0000 7faef4fad740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177ae820 0x55d3177bf770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.643+0000 7faef4fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.643+0000 7faef4fad740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 0x55d31782f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.643+0000 7faef4fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faeec1ae310 con 0x55d3177af140 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.643+0000 7faef4fad740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faeec1af780 con 0x55d3177b9a00 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.643+0000 7faef4fad740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7faeec1ae9e0 con 0x55d3177ae820 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.644+0000 7faef4fad740 10 client.0 _ll_register_callbacks cb 0x7faed80d4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.644+0000 7faeeaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.644+0000 7faeeaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177bfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59012/0 (socket says 172.21.15.28:59012) 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.644+0000 7faeeaa00640 1 -- 172.21.15.28:0/2870315737 learned_addr learned my addr 172.21.15.28:0/2870315737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.644+0000 7faeeb400640 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177ae820 0x55d3177bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.644+0000 7faeebe00640 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 0x55d31782f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faeeaa00640 1 -- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177ae820 msgr2=0x55d3177bf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faeeaa00640 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177ae820 0x55d3177bf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faeeaa00640 1 -- 172.21.15.28:0/2870315737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 msgr2=0x55d31782f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faeeaa00640 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 0x55d31782f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faeeaa00640 1 -- 172.21.15.28:0/2870315737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3177aab60 con 0x55d3177af140 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faeeb400640 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177ae820 0x55d3177bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faeeaa00640 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177bfcb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faed800e970 tx=0x7faed80a53d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c85c5af12e00d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faed4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faed4c00640 1 -- 172.21.15.28:0/2870315737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed80a56d0 con 0x55d3177af140 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faed4c00640 1 -- 172.21.15.28:0/2870315737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7faed809beb0 con 0x55d3177af140 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.645+0000 7faef4fad740 1 -- 172.21.15.28:0/2870315737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faee4003e20 con 0x55d3177af140 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faed4c00640 1 -- 172.21.15.28:0/2870315737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faed8098d00 con 0x55d3177af140 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faef4fad740 10 client.15390 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faef4fad740 1 -- 172.21.15.28:0/2870315737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d317830720 con 0x55d3177af140 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faed4c00640 1 -- 172.21.15.28:0/2870315737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faee4003e20 con 0x55d3177af140 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faed4c00640 1 -- 172.21.15.28:0/2870315737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7faed8087120 con 0x55d3177af140 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faef4fad740 10 client.15390 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faef4fad740 10 client.15390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faef4fad740 20 client.15390 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faef4fad740 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d317830d90 0x55d3178331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faef4fad740 4 client.15390 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:44:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.646+0000 7faef4fad740 1 -- 172.21.15.28:0/2870315737 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d317722320 con 0x55d317830d90 2024-04-26T01:44:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.647+0000 7faeeb400640 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d317830d90 0x55d3178331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.647+0000 7faeeb400640 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d317830d90 0x55d3178331b0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83290929e98f3493 server_cookie=e19e7cd7d1b6165b in_seq=0 out_seq=0 2024-04-26T01:44:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.647+0000 7faed4c00640 10 client.15390 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.660+0000 7faed4c00640 1 -- 172.21.15.28:0/2870315737 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d317722320 con 0x55d317830d90 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.660+0000 7faed4c00640 10 client.15390 handle_command_reply: tid=0 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.660+0000 7faef4fad740 1 client.15390 shutdown 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 -- 172.21.15.28:0/2870315737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 msgr2=0x55d3177bfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177bfcb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faed800e970 tx=0x7faed80a53d0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 -- 172.21.15.28:0/2870315737 shutdown_connections 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d317830d90 0x55d3178331b0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3177b9a00 0x55d31782f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3177af140 0x55d3177bfcb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 --2- 172.21.15.28:0/2870315737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d3177ae820 0x55d3177bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 -- 172.21.15.28:0/2870315737 >> 172.21.15.28:0/2870315737 conn(0x55d3177aad30 msgr2=0x55d3177a9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faed4c00640 0 client.15390 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 -- 172.21.15.28:0/2870315737 shutdown_connections 2024-04-26T01:44:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 1 -- 172.21.15.28:0/2870315737 wait complete. 2024-04-26T01:44:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.661+0000 7faef4fad740 20 client.15390 trim_cache size 0 max 16384 2024-04-26T01:44:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:44:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:44:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:44:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 -- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7faed00a3dc0 msgr2=0x7faed00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7faed00a3dc0 0x7faed00a6280 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faee4005fd0 tx=0x7faee4009500 comp rx=0 tx=0).stop 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 -- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec109470 msgr2=0x7faeec1a0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec109470 0x7faeec1a0b30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faed800e9a0 tx=0x7faed800eda0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 -- 172.21.15.28:0/955236898 shutdown_connections 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7faed00a3dc0 0x7faed00a6280 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeec19a700 0x7faeec19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faeec109470 0x7faeec1a0b30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 --2- 172.21.15.28:0/955236898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faeec1043a0 0x7faeec1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 -- 172.21.15.28:0/955236898 >> 172.21.15.28:0/955236898 conn(0x7faeec0ffc40 msgr2=0x7faeec101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 -- 172.21.15.28:0/955236898 shutdown_connections 2024-04-26T01:44:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:06.662+0000 7faef2200640 1 -- 172.21.15.28:0/955236898 wait complete. 2024-04-26T01:44:07.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:07 smithi053 ceph-mon[32593]: pgmap v97: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 381 KiB/s wr, 35 op/s 2024-04-26T01:44:07.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:07 smithi028 ceph-mon[28247]: pgmap v97: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 381 KiB/s wr, 35 op/s 2024-04-26T01:44:07.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:07 smithi169 ceph-mon[32851]: pgmap v97: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 381 KiB/s wr, 35 op/s 2024-04-26T01:44:09.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:09 smithi053 ceph-mon[32593]: pgmap v98: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 381 KiB/s wr, 35 op/s 2024-04-26T01:44:09.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:09 smithi028 ceph-mon[28247]: pgmap v98: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 381 KiB/s wr, 35 op/s 2024-04-26T01:44:09.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:09 smithi169 ceph-mon[32851]: pgmap v98: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 381 KiB/s wr, 35 op/s 2024-04-26T01:44:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 135.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:11.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:11 smithi169 ceph-mon[32851]: pgmap v99: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-04-26T01:44:11.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:11 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:11.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:11 smithi053 ceph-mon[32593]: pgmap v99: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-04-26T01:44:11.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:11 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:11.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:11 smithi028 ceph-mon[28247]: pgmap v99: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-04-26T01:44:11.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:11 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:12.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:44:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:13 smithi053 ceph-mon[32593]: pgmap v100: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 838 KiB/s wr, 52 op/s 2024-04-26T01:44:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:13 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:13 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:13 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:44:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:13 smithi028 ceph-mon[28247]: pgmap v100: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 838 KiB/s wr, 52 op/s 2024-04-26T01:44:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:13 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:13.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:13 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:13.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:13 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:44:13.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:13 smithi169 ceph-mon[32851]: pgmap v100: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 838 KiB/s wr, 52 op/s 2024-04-26T01:44:13.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:13 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:13.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:13 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:13.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:13 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:44:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 -- 172.21.15.28:0/816033074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f764406a750 msgr2=0x7f764406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 --2- 172.21.15.28:0/816033074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f764406a750 0x7f764406abb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f762c0098f0 tx=0x7f762c009c00 comp rx=0 tx=0).stop 2024-04-26T01:44:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 -- 172.21.15.28:0/816033074 shutdown_connections 2024-04-26T01:44:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 --2- 172.21.15.28:0/816033074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f764406b0f0 0x7f764410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 --2- 172.21.15.28:0/816033074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7644108a30 0x7f7644108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 --2- 172.21.15.28:0/816033074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f764406a750 0x7f764406abb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 -- 172.21.15.28:0/816033074 >> 172.21.15.28:0/816033074 conn(0x7f76440788f0 msgr2=0x7f7644078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 -- 172.21.15.28:0/816033074 shutdown_connections 2024-04-26T01:44:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.375+0000 7f7649800700 1 -- 172.21.15.28:0/816033074 wait complete. 2024-04-26T01:44:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.376+0000 7f7649800700 1 Processor -- start 2024-04-26T01:44:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7649800700 1 -- start start 2024-04-26T01:44:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7649800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f764406a750 0x7f764419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7649800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f764406b0f0 0x7f764419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7649800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644108a30 0x7f764419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7649800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7644112bd0 con 0x7f7644108a30 2024-04-26T01:44:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7649800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7644112a50 con 0x7f764406b0f0 2024-04-26T01:44:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7649800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7644112d50 con 0x7f764406a750 2024-04-26T01:44:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7642a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f764406a750 0x7f764419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.377+0000 7f7642a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f764406a750 0x7f764419b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:60118/0 (socket says 172.21.15.28:60118) 2024-04-26T01:44:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.378+0000 7f7642a00700 1 -- 172.21.15.28:0/3151418424 learned_addr learned my addr 172.21.15.28:0/3151418424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.378+0000 7f763a000700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f764406b0f0 0x7f764419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.378+0000 7f7643400700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644108a30 0x7f764419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.378+0000 7f763a000700 1 -- 172.21.15.28:0/3151418424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f764406a750 msgr2=0x7f764419b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.378+0000 7f763a000700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f764406a750 0x7f764419b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.378+0000 7f763a000700 1 -- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644108a30 msgr2=0x7f764419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.378+0000 7f763a000700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644108a30 0x7f764419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.378+0000 7f763a000700 1 -- 172.21.15.28:0/3151418424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76441a0370 con 0x7f764406b0f0 2024-04-26T01:44:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.379+0000 7f763a000700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f764406b0f0 0x7f764419b8a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f762c009230 tx=0x7f762c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.379+0000 7f763be00700 1 -- 172.21.15.28:0/3151418424 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c005140 con 0x7f764406b0f0 2024-04-26T01:44:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.379+0000 7f763be00700 1 -- 172.21.15.28:0/3151418424 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f762c00b560 con 0x7f764406b0f0 2024-04-26T01:44:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.379+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76441a0600 con 0x7f764406b0f0 2024-04-26T01:44:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.379+0000 7f763be00700 1 -- 172.21.15.28:0/3151418424 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0038e0 con 0x7f764406b0f0 2024-04-26T01:44:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.379+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76441a7ee0 con 0x7f764406b0f0 2024-04-26T01:44:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.380+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7644069730 con 0x7f764406b0f0 2024-04-26T01:44:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.382+0000 7f763be00700 1 -- 172.21.15.28:0/3151418424 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f762c00b9c0 con 0x7f764406b0f0 2024-04-26T01:44:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.384+0000 7f763be00700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f76280a3e30 0x7f76280a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.384+0000 7f763be00700 1 -- 172.21.15.28:0/3151418424 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f762c0d1990 con 0x7f764406b0f0 2024-04-26T01:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.384+0000 7f7642a00700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f76280a3e30 0x7f76280a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.385+0000 7f7642a00700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f76280a3e30 0x7f76280a62f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7634009750 tx=0x7f7634009310 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.386+0000 7f763be00700 1 -- 172.21.15.28:0/3151418424 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f762c013030 con 0x7f764406b0f0 2024-04-26T01:44:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.670+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f764402df80 con 0x7f764406b0f0 2024-04-26T01:44:15.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:15 smithi028 ceph-mon[28247]: pgmap v101: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 840 KiB/s wr, 55 op/s 2024-04-26T01:44:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.672+0000 7f763be00700 1 -- 172.21.15.28:0/3151418424 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11864 (secure 0 0 0) 0x7f762c021380 con 0x7f764406b0f0 2024-04-26T01:44:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:44:15.670 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:44:00.873936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24509,24523]},"id":1}]} 2024-04-26T01:44:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f76280a3e30 msgr2=0x7f76280a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f76280a3e30 0x7f76280a62f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7634009750 tx=0x7f7634009310 comp rx=0 tx=0).stop 2024-04-26T01:44:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f764406b0f0 msgr2=0x7f764419b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f764406b0f0 0x7f764419b8a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f762c009230 tx=0x7f762c00fa00 comp rx=0 tx=0).stop 2024-04-26T01:44:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 shutdown_connections 2024-04-26T01:44:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f76280a3e30 0x7f76280a62f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644108a30 0x7f764419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f764406a750 0x7f764419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 --2- 172.21.15.28:0/3151418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f764406b0f0 0x7f764419b8a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.675+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 >> 172.21.15.28:0/3151418424 conn(0x7f76440788f0 msgr2=0x7f764410dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.676+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 shutdown_connections 2024-04-26T01:44:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:15.676+0000 7f7649800700 1 -- 172.21.15.28:0/3151418424 wait complete. 2024-04-26T01:44:15.675 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-04-26T01:44:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:15 smithi053 ceph-mon[32593]: pgmap v101: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 840 KiB/s wr, 55 op/s 2024-04-26T01:44:15.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:15 smithi169 ceph-mon[32851]: pgmap v101: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 840 KiB/s wr, 55 op/s 2024-04-26T01:44:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 140.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:16.661 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id b7020b43 --release --await' 2024-04-26T01:44:16.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id b7020b43 --release --await 2024-04-26T01:44:16.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:16 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3151418424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:16 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3151418424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:16.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:16 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3151418424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:17.148 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:17.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:17 smithi028 ceph-mon[28247]: pgmap v102: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 724 KiB/s wr, 34 op/s 2024-04-26T01:44:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:17 smithi053 ceph-mon[32593]: pgmap v102: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 724 KiB/s wr, 34 op/s 2024-04-26T01:44:17.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:17 smithi169 ceph-mon[32851]: pgmap v102: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 724 KiB/s wr, 34 op/s 2024-04-26T01:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.579+0000 7fcc2e800700 1 -- 172.21.15.28:0/4174809835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 msgr2=0x7fcc281096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.579+0000 7fcc2e800700 1 --2- 172.21.15.28:0/4174809835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 0x7fcc281096b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009f90 tx=0x7fcc1800b330 comp rx=0 tx=0).stop 2024-04-26T01:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.580+0000 7fcc2e800700 1 -- 172.21.15.28:0/4174809835 shutdown_connections 2024-04-26T01:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.580+0000 7fcc2e800700 1 --2- 172.21.15.28:0/4174809835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2810e3b0 0x7fcc28115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.580+0000 7fcc2e800700 1 --2- 172.21.15.28:0/4174809835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 0x7fcc281096b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.580+0000 7fcc2e800700 1 --2- 172.21.15.28:0/4174809835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc28109c80 0x7fcc2810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.580+0000 7fcc2e800700 1 -- 172.21.15.28:0/4174809835 >> 172.21.15.28:0/4174809835 conn(0x7fcc28076210 msgr2=0x7fcc28107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.580+0000 7fcc2e800700 1 -- 172.21.15.28:0/4174809835 shutdown_connections 2024-04-26T01:44:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.580+0000 7fcc2e800700 1 -- 172.21.15.28:0/4174809835 wait complete. 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.581+0000 7fcc2e800700 1 Processor -- start 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.581+0000 7fcc2e800700 1 -- start start 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.581+0000 7fcc2e800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 0x7fcc2819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.581+0000 7fcc2e800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc28109c80 0x7fcc2819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.581+0000 7fcc2e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2810e3b0 0x7fcc281a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.581+0000 7fcc2e800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc28119240 con 0x7fcc2810e3b0 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.581+0000 7fcc2e800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281190c0 con 0x7fcc281092d0 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc2e800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281193c0 con 0x7fcc28109c80 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 0x7fcc2819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc1ea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc28109c80 0x7fcc2819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 0x7fcc2819f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36102/0 (socket says 172.21.15.28:36102) 2024-04-26T01:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27400700 1 -- 172.21.15.28:0/2505061067 learned_addr learned my addr 172.21.15.28:0/2505061067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27e00700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2810e3b0 0x7fcc281a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27400700 1 -- 172.21.15.28:0/2505061067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc28109c80 msgr2=0x7fcc2819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27400700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc28109c80 0x7fcc2819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27400700 1 -- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2810e3b0 msgr2=0x7fcc281a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27400700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2810e3b0 0x7fcc281a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.582+0000 7fcc27400700 1 -- 172.21.15.28:0/2505061067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc281a4720 con 0x7fcc281092d0 2024-04-26T01:44:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.583+0000 7fcc27400700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 0x7fcc2819f710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009f60 tx=0x7fcc1800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.583+0000 7fcc24c00700 1 -- 172.21.15.28:0/2505061067 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18021070 con 0x7fcc281092d0 2024-04-26T01:44:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.583+0000 7fcc24c00700 1 -- 172.21.15.28:0/2505061067 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc1801d450 con 0x7fcc281092d0 2024-04-26T01:44:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.583+0000 7fcc24c00700 1 -- 172.21.15.28:0/2505061067 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc180052a0 con 0x7fcc281092d0 2024-04-26T01:44:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.583+0000 7fcc2e800700 1 -- 172.21.15.28:0/2505061067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc281a49b0 con 0x7fcc281092d0 2024-04-26T01:44:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.583+0000 7fcc2e800700 1 -- 172.21.15.28:0/2505061067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc281ac290 con 0x7fcc281092d0 2024-04-26T01:44:19.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.588+0000 7fcc24c00700 1 -- 172.21.15.28:0/2505061067 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcc18024070 con 0x7fcc281092d0 2024-04-26T01:44:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.589+0000 7fcc39242b80 1 Processor -- start 2024-04-26T01:44:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc39242b80 1 -- start start 2024-04-26T01:44:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d0521120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d052bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc24c00700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc0c0a3e30 0x7fcc0c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d052e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc39242b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602d0473f90 con 0x5602d0520d40 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc39242b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5602d041a830 con 0x5602d052c4d0 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc39242b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5602d046ff50 con 0x5602d0521660 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc24c00700 1 -- 172.21.15.28:0/2505061067 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcc1800f020 con 0x7fcc281092d0 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc27400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d052bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc1ea00700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc0c0a3e30 0x7fcc0c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc27400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d052bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51722/0 (socket says 172.21.15.28:51722) 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.590+0000 7fcc27400700 1 -- 172.21.15.28:0/3077638853 learned_addr learned my addr 172.21.15.28:0/3077638853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27e00700 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d0521120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27e00700 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d052e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27400700 1 -- 172.21.15.28:0/3077638853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 msgr2=0x5602d052e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27400700 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d052e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27400700 1 -- 172.21.15.28:0/3077638853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 msgr2=0x5602d0521120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27400700 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d0521120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27400700 1 -- 172.21.15.28:0/3077638853 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d0410780 con 0x5602d0521660 2024-04-26T01:44:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27e00700 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d052e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:44:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc27400700 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d052bf90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcc1801dfd0 tx=0x7fcc1800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c6f5074ce59d1b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.591+0000 7fcc1ea00700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc0c0a3e30 0x7fcc0c0a62f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc28078270 tx=0x7fcc10009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.592+0000 7fcc1cc00700 1 -- 172.21.15.28:0/3077638853 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1809db70 con 0x5602d0521660 2024-04-26T01:44:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.592+0000 7fcc1cc00700 1 -- 172.21.15.28:0/3077638853 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc180a1f20 con 0x5602d0521660 2024-04-26T01:44:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.592+0000 7fcc1cc00700 1 -- 172.21.15.28:0/3077638853 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1801d900 con 0x5602d0521660 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.592+0000 7fcc39242b80 1 -- 172.21.15.28:0/3077638853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 msgr2=0x5602d052bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.592+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d052bf90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcc1801dfd0 tx=0x7fcc1800b330 comp rx=0 tx=0).stop 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 -- 172.21.15.28:0/3077638853 shutdown_connections 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d052e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d0521120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3077638853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d052bf90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 -- 172.21.15.28:0/3077638853 >> 172.21.15.28:0/3077638853 conn(0x5602d0450050 msgr2=0x5602d051d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 -- 172.21.15.28:0/3077638853 shutdown_connections 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 -- 172.21.15.28:0/3077638853 wait complete. 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 -- start start 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.593+0000 7fcc39242b80 1 Processor -- start 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d052bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0528f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d05294c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc39242b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602d052ee00 con 0x5602d0520d40 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc39242b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5602d042d960 con 0x5602d052c4d0 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc39242b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5602d052ef80 con 0x5602d0521660 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc39242b80 10 client.0 _ll_register_callbacks cb 0x5602d040c820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc27400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d052bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc27400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d052bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51032/0 (socket says 172.21.15.28:51032) 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.594+0000 7fcc27400700 1 -- 172.21.15.28:0/3885568789 learned_addr learned my addr 172.21.15.28:0/3885568789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc27e00700 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d05294c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc1ea00700 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0528f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc27e00700 1 -- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 msgr2=0x5602d0528f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc27e00700 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0528f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc27e00700 1 -- 172.21.15.28:0/3885568789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 msgr2=0x5602d052bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc27e00700 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d052bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc27e00700 1 -- 172.21.15.28:0/3885568789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d03c19d0 con 0x5602d052c4d0 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc1ea00700 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0528f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc27e00700 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d05294c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x5602d0522710 tx=0x7fcc20012ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e7930bab321a2a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.595+0000 7fcc15600700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.596+0000 7fcc15600700 1 -- 172.21.15.28:0/3885568789 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc20004030 con 0x5602d052c4d0 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.596+0000 7fcc15600700 1 -- 172.21.15.28:0/3885568789 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcc200041d0 con 0x5602d052c4d0 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.596+0000 7fcc39242b80 1 -- 172.21.15.28:0/3885568789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc180a1f20 con 0x5602d052c4d0 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.596+0000 7fcc15600700 1 -- 172.21.15.28:0/3885568789 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc200043d0 con 0x5602d052c4d0 2024-04-26T01:44:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.596+0000 7fcc39242b80 10 client.25547 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.596+0000 7fcc39242b80 1 -- 172.21.15.28:0/3885568789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5602d040abf0 con 0x5602d052c4d0 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.596+0000 7fcc15600700 1 -- 172.21.15.28:0/3885568789 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc180a1f20 con 0x5602d052c4d0 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.597+0000 7fcc15600700 1 -- 172.21.15.28:0/3885568789 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fcc2001d5f0 con 0x5602d052c4d0 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.597+0000 7fcc39242b80 10 client.25547 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.597+0000 7fcc39242b80 10 client.25547 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.597+0000 7fcc39242b80 20 client.25547 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.597+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602d053b9c0 0x5602d053ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.597+0000 7fcc39242b80 4 client.25547 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.597+0000 7fcc39242b80 1 -- 172.21.15.28:0/3885568789 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602d0481780 con 0x5602d053b9c0 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.597+0000 7fcc27400700 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602d053b9c0 0x5602d053ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.598+0000 7fcc27400700 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602d053b9c0 0x5602d053ddb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9dd7861eea2b45a server_cookie=87b1f239c865c7da in_seq=0 out_seq=0 2024-04-26T01:44:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.598+0000 7fcc15600700 10 client.25547 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc15600700 1 -- 172.21.15.28:0/3885568789 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602d0481780 con 0x5602d053b9c0 2024-04-26T01:44:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc15600700 10 client.25547 handle_command_reply: tid=0 2024-04-26T01:44:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 client.25547 shutdown 2024-04-26T01:44:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 -- 172.21.15.28:0/3885568789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 msgr2=0x5602d05294c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d05294c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x5602d0522710 tx=0x7fcc20012ae0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 -- 172.21.15.28:0/3885568789 shutdown_connections 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d05294c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0520d40 0x5602d052bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602d053b9c0 0x5602d053ddb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 --2- 172.21.15.28:0/3885568789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0528f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 -- 172.21.15.28:0/3885568789 >> 172.21.15.28:0/3885568789 conn(0x5602d0450050 msgr2=0x5602d051ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc15600700 0 client.25547 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 -- 172.21.15.28:0/3885568789 shutdown_connections 2024-04-26T01:44:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 1 -- 172.21.15.28:0/3885568789 wait complete. 2024-04-26T01:44:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.599+0000 7fcc39242b80 20 client.25547 trim_cache size 0 max 16384 2024-04-26T01:44:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.621+0000 7fcc39242b80 1 Processor -- start 2024-04-26T01:44:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.621+0000 7fcc39242b80 1 -- start start 2024-04-26T01:44:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0520d40 0x5602d0537c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0521660 0x5602d0543650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d0543b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc39242b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281b60f0 con 0x5602d0521660 2024-04-26T01:44:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc39242b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281b3980 con 0x5602d052c4d0 2024-04-26T01:44:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc39242b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281b17d0 con 0x5602d0520d40 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc1ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0521660 0x5602d0543650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc27e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d0543b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc27400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0520d40 0x5602d0537c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc27e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d0543b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36116/0 (socket says 172.21.15.28:36116) 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc1ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0521660 0x5602d0543650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51036/0 (socket says 172.21.15.28:51036) 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.622+0000 7fcc27e00700 1 -- 172.21.15.28:0/1408848525 learned_addr learned my addr 172.21.15.28:0/1408848525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc27e00700 1 -- 172.21.15.28:0/1408848525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0520d40 msgr2=0x5602d0537c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc27e00700 1 --2- 172.21.15.28:0/1408848525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0520d40 0x5602d0537c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc27e00700 1 -- 172.21.15.28:0/1408848525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0521660 msgr2=0x5602d0543650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc27e00700 1 --2- 172.21.15.28:0/1408848525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0521660 0x5602d0543650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc27e00700 1 -- 172.21.15.28:0/1408848525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d0522bb0 con 0x5602d052c4d0 2024-04-26T01:44:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc27400700 1 --2- 172.21.15.28:0/1408848525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0520d40 0x5602d0537c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc27e00700 1 --2- 172.21.15.28:0/1408848525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d0543b90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcc2001dec0 tx=0x7fcc2001def0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=909e93f9f3921427 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc17e00700 1 -- 172.21.15.28:0/1408848525 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2002a040 con 0x5602d052c4d0 2024-04-26T01:44:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc17e00700 1 -- 172.21.15.28:0/1408848525 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc20005070 con 0x5602d052c4d0 2024-04-26T01:44:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.623+0000 7fcc17e00700 1 -- 172.21.15.28:0/1408848525 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc200047d0 con 0x5602d052c4d0 2024-04-26T01:44:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 -- 172.21.15.28:0/1408848525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 msgr2=0x5602d0543b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1408848525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d0543b90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcc2001dec0 tx=0x7fcc2001def0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 -- 172.21.15.28:0/1408848525 shutdown_connections 2024-04-26T01:44:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1408848525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d052c4d0 0x5602d0543b90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1408848525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0520d40 0x5602d0537c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1408848525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d0521660 0x5602d0543650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 -- 172.21.15.28:0/1408848525 >> 172.21.15.28:0/1408848525 conn(0x5602d0450050 msgr2=0x5602d0523670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 -- 172.21.15.28:0/1408848525 shutdown_connections 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.624+0000 7fcc39242b80 1 -- 172.21.15.28:0/1408848525 wait complete. 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 1 -- start start 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 1 Processor -- start 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d0520d40 0x5602d0531240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0531780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d052c4d0 0x5602d05a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281b1e40 con 0x5602d052c4d0 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281b32b0 con 0x5602d0520d40 2024-04-26T01:44:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281b2510 con 0x5602d0521660 2024-04-26T01:44:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.625+0000 7fcc39242b80 10 client.0 _ll_register_callbacks cb 0x5602d040c820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc27e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d052c4d0 0x5602d05a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc27400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d0520d40 0x5602d0531240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc27e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d052c4d0 0x5602d05a1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51044/0 (socket says 172.21.15.28:51044) 2024-04-26T01:44:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc1ea00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0531780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc27400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d0520d40 0x5602d0531240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36128/0 (socket says 172.21.15.28:36128) 2024-04-26T01:44:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc27e00700 1 -- 172.21.15.28:0/1790471235 learned_addr learned my addr 172.21.15.28:0/1790471235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:19 smithi028 ceph-mon[28247]: pgmap v103: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 724 KiB/s wr, 34 op/s 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc1ea00700 1 -- 172.21.15.28:0/1790471235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d0520d40 msgr2=0x5602d0531240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc1ea00700 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d0520d40 0x5602d0531240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc1ea00700 1 -- 172.21.15.28:0/1790471235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d052c4d0 msgr2=0x5602d05a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc1ea00700 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d052c4d0 0x5602d05a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc1ea00700 1 -- 172.21.15.28:0/1790471235 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d0529ea0 con 0x5602d0521660 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.626+0000 7fcc27e00700 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d052c4d0 0x5602d05a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.627+0000 7fcc1ea00700 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0531780 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x5602d0519ae0 tx=0x7fcc100047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd6cbd146633fb83 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.627+0000 7fcc15600700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.627+0000 7fcc15600700 1 -- 172.21.15.28:0/1790471235 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10004320 con 0x5602d0521660 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.627+0000 7fcc15600700 1 -- 172.21.15.28:0/1790471235 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcc10009040 con 0x5602d0521660 2024-04-26T01:44:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.627+0000 7fcc39242b80 1 -- 172.21.15.28:0/1790471235 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc20005070 con 0x5602d0521660 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc15600700 1 -- 172.21.15.28:0/1790471235 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc100044c0 con 0x5602d0521660 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc39242b80 10 client.25579 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc39242b80 1 -- 172.21.15.28:0/1790471235 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5602d05a35e0 con 0x5602d0521660 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc15600700 1 -- 172.21.15.28:0/1790471235 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc20005070 con 0x5602d0521660 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc15600700 1 -- 172.21.15.28:0/1790471235 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fcc1001c7e0 con 0x5602d0521660 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc39242b80 10 client.25579 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc39242b80 10 client.25579 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc39242b80 20 client.25579 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602d05a3d80 0x5602d05a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.628+0000 7fcc39242b80 4 client.25579 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b7020b43", "release": true, "await": true}] 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.629+0000 7fcc39242b80 1 -- 172.21.15.28:0/1790471235 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b7020b43", "release": true, "await": true}) -- 0x5602d047b820 con 0x5602d05a3d80 2024-04-26T01:44:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.629+0000 7fcc27400700 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602d05a3d80 0x5602d05a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.630+0000 7fcc27400700 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602d05a3d80 0x5602d05a61a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5302c4a66996205 server_cookie=c4ba7f02ccea5881 in_seq=0 out_seq=0 2024-04-26T01:44:19.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.630+0000 7fcc15600700 10 client.25579 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:19.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc15600700 1 -- 172.21.15.28:0/1790471235 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5602d047b820 con 0x5602d05a3d80 2024-04-26T01:44:19.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc15600700 10 client.25579 handle_command_reply: tid=0 2024-04-26T01:44:19.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 client.25579 shutdown 2024-04-26T01:44:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 -- 172.21.15.28:0/1790471235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 msgr2=0x5602d0531780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0531780 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x5602d0519ae0 tx=0x7fcc100047c0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 -- 172.21.15.28:0/1790471235 shutdown_connections 2024-04-26T01:44:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d052c4d0 0x5602d05a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602d0520d40 0x5602d0531240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602d05a3d80 0x5602d05a61a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 --2- 172.21.15.28:0/1790471235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602d0521660 0x5602d0531780 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 -- 172.21.15.28:0/1790471235 >> 172.21.15.28:0/1790471235 conn(0x5602d0450050 msgr2=0x5602d052a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc15600700 0 client.25579 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.638+0000 7fcc39242b80 1 -- 172.21.15.28:0/1790471235 shutdown_connections 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.639+0000 7fcc39242b80 1 -- 172.21.15.28:0/1790471235 wait complete. 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.639+0000 7fcc39242b80 20 client.25579 trim_cache size 0 max 16384 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:44:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 29, 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 8, 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "b7020b43": { 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 8, 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 60.0, 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 90.0, 2024-04-26T01:44:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:19.639 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:44:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 -- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc0c0a3e30 msgr2=0x7fcc0c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc0c0a3e30 0x7fcc0c0a62f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc28078270 tx=0x7fcc10009260 comp rx=0 tx=0).stop 2024-04-26T01:44:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 -- 172.21.15.28:0/2505061067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 msgr2=0x7fcc2819f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 0x7fcc2819f710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009f60 tx=0x7fcc1800ccb0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 -- 172.21.15.28:0/2505061067 shutdown_connections 2024-04-26T01:44:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc0c0a3e30 0x7fcc0c0a62f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2810e3b0 0x7fcc281a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc281092d0 0x7fcc2819f710 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 --2- 172.21.15.28:0/2505061067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc28109c80 0x7fcc2819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.640+0000 7fcc2e800700 1 -- 172.21.15.28:0/2505061067 >> 172.21.15.28:0/2505061067 conn(0x7fcc28076210 msgr2=0x7fcc2810cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.641+0000 7fcc2e800700 1 -- 172.21.15.28:0/2505061067 shutdown_connections 2024-04-26T01:44:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:19.641+0000 7fcc2e800700 1 -- 172.21.15.28:0/2505061067 wait complete. 2024-04-26T01:44:19.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:19 smithi053 ceph-mon[32593]: pgmap v103: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 724 KiB/s wr, 34 op/s 2024-04-26T01:44:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 -- 172.21.15.28:0/2343925883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25fc100b30 msgr2=0x7f25fc100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 --2- 172.21.15.28:0/2343925883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25fc100b30 0x7f25fc100f10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009900 tx=0x7f25ec02a410 comp rx=0 tx=0).stop 2024-04-26T01:44:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 -- 172.21.15.28:0/2343925883 shutdown_connections 2024-04-26T01:44:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 --2- 172.21.15.28:0/2343925883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc10e6e0 0x7f25fc110ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 --2- 172.21.15.28:0/2343925883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc1014e0 0x7f25fc10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 --2- 172.21.15.28:0/2343925883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25fc100b30 0x7f25fc100f10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 -- 172.21.15.28:0/2343925883 >> 172.21.15.28:0/2343925883 conn(0x7f25fc0fc770 msgr2=0x7f25fc0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 -- 172.21.15.28:0/2343925883 shutdown_connections 2024-04-26T01:44:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.824+0000 7f2600a00640 1 -- 172.21.15.28:0/2343925883 wait complete. 2024-04-26T01:44:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.825+0000 7f2600a00640 1 Processor -- start 2024-04-26T01:44:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.825+0000 7f2600a00640 1 -- start start 2024-04-26T01:44:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f2600a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc100b30 0x7f25fc1a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f2600a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25fc1014e0 0x7f25fc1a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f2600a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc10e6e0 0x7f25fc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f2600a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1142c0 con 0x7f25fc10e6e0 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f2600a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc114140 con 0x7f25fc1014e0 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f2600a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc114440 con 0x7f25fc100b30 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fa800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc100b30 0x7f25fc1a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fa800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc100b30 0x7f25fc1a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51766/0 (socket says 172.21.15.28:51766) 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fa800640 1 -- 172.21.15.28:0/4070053319 learned_addr learned my addr 172.21.15.28:0/4070053319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25f9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25fc1014e0 0x7f25fc1a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fb200640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc10e6e0 0x7f25fc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fa800640 1 -- 172.21.15.28:0/4070053319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25fc1014e0 msgr2=0x7f25fc1a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fa800640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25fc1014e0 0x7f25fc1a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fa800640 1 -- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc10e6e0 msgr2=0x7f25fc19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fa800640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc10e6e0 0x7f25fc19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fa800640 1 -- 172.21.15.28:0/4070053319 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25fc19af80 con 0x7f25fc100b30 2024-04-26T01:44:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.826+0000 7f25fb200640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc10e6e0 0x7f25fc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.827+0000 7f25fa800640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc100b30 0x7f25fc1a05b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f25ec031440 tx=0x7f25ec003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.827+0000 7f25f3400640 1 -- 172.21.15.28:0/4070053319 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec03f070 con 0x7f25fc100b30 2024-04-26T01:44:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.827+0000 7f25f3400640 1 -- 172.21.15.28:0/4070053319 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25ec031b50 con 0x7f25fc100b30 2024-04-26T01:44:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.827+0000 7f25f3400640 1 -- 172.21.15.28:0/4070053319 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec035030 con 0x7f25fc100b30 2024-04-26T01:44:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.827+0000 7f2600a00640 1 -- 172.21.15.28:0/4070053319 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25fc19b210 con 0x7f25fc100b30 2024-04-26T01:44:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.827+0000 7f2600a00640 1 -- 172.21.15.28:0/4070053319 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25fc1a7530 con 0x7f25fc100b30 2024-04-26T01:44:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.831+0000 7f25f3400640 1 -- 172.21.15.28:0/4070053319 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f25ec042070 con 0x7f25fc100b30 2024-04-26T01:44:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.832+0000 7f2604391740 1 Processor -- start 2024-04-26T01:44:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.832+0000 7f2604391740 1 -- start start 2024-04-26T01:44:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.833+0000 7f2604391740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77abd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.833+0000 7f2604391740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 0x55b0f77b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.833+0000 7f2604391740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77b6b90 0x55b0f77b8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.833+0000 7f2604391740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b0f7711900 con 0x55b0f77b6b90 2024-04-26T01:44:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.833+0000 7f2604391740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b0f76b7ef0 con 0x55b0f77ac2d0 2024-04-26T01:44:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.833+0000 7f2604391740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55b0f770d8c0 con 0x55b0f77ab9b0 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.833+0000 7f25f3400640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f25e00a3d90 0x7f25e00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.833+0000 7f25f3400640 1 -- 172.21.15.28:0/4070053319 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f25ec0f0db0 con 0x7f25fc100b30 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51776/0 (socket says 172.21.15.28:51776) 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 -- 172.21.15.28:0/3613315596 learned_addr learned my addr 172.21.15.28:0/3613315596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fa800640 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 0x55b0f77b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f25e00a3d90 0x7f25e00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25f9e00640 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77b6b90 0x55b0f77b8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 -- 172.21.15.28:0/3613315596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 msgr2=0x55b0f77b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 0x55b0f77b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 -- 172.21.15.28:0/3613315596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77b6b90 msgr2=0x55b0f77b8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77b6b90 0x55b0f77b8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25fb200640 1 -- 172.21.15.28:0/3613315596 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0f7424210 con 0x55b0f77ab9b0 2024-04-26T01:44:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.834+0000 7f25f9e00640 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77b6b90 0x55b0f77b8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:44:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.835+0000 7f25fb200640 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77abd90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f25f400b900 tx=0x7f25f400bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87d77e3b89e54695 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.835+0000 7f25e7400640 1 -- 172.21.15.28:0/3613315596 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f400b410 con 0x55b0f77ab9b0 2024-04-26T01:44:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.835+0000 7f25e7400640 1 -- 172.21.15.28:0/3613315596 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25f4022440 con 0x55b0f77ab9b0 2024-04-26T01:44:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.835+0000 7f25fb200640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f25e00a3d90 0x7f25e00a6250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f25f401f000 tx=0x7f25f4018000 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.835+0000 7f25e7400640 1 -- 172.21.15.28:0/3613315596 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f40165e0 con 0x55b0f77ab9b0 2024-04-26T01:44:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 -- 172.21.15.28:0/3613315596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 msgr2=0x55b0f77abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77abd90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f25f400b900 tx=0x7f25f400bdc0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 -- 172.21.15.28:0/3613315596 shutdown_connections 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77b6b90 0x55b0f77b8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 0x55b0f77b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 --2- 172.21.15.28:0/3613315596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77abd90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 -- 172.21.15.28:0/3613315596 >> 172.21.15.28:0/3613315596 conn(0x55b0f77a7ec0 msgr2=0x55b0f77a82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 -- 172.21.15.28:0/3613315596 shutdown_connections 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.836+0000 7f2604391740 1 -- 172.21.15.28:0/3613315596 wait complete. 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.837+0000 7f2604391740 1 -- start start 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.837+0000 7f2604391740 1 Processor -- start 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.837+0000 7f2604391740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.837+0000 7f2604391740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.837+0000 7f2604391740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77b4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.837+0000 7f2604391740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b0f77b97c0 con 0x55b0f77ac2d0 2024-04-26T01:44:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.837+0000 7f2604391740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b0f76cb310 con 0x55b0f77b6b90 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.837+0000 7f2604391740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55b0f77b9940 con 0x55b0f77ab9b0 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f2604391740 10 client.0 _ll_register_callbacks cb 0x55b0f7699cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36170/0 (socket says 172.21.15.28:36170) 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fb200640 1 -- 172.21.15.28:0/1661805927 learned_addr learned my addr 172.21.15.28:0/1661805927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fb200640 1 -- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 msgr2=0x55b0f77b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fa800640 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fb200640 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fb200640 1 -- 172.21.15.28:0/1661805927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 msgr2=0x55b0f77b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fb200640 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fb200640 1 -- 172.21.15.28:0/1661805927 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0f74f0620 con 0x55b0f77b6b90 2024-04-26T01:44:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.838+0000 7f25fa800640 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:44:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.839+0000 7f25fb200640 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77b4630 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f25f40165e0 tx=0x7f25f4016610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fd254733405a14f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.839+0000 7f25dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.839+0000 7f25dbe00640 1 -- 172.21.15.28:0/1661805927 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4030060 con 0x55b0f77b6b90 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.839+0000 7f25dbe00640 1 -- 172.21.15.28:0/1661805927 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f25f4003c20 con 0x55b0f77b6b90 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.839+0000 7f2604391740 1 -- 172.21.15.28:0/1661805927 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25f4022440 con 0x55b0f77b6b90 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.840+0000 7f25dbe00640 1 -- 172.21.15.28:0/1661805927 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f25f4026070 con 0x55b0f77b6b90 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.840+0000 7f2604391740 10 client.25565 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.840+0000 7f2604391740 1 -- 172.21.15.28:0/1661805927 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0f77c5a20 con 0x55b0f77b6b90 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.840+0000 7f25dbe00640 1 -- 172.21.15.28:0/1661805927 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25f4022440 con 0x55b0f77b6b90 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.840+0000 7f25dbe00640 1 -- 172.21.15.28:0/1661805927 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f25f4033050 con 0x55b0f77b6b90 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.840+0000 7f2604391740 10 client.25565 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.841+0000 7f2604391740 10 client.25565 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.841+0000 7f2604391740 20 client.25565 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.841+0000 7f2604391740 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b0f77c6dc0 0x55b0f77c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.841+0000 7f2604391740 4 client.25565 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.841+0000 7f2604391740 1 -- 172.21.15.28:0/1661805927 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0f77258b0 con 0x55b0f77c6dc0 2024-04-26T01:44:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.841+0000 7f25fa800640 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b0f77c6dc0 0x55b0f77c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.842+0000 7f25dbe00640 10 client.25565 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:19.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.845+0000 7f25fa800640 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b0f77c6dc0 0x55b0f77c91b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4d0f0414f5a1029 server_cookie=aec1db35b7387e69 in_seq=0 out_seq=0 2024-04-26T01:44:19.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.846+0000 7f25dbe00640 1 -- 172.21.15.28:0/1661805927 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0f77258b0 con 0x55b0f77c6dc0 2024-04-26T01:44:19.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.846+0000 7f25dbe00640 10 client.25565 handle_command_reply: tid=0 2024-04-26T01:44:19.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.846+0000 7f2604391740 1 client.25565 shutdown 2024-04-26T01:44:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.846+0000 7f2604391740 1 -- 172.21.15.28:0/1661805927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 msgr2=0x55b0f77b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f2604391740 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77b4630 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f25f40165e0 tx=0x7f25f4016610 comp rx=0 tx=0).stop 2024-04-26T01:44:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f2604391740 1 -- 172.21.15.28:0/1661805927 shutdown_connections 2024-04-26T01:44:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f2604391740 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b0f77c6dc0 0x55b0f77c91b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f25dbe00640 0 client.25565 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f2604391740 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77b4630 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f2604391740 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f2604391740 1 --2- 172.21.15.28:0/1661805927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f2604391740 1 -- 172.21.15.28:0/1661805927 >> 172.21.15.28:0/1661805927 conn(0x55b0f77a7ec0 msgr2=0x55b0f77adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.847+0000 7f2604391740 1 -- 172.21.15.28:0/1661805927 shutdown_connections 2024-04-26T01:44:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.848+0000 7f2604391740 1 -- 172.21.15.28:0/1661805927 wait complete. 2024-04-26T01:44:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.848+0000 7f2604391740 20 client.25565 trim_cache size 0 max 16384 2024-04-26T01:44:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.876+0000 7f2604391740 1 Processor -- start 2024-04-26T01:44:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.876+0000 7f2604391740 1 -- start start 2024-04-26T01:44:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.877+0000 7f2604391740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ab9b0 0x55b0f77c2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.877+0000 7f2604391740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 0x55b0f77a77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.877+0000 7f2604391740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77b6b90 0x55b0f77ada70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.877+0000 7f2604391740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1b24d0 con 0x55b0f77ab9b0 2024-04-26T01:44:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.877+0000 7f2604391740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1afda0 con 0x55b0f77ac2d0 2024-04-26T01:44:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.877+0000 7f2604391740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1adbb0 con 0x55b0f77b6b90 2024-04-26T01:44:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.877+0000 7f25fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ab9b0 0x55b0f77c2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.877+0000 7f25fb200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77b6b90 0x55b0f77ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.878+0000 7f25fb200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77b6b90 0x55b0f77ada70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51786/0 (socket says 172.21.15.28:51786) 2024-04-26T01:44:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.878+0000 7f25fb200640 1 -- 172.21.15.28:0/3355166248 learned_addr learned my addr 172.21.15.28:0/3355166248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.878+0000 7f25fb200640 1 -- 172.21.15.28:0/3355166248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 msgr2=0x55b0f77a77a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:44:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.878+0000 7f25fb200640 1 --2- 172.21.15.28:0/3355166248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 0x55b0f77a77a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.878+0000 7f25fb200640 1 -- 172.21.15.28:0/3355166248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ab9b0 msgr2=0x55b0f77c2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.878+0000 7f25fb200640 1 --2- 172.21.15.28:0/3355166248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ab9b0 0x55b0f77c2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.878+0000 7f25fb200640 1 -- 172.21.15.28:0/3355166248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0f77a7ce0 con 0x55b0f77b6b90 2024-04-26T01:44:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.878+0000 7f25fa800640 1 --2- 172.21.15.28:0/3355166248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ab9b0 0x55b0f77c2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:19.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.880+0000 7f25fb200640 1 --2- 172.21.15.28:0/3355166248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77b6b90 0x55b0f77ada70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f25f4000f80 tx=0x7f25f402ae70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48839bbacebcc463 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.881+0000 7f25e6a00640 1 -- 172.21.15.28:0/3355166248 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f403a070 con 0x55b0f77b6b90 2024-04-26T01:44:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.881+0000 7f25e6a00640 1 -- 172.21.15.28:0/3355166248 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25f4027040 con 0x55b0f77b6b90 2024-04-26T01:44:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.881+0000 7f25e6a00640 1 -- 172.21.15.28:0/3355166248 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4026070 con 0x55b0f77b6b90 2024-04-26T01:44:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.882+0000 7f2604391740 1 -- 172.21.15.28:0/3355166248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77b6b90 msgr2=0x55b0f77ada70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.882+0000 7f2604391740 1 --2- 172.21.15.28:0/3355166248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77b6b90 0x55b0f77ada70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f25f4000f80 tx=0x7f25f402ae70 comp rx=0 tx=0).stop 2024-04-26T01:44:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.882+0000 7f2604391740 1 -- 172.21.15.28:0/3355166248 shutdown_connections 2024-04-26T01:44:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.882+0000 7f2604391740 1 --2- 172.21.15.28:0/3355166248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77b6b90 0x55b0f77ada70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.882+0000 7f2604391740 1 --2- 172.21.15.28:0/3355166248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77ac2d0 0x55b0f77a77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.882+0000 7f2604391740 1 --2- 172.21.15.28:0/3355166248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ab9b0 0x55b0f77c2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.882+0000 7f2604391740 1 -- 172.21.15.28:0/3355166248 >> 172.21.15.28:0/3355166248 conn(0x55b0f77a7ec0 msgr2=0x55b0f77ad360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.883+0000 7f2604391740 1 -- 172.21.15.28:0/3355166248 shutdown_connections 2024-04-26T01:44:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.883+0000 7f2604391740 1 -- 172.21.15.28:0/3355166248 wait complete. 2024-04-26T01:44:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.883+0000 7f2604391740 1 -- start start 2024-04-26T01:44:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.883+0000 7f2604391740 1 Processor -- start 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f2604391740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77bc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f2604391740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77bcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f2604391740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77bd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f2604391740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ae260 con 0x55b0f77ac2d0 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f2604391740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1af6d0 con 0x55b0f77b6b90 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f2604391740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ae930 con 0x55b0f77ab9b0 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f2604391740 10 client.0 _ll_register_callbacks cb 0x7f25f4009330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f25fa800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77bc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f25fb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77bd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f25fa800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77bc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51794/0 (socket says 172.21.15.28:51794) 2024-04-26T01:44:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.884+0000 7f25fa800640 1 -- 172.21.15.28:0/2849131225 learned_addr learned my addr 172.21.15.28:0/2849131225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25f9e00640 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77bcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25fa800640 1 -- 172.21.15.28:0/2849131225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 msgr2=0x55b0f77bd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25fa800640 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77bd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25fa800640 1 -- 172.21.15.28:0/2849131225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 msgr2=0x55b0f77bcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25fa800640 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77bcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25fa800640 1 -- 172.21.15.28:0/2849131225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0f7834bf0 con 0x55b0f77ab9b0 2024-04-26T01:44:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25f9e00640 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77bcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:44:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25fa800640 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77bc6d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0bcad0 tx=0x7f25ec03b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57fb6a0c95f26df5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.885+0000 7f25dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:44:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.886+0000 7f25dbe00640 1 -- 172.21.15.28:0/2849131225 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec0a0060 con 0x55b0f77ab9b0 2024-04-26T01:44:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.886+0000 7f25dbe00640 1 -- 172.21.15.28:0/2849131225 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f25ec0bb0d0 con 0x55b0f77ab9b0 2024-04-26T01:44:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.886+0000 7f2604391740 1 -- 172.21.15.28:0/2849131225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25f4027040 con 0x55b0f77ab9b0 2024-04-26T01:44:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f25dbe00640 1 -- 172.21.15.28:0/2849131225 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f25ec049040 con 0x55b0f77ab9b0 2024-04-26T01:44:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f2604391740 10 client.25591 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f2604391740 1 -- 172.21.15.28:0/2849131225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0f7835f00 con 0x55b0f77ab9b0 2024-04-26T01:44:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f25dbe00640 1 -- 172.21.15.28:0/2849131225 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25f4027040 con 0x55b0f77ab9b0 2024-04-26T01:44:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f25dbe00640 1 -- 172.21.15.28:0/2849131225 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f25ec0bd030 con 0x55b0f77ab9b0 2024-04-26T01:44:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f2604391740 10 client.25591 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f2604391740 10 client.25591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f2604391740 20 client.25591 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f2604391740 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b0f78365c0 0x55b0f78389e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f2604391740 4 client.25591 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:44:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.887+0000 7f2604391740 1 -- 172.21.15.28:0/2849131225 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b0f771f320 con 0x55b0f78365c0 2024-04-26T01:44:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.888+0000 7f25f9e00640 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b0f78365c0 0x55b0f78389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.888+0000 7f25f9e00640 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b0f78365c0 0x55b0f78389e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8785f869bada218d server_cookie=abc41f4882b5e1d6 in_seq=0 out_seq=0 2024-04-26T01:44:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.888+0000 7f25dbe00640 10 client.25591 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.897+0000 7f25dbe00640 1 -- 172.21.15.28:0/2849131225 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b0f771f320 con 0x55b0f78365c0 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.897+0000 7f25dbe00640 10 client.25591 handle_command_reply: tid=0 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.897+0000 7f2604391740 1 client.25591 shutdown 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 -- 172.21.15.28:0/2849131225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 msgr2=0x55b0f77bc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77bc6d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0bcad0 tx=0x7f25ec03b790 comp rx=0 tx=0).stop 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 -- 172.21.15.28:0/2849131225 shutdown_connections 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b0f78365c0 0x55b0f78389e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b0f77b6b90 0x55b0f77bd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f77ac2d0 0x55b0f77bcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 --2- 172.21.15.28:0/2849131225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0f77ab9b0 0x55b0f77bc6d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f25dbe00640 0 client.25591 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 -- 172.21.15.28:0/2849131225 >> 172.21.15.28:0/2849131225 conn(0x55b0f77a7ec0 msgr2=0x55b0f77b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 -- 172.21.15.28:0/2849131225 shutdown_connections 2024-04-26T01:44:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 1 -- 172.21.15.28:0/2849131225 wait complete. 2024-04-26T01:44:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.898+0000 7f2604391740 20 client.25591 trim_cache size 0 max 16384 2024-04-26T01:44:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:44:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:44:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:44:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:44:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.899+0000 7f2600a00640 1 -- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f25e00a3d90 msgr2=0x7f25e00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.899+0000 7f2600a00640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f25e00a3d90 0x7f25e00a6250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f25f401f000 tx=0x7f25f4018000 comp rx=0 tx=0).stop 2024-04-26T01:44:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.899+0000 7f2600a00640 1 -- 172.21.15.28:0/4070053319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc100b30 msgr2=0x7f25fc1a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.899+0000 7f2600a00640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc100b30 0x7f25fc1a05b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f25ec031440 tx=0x7f25ec003b90 comp rx=0 tx=0).stop 2024-04-26T01:44:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.900+0000 7f2600a00640 1 -- 172.21.15.28:0/4070053319 shutdown_connections 2024-04-26T01:44:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.900+0000 7f2600a00640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f25e00a3d90 0x7f25e00a6250 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.900+0000 7f2600a00640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc10e6e0 0x7f25fc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.900+0000 7f2600a00640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25fc1014e0 0x7f25fc1a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.900+0000 7f2600a00640 1 --2- 172.21.15.28:0/4070053319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc100b30 0x7f25fc1a05b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.900+0000 7f2600a00640 1 -- 172.21.15.28:0/4070053319 >> 172.21.15.28:0/4070053319 conn(0x7f25fc0fc770 msgr2=0x7f25fc0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.900+0000 7f2600a00640 1 -- 172.21.15.28:0/4070053319 shutdown_connections 2024-04-26T01:44:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:19.900+0000 7f2600a00640 1 -- 172.21.15.28:0/4070053319 wait complete. 2024-04-26T01:44:19.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:19 smithi169 ceph-mon[32851]: pgmap v103: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 724 KiB/s wr, 34 op/s 2024-04-26T01:44:20.524 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b7020b43', seconds elapsed: 47.9 2024-04-26T01:44:20.524 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-26T01:44:20.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:44:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 145.0 s, 131.3 tps, lat 6074.732 ms stddev 13735.517 2024-04-26T01:44:21.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:21 smithi028 ceph-mon[28247]: pgmap v104: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 63 op/s 2024-04-26T01:44:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:21 smithi053 ceph-mon[32593]: pgmap v104: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 63 op/s 2024-04-26T01:44:21.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:21 smithi169 ceph-mon[32851]: pgmap v104: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.4 MiB/s wr, 63 op/s 2024-04-26T01:44:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.534+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4090551962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84074c70 msgr2=0x7fdd84075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.534+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4090551962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84074c70 0x7fdd84075050 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009990 tx=0x7fdd78009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.534+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4090551962 shutdown_connections 2024-04-26T01:44:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.534+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4090551962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8410f440 0x7fdd84111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.534+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4090551962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84074c70 0x7fdd84075050 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.534+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4090551962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd84075590 0x7fdd8410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.534+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4090551962 >> 172.21.15.28:0/4090551962 conn(0x7fdd84079ba0 msgr2=0x7fdd84079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.535+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4090551962 shutdown_connections 2024-04-26T01:44:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.535+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4090551962 wait complete. 2024-04-26T01:44:23.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.536+0000 7fdd8bc00700 1 Processor -- start 2024-04-26T01:44:23.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.536+0000 7fdd8bc00700 1 -- start start 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd8bc00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd84074c70 0x7fdd8419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd8bc00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84075590 0x7fdd8419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd8bc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8410f440 0x7fdd8419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd8bc00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84114e40 con 0x7fdd8410f440 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd8bc00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84114cc0 con 0x7fdd84074c70 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd8bc00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdd84114fc0 con 0x7fdd84075590 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd89200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd84074c70 0x7fdd8419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd89200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd84074c70 0x7fdd8419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36222/0 (socket says 172.21.15.28:36222) 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd83e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84075590 0x7fdd8419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:23.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd89200700 1 -- 172.21.15.28:0/4012477595 learned_addr learned my addr 172.21.15.28:0/4012477595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:23.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd89c00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8410f440 0x7fdd8419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:23.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd83e00700 1 -- 172.21.15.28:0/4012477595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd84074c70 msgr2=0x7fdd8419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:23.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd83e00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd84074c70 0x7fdd8419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd83e00700 1 -- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8410f440 msgr2=0x7fdd8419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:23.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.537+0000 7fdd83e00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8410f440 0x7fdd8419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.538+0000 7fdd83e00700 1 -- 172.21.15.28:0/4012477595 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd841a02d0 con 0x7fdd84075590 2024-04-26T01:44:23.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.539+0000 7fdd89200700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd84074c70 0x7fdd8419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:23.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.539+0000 7fdd83e00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84075590 0x7fdd8419b850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000d740 tx=0x7fdd7000da50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:23.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.539+0000 7fdd81600700 1 -- 172.21.15.28:0/4012477595 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd70017070 con 0x7fdd84075590 2024-04-26T01:44:23.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.539+0000 7fdd81600700 1 -- 172.21.15.28:0/4012477595 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd7000f690 con 0x7fdd84075590 2024-04-26T01:44:23.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.539+0000 7fdd81600700 1 -- 172.21.15.28:0/4012477595 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd70003b50 con 0x7fdd84075590 2024-04-26T01:44:23.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.539+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd841a05c0 con 0x7fdd84075590 2024-04-26T01:44:23.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.539+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd841a7e40 con 0x7fdd84075590 2024-04-26T01:44:23.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.540+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd84069730 con 0x7fdd84075590 2024-04-26T01:44:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.543+0000 7fdd81600700 1 -- 172.21.15.28:0/4012477595 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdd7000f840 con 0x7fdd84075590 2024-04-26T01:44:23.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.544+0000 7fdd81600700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd6c0a3e30 0x7fdd6c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:23.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.544+0000 7fdd81600700 1 -- 172.21.15.28:0/4012477595 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fdd700cfbc0 con 0x7fdd84075590 2024-04-26T01:44:23.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.545+0000 7fdd89200700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd6c0a3e30 0x7fdd6c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.545+0000 7fdd89200700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd6c0a3e30 0x7fdd6c0a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd78006010 tx=0x7fdd7800b280 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:23.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.549+0000 7fdd81600700 1 -- 172.21.15.28:0/4012477595 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd7009c230 con 0x7fdd84075590 2024-04-26T01:44:23.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:23 smithi053 ceph-mon[32593]: pgmap v105: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 732 KiB/s wr, 31 op/s 2024-04-26T01:44:23.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:23 smithi028 ceph-mon[28247]: pgmap v105: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 732 KiB/s wr, 31 op/s 2024-04-26T01:44:23.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:23 smithi169 ceph-mon[32851]: pgmap v105: 97 pgs: 97 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 732 KiB/s wr, 31 op/s 2024-04-26T01:44:23.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.934+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd8402df80 con 0x7fdd84075590 2024-04-26T01:44:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.935+0000 7fdd81600700 1 -- 172.21.15.28:0/4012477595 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11864 (secure 0 0 0) 0x7fdd7009bb30 con 0x7fdd84075590 2024-04-26T01:44:23.933 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:44:23.933 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:44:00.873936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24509,24523]},"id":1}]} 2024-04-26T01:44:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.938+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd6c0a3e30 msgr2=0x7fdd6c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.938+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd6c0a3e30 0x7fdd6c0a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd78006010 tx=0x7fdd7800b280 comp rx=0 tx=0).stop 2024-04-26T01:44:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.938+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84075590 msgr2=0x7fdd8419b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.938+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84075590 0x7fdd8419b850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000d740 tx=0x7fdd7000da50 comp rx=0 tx=0).stop 2024-04-26T01:44:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.938+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 shutdown_connections 2024-04-26T01:44:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.939+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd6c0a3e30 0x7fdd6c0a62f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.939+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8410f440 0x7fdd8419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.939+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd84074c70 0x7fdd8419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.939+0000 7fdd8bc00700 1 --2- 172.21.15.28:0/4012477595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd84075590 0x7fdd8419b850 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.939+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 >> 172.21.15.28:0/4012477595 conn(0x7fdd84079ba0 msgr2=0x7fdd840ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.939+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 shutdown_connections 2024-04-26T01:44:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:23.939+0000 7fdd8bc00700 1 -- 172.21.15.28:0/4012477595 wait complete. 2024-04-26T01:44:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-04-26T01:44:24.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:24 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/4012477595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:24.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:24 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/4012477595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:24.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:24 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/4012477595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:24.990 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-26T01:44:24.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-26T01:44:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:25.734 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:25 smithi028 ceph-mon[28247]: pgmap v106: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.5 MiB/s wr, 174 op/s 2024-04-26T01:44:25.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:25 smithi169 ceph-mon[32851]: pgmap v106: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.5 MiB/s wr, 174 op/s 2024-04-26T01:44:25.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:25 smithi053 ceph-mon[32593]: pgmap v106: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.5 MiB/s wr, 174 op/s 2024-04-26T01:44:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 150.0 s, 291.6 tps, lat 325.165 ms stddev 207.621 2024-04-26T01:44:27.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:27 smithi053 ceph-mon[32593]: pgmap v107: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2024-04-26T01:44:27.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:27 smithi028 ceph-mon[28247]: pgmap v107: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2024-04-26T01:44:27.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:27 smithi169 ceph-mon[32851]: pgmap v107: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2024-04-26T01:44:28.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 -- 172.21.15.28:0/965377213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2cc105890 msgr2=0x7fa2cc1098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/965377213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2cc105890 0x7fa2cc1098e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc009990 tx=0x7fa2bc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 -- 172.21.15.28:0/965377213 shutdown_connections 2024-04-26T01:44:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/965377213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2cc109fc0 0x7fa2cc111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/965377213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2cc104ee0 0x7fa2cc1052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/965377213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2cc105890 0x7fa2cc1098e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 -- 172.21.15.28:0/965377213 >> 172.21.15.28:0/965377213 conn(0x7fa2cc100770 msgr2=0x7fa2cc102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 -- 172.21.15.28:0/965377213 shutdown_connections 2024-04-26T01:44:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.710+0000 7fa2d2a00700 1 -- 172.21.15.28:0/965377213 wait complete. 2024-04-26T01:44:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.712+0000 7fa2d2a00700 1 Processor -- start 2024-04-26T01:44:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.712+0000 7fa2d2a00700 1 -- start start 2024-04-26T01:44:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2d2a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2cc104ee0 0x7fa2cc19b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2d2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2cc109fc0 0x7fa2cc19bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2d2a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2cc19fe40 0x7fa2cc1a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2d2a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc114f00 con 0x7fa2cc109fc0 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2d2a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc114d80 con 0x7fa2cc104ee0 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2d2a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc115080 con 0x7fa2cc19fe40 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2cb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2cc109fc0 0x7fa2cc19bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2cb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2cc109fc0 0x7fa2cc19bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51120/0 (socket says 172.21.15.28:51120) 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2cb400700 1 -- 172.21.15.28:0/3790598077 learned_addr learned my addr 172.21.15.28:0/3790598077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.715+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2cc19fe40 0x7fa2cc1a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.716+0000 7fa2cb400700 1 -- 172.21.15.28:0/3790598077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2cc19fe40 msgr2=0x7fa2cc1a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.716+0000 7fa2cb400700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2cc19fe40 0x7fa2cc1a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.716+0000 7fa2cb400700 1 -- 172.21.15.28:0/3790598077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2cc104ee0 msgr2=0x7fa2cc19b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.716+0000 7fa2cb400700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2cc104ee0 0x7fa2cc19b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.716+0000 7fa2cb400700 1 -- 172.21.15.28:0/3790598077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2cc1a0960 con 0x7fa2cc109fc0 2024-04-26T01:44:28.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.716+0000 7fa2cb400700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2cc109fc0 0x7fa2cc19bb00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:28.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.717+0000 7fa2c8c00700 1 -- 172.21.15.28:0/3790598077 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc026070 con 0x7fa2cc109fc0 2024-04-26T01:44:28.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.717+0000 7fa2d2a00700 1 -- 172.21.15.28:0/3790598077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2cc1a0bf0 con 0x7fa2cc109fc0 2024-04-26T01:44:28.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.717+0000 7fa2c8c00700 1 -- 172.21.15.28:0/3790598077 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2bc004020 con 0x7fa2cc109fc0 2024-04-26T01:44:28.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.717+0000 7fa2c8c00700 1 -- 172.21.15.28:0/3790598077 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc00fc00 con 0x7fa2cc109fc0 2024-04-26T01:44:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.717+0000 7fa2d2a00700 1 -- 172.21.15.28:0/3790598077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2cc1a8310 con 0x7fa2cc109fc0 2024-04-26T01:44:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.724+0000 7fa2dd45db80 1 Processor -- start 2024-04-26T01:44:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.724+0000 7fa2dd45db80 1 -- start start 2024-04-26T01:44:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.725+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e4cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.725+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4d2d0 0x556f68e57c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 0x556f68e5a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556f68da0020 con 0x556f68e58180 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2d0a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e4cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556f68d468c0 con 0x556f68e4d2d0 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x556f68d9bfe0 con 0x556f68e4c9b0 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2d0a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e4cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51840/0 (socket says 172.21.15.28:51840) 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2d0a00700 1 -- 172.21.15.28:0/3421683155 learned_addr learned my addr 172.21.15.28:0/3421683155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/3421683155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4d2d0 0x556f68e57c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2d0a00700 1 -- 172.21.15.28:0/3421683155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4d2d0 msgr2=0x556f68e57c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.726+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/3421683155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4d2d0 0x556f68e57c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.727+0000 7fa2d0a00700 1 -- 172.21.15.28:0/3421683155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 msgr2=0x556f68e5a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.727+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/3421683155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 0x556f68e5a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.727+0000 7fa2d0a00700 1 -- 172.21.15.28:0/3421683155 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f68d3c810 con 0x556f68e4c9b0 2024-04-26T01:44:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.727+0000 7fa2c8c00700 1 -- 172.21.15.28:0/3790598077 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa2bc029070 con 0x7fa2cc109fc0 2024-04-26T01:44:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.727+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/3421683155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e4cd90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4015bf0 tx=0x7fa2c40078d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7273301d86ecbeab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.728+0000 7fa2c1600700 1 -- 172.21.15.28:0/3421683155 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4003a00 con 0x556f68e4c9b0 2024-04-26T01:44:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.728+0000 7fa2c1600700 1 -- 172.21.15.28:0/3421683155 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c400adf0 con 0x556f68e4c9b0 2024-04-26T01:44:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.728+0000 7fa2c1600700 1 -- 172.21.15.28:0/3421683155 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4004580 con 0x556f68e4c9b0 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.729+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3421683155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 msgr2=0x556f68e4cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.729+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/3421683155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e4cd90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4015bf0 tx=0x7fa2c40078d0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.729+0000 7fa2c8c00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa2b00a3f50 0x7fa2b00a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3421683155 shutdown_connections 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/3421683155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 0x556f68e5a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/3421683155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e4cd90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/3421683155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4d2d0 0x556f68e57c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3421683155 >> 172.21.15.28:0/3421683155 conn(0x556f68d7b730 msgr2=0x556f68e48e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3421683155 shutdown_connections 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2c8c00700 1 -- 172.21.15.28:0/3790598077 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fa2bc0d1220 con 0x7fa2cc109fc0 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa2b00a3f50 0x7fa2b00a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.730+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3421683155 wait complete. 2024-04-26T01:44:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.731+0000 7fa2dd45db80 1 -- start start 2024-04-26T01:44:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.731+0000 7fa2dd45db80 1 Processor -- start 2024-04-26T01:44:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.731+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa2b00a3f50 0x7fa2b00a6410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400b810 tx=0x7fa2b4008040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.732+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4c9b0 0x556f68e57960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.732+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4d2d0 0x556f68e54c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.732+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 0x556f68e551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.732+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556f68e5aca0 con 0x556f68e58180 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.732+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556f68d599f0 con 0x556f68e4c9b0 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.732+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x556f68e5ae20 con 0x556f68e4d2d0 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.732+0000 7fa2dd45db80 10 client.0 _ll_register_callbacks cb 0x556f68d388b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.732+0000 7fa2cb400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4d2d0 0x556f68e54c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2cb400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4d2d0 0x556f68e54c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51846/0 (socket says 172.21.15.28:51846) 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2cb400700 1 -- 172.21.15.28:0/2275762631 learned_addr learned my addr 172.21.15.28:0/2275762631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 0x556f68e551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4c9b0 0x556f68e57960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2cb400700 1 -- 172.21.15.28:0/2275762631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4c9b0 msgr2=0x556f68e57960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2cb400700 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4c9b0 0x556f68e57960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2cb400700 1 -- 172.21.15.28:0/2275762631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 msgr2=0x556f68e551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2cb400700 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 0x556f68e551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.733+0000 7fa2cb400700 1 -- 172.21.15.28:0/2275762631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f68ceda60 con 0x556f68e4d2d0 2024-04-26T01:44:28.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.734+0000 7fa2cb400700 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4d2d0 0x556f68e54c60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0a58d0 tx=0x7fa2bc0ceea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=add17bf6f0fff9e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:28.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.735+0000 7fa2ba000700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:44:28.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.735+0000 7fa2ba000700 1 -- 172.21.15.28:0/2275762631 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc09cb80 con 0x556f68e4d2d0 2024-04-26T01:44:28.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.735+0000 7fa2ba000700 1 -- 172.21.15.28:0/2275762631 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fa2bc0a5b70 con 0x556f68e4d2d0 2024-04-26T01:44:28.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.735+0000 7fa2dd45db80 1 -- 172.21.15.28:0/2275762631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c400adf0 con 0x556f68e4d2d0 2024-04-26T01:44:28.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.736+0000 7fa2ba000700 1 -- 172.21.15.28:0/2275762631 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa2bc0041d0 con 0x556f68e4d2d0 2024-04-26T01:44:28.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.736+0000 7fa2dd45db80 10 client.25606 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:28.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.736+0000 7fa2dd45db80 1 -- 172.21.15.28:0/2275762631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f68d36c80 con 0x556f68e4d2d0 2024-04-26T01:44:28.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.737+0000 7fa2ba000700 1 -- 172.21.15.28:0/2275762631 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c400adf0 con 0x556f68e4d2d0 2024-04-26T01:44:28.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.737+0000 7fa2ba000700 1 -- 172.21.15.28:0/2275762631 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fa2bc09c670 con 0x556f68e4d2d0 2024-04-26T01:44:28.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.737+0000 7fa2dd45db80 10 client.25606 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:28.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.737+0000 7fa2dd45db80 10 client.25606 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:44:28.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.737+0000 7fa2dd45db80 20 client.25606 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:28.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.737+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556f68e67950 0x556f68e69d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.737+0000 7fa2dd45db80 4 client.25606 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:28.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.737+0000 7fa2dd45db80 1 -- 172.21.15.28:0/2275762631 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f68dad810 con 0x556f68e67950 2024-04-26T01:44:28.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.738+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556f68e67950 0x556f68e69d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.739+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556f68e67950 0x556f68e69d70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e07e8437a257788 server_cookie=efbeffa30a0a162c in_seq=0 out_seq=0 2024-04-26T01:44:28.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.739+0000 7fa2ba000700 10 client.25606 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:28.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.740+0000 7fa2ba000700 1 -- 172.21.15.28:0/2275762631 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f68dad810 con 0x556f68e67950 2024-04-26T01:44:28.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.740+0000 7fa2ba000700 10 client.25606 handle_command_reply: tid=0 2024-04-26T01:44:28.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.740+0000 7fa2dd45db80 1 client.25606 shutdown 2024-04-26T01:44:28.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 -- 172.21.15.28:0/2275762631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4d2d0 msgr2=0x556f68e54c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4d2d0 0x556f68e54c60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0a58d0 tx=0x7fa2bc0ceea0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 -- 172.21.15.28:0/2275762631 shutdown_connections 2024-04-26T01:44:28.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e58180 0x556f68e551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e4c9b0 0x556f68e57960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556f68e67950 0x556f68e69d70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/2275762631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4d2d0 0x556f68e54c60 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0a58d0 tx=0x7fa2bc0ceea0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 -- 172.21.15.28:0/2275762631 >> 172.21.15.28:0/2275762631 conn(0x556f68d7b730 msgr2=0x556f68e48a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2ba000700 0 client.25606 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 -- 172.21.15.28:0/2275762631 shutdown_connections 2024-04-26T01:44:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.741+0000 7fa2dd45db80 1 -- 172.21.15.28:0/2275762631 wait complete. 2024-04-26T01:44:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.742+0000 7fa2dd45db80 20 client.25606 trim_cache size 0 max 16384 2024-04-26T01:44:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.778+0000 7fa2dd45db80 1 Processor -- start 2024-04-26T01:44:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.778+0000 7fa2dd45db80 1 -- start start 2024-04-26T01:44:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e48f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 0x556f68e55150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e6faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc071c40 con 0x556f68e6f620 2024-04-26T01:44:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc06f510 con 0x556f68e58180 2024-04-26T01:44:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc07af60 con 0x556f68e4c9b0 2024-04-26T01:44:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2cbe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e48f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2cbe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e48f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51854/0 (socket says 172.21.15.28:51854) 2024-04-26T01:44:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2cbe00700 1 -- 172.21.15.28:0/3427485877 learned_addr learned my addr 172.21.15.28:0/3427485877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.779+0000 7fa2cb400700 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 0x556f68e55150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.780+0000 7fa2cbe00700 1 -- 172.21.15.28:0/3427485877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 msgr2=0x556f68e55150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.780+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 0x556f68e55150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.780+0000 7fa2cbe00700 1 -- 172.21.15.28:0/3427485877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 msgr2=0x556f68e6faa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.780+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e6faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.780+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e6faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.780+0000 7fa2cbe00700 1 -- 172.21.15.28:0/3427485877 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f68e6a350 con 0x556f68e4c9b0 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.780+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e6faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.780+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e48f70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400b5f0 tx=0x7fa2b400bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f32fbf783acbcdeb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.781+0000 7fa2c0c00700 1 -- 172.21.15.28:0/3427485877 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b402a030 con 0x556f68e4c9b0 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.781+0000 7fa2c0c00700 1 -- 172.21.15.28:0/3427485877 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b4025590 con 0x556f68e4c9b0 2024-04-26T01:44:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.781+0000 7fa2c0c00700 1 -- 172.21.15.28:0/3427485877 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4023960 con 0x556f68e4c9b0 2024-04-26T01:44:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.782+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3427485877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 msgr2=0x556f68e48f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.782+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e48f70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400b5f0 tx=0x7fa2b400bf90 comp rx=0 tx=0).stop 2024-04-26T01:44:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.782+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3427485877 shutdown_connections 2024-04-26T01:44:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.782+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e6faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.782+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e48f70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.782+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/3427485877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 0x556f68e55150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.782+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3427485877 >> 172.21.15.28:0/3427485877 conn(0x556f68d7b730 msgr2=0x556f68e54a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:28.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.783+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3427485877 shutdown_connections 2024-04-26T01:44:28.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.783+0000 7fa2dd45db80 1 -- 172.21.15.28:0/3427485877 wait complete. 2024-04-26T01:44:28.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.783+0000 7fa2dd45db80 1 -- start start 2024-04-26T01:44:28.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.784+0000 7fa2dd45db80 1 Processor -- start 2024-04-26T01:44:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.784+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e60050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.784+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 0x556f68e60590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.784+0000 7fa2dd45db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e5e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.784+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc06d9d0 con 0x556f68e6f620 2024-04-26T01:44:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.784+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc06ee40 con 0x556f68e58180 2024-04-26T01:44:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.784+0000 7fa2dd45db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc06e0a0 con 0x556f68e4c9b0 2024-04-26T01:44:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.784+0000 7fa2dd45db80 10 client.0 _ll_register_callbacks cb 0x556f68d388b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.785+0000 7fa2cb400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 0x556f68e60590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.785+0000 7fa2cbe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e60050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.785+0000 7fa2cbe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e60050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51870/0 (socket says 172.21.15.28:51870) 2024-04-26T01:44:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.785+0000 7fa2cbe00700 1 -- 172.21.15.28:0/783119543 learned_addr learned my addr 172.21.15.28:0/783119543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.785+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e5e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.786+0000 7fa2cbe00700 1 -- 172.21.15.28:0/783119543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 msgr2=0x556f68e60590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.786+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 0x556f68e60590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.786+0000 7fa2cbe00700 1 -- 172.21.15.28:0/783119543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 msgr2=0x556f68e5e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.786+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e5e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.786+0000 7fa2cbe00700 1 -- 172.21.15.28:0/783119543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f68e5eda0 con 0x556f68e4c9b0 2024-04-26T01:44:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.786+0000 7fa2d0a00700 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e5e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.786+0000 7fa2cbe00700 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e60050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4025a60 tx=0x7fa2b4022420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20925e3950870c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.786+0000 7fa2ba000700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:44:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.787+0000 7fa2ba000700 1 -- 172.21.15.28:0/783119543 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b402a040 con 0x556f68e4c9b0 2024-04-26T01:44:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.787+0000 7fa2dd45db80 1 -- 172.21.15.28:0/783119543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2b4025590 con 0x556f68e4c9b0 2024-04-26T01:44:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.787+0000 7fa2ba000700 1 -- 172.21.15.28:0/783119543 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fa2b400f040 con 0x556f68e4c9b0 2024-04-26T01:44:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.787+0000 7fa2ba000700 1 -- 172.21.15.28:0/783119543 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa2b4025070 con 0x556f68e4c9b0 2024-04-26T01:44:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.787+0000 7fa2dd45db80 10 client.25612 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.787+0000 7fa2dd45db80 1 -- 172.21.15.28:0/783119543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f68ecea90 con 0x556f68e4c9b0 2024-04-26T01:44:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.788+0000 7fa2ba000700 1 -- 172.21.15.28:0/783119543 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b4025590 con 0x556f68e4c9b0 2024-04-26T01:44:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.788+0000 7fa2ba000700 1 -- 172.21.15.28:0/783119543 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fa2b4013070 con 0x556f68e4c9b0 2024-04-26T01:44:28.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.788+0000 7fa2dd45db80 10 client.25612 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:28.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.788+0000 7fa2dd45db80 10 client.25612 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:44:28.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.788+0000 7fa2dd45db80 20 client.25612 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:28.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.788+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556f68ecf160 0x556f68ed1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:28.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.788+0000 7fa2dd45db80 4 client.25612 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-26T01:44:28.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.788+0000 7fa2dd45db80 1 -- 172.21.15.28:0/783119543 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x556f68da78b0 con 0x556f68ecf160 2024-04-26T01:44:28.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.791+0000 7fa2cb400700 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556f68ecf160 0x556f68ed1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:28.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.792+0000 7fa2cb400700 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556f68ecf160 0x556f68ed1580 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=312dbc201e7d76fe server_cookie=7346b2617fb9332c in_seq=0 out_seq=0 2024-04-26T01:44:28.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.792+0000 7fa2ba000700 10 client.25612 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2ba000700 1 -- 172.21.15.28:0/783119543 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x556f68da78b0 con 0x556f68ecf160 2024-04-26T01:44:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2ba000700 10 client.25612 handle_command_reply: tid=0 2024-04-26T01:44:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 client.25612 shutdown 2024-04-26T01:44:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 -- 172.21.15.28:0/783119543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 msgr2=0x556f68e60050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e60050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4025a60 tx=0x7fa2b4022420 comp rx=0 tx=0).stop 2024-04-26T01:44:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 -- 172.21.15.28:0/783119543 shutdown_connections 2024-04-26T01:44:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f68e6f620 0x556f68e5e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556f68ecf160 0x556f68ed1580 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2ba000700 0 client.25612 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:28.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556f68e4c9b0 0x556f68e60050 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 --2- 172.21.15.28:0/783119543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f68e58180 0x556f68e60590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.950+0000 7fa2dd45db80 1 -- 172.21.15.28:0/783119543 >> 172.21.15.28:0/783119543 conn(0x556f68d7b730 msgr2=0x556f68e6e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:28.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.951+0000 7fa2dd45db80 1 -- 172.21.15.28:0/783119543 shutdown_connections 2024-04-26T01:44:28.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.951+0000 7fa2dd45db80 1 -- 172.21.15.28:0/783119543 wait complete. 2024-04-26T01:44:28.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.951+0000 7fa2dd45db80 20 client.25612 trim_cache size 0 max 16384 2024-04-26T01:44:28.949 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:44:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 29, 2024-04-26T01:44:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:44:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 10, 2024-04-26T01:44:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "a7997608": { 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 10, 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 60.0, 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 90.0, 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-26T01:44:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:44:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 -- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa2b00a3f50 msgr2=0x7fa2b00a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa2b00a3f50 0x7fa2b00a6410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400b810 tx=0x7fa2b4008040 comp rx=0 tx=0).stop 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 -- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2cc109fc0 msgr2=0x7fa2cc19bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2cc109fc0 0x7fa2cc19bb00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc0047b0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 -- 172.21.15.28:0/3790598077 shutdown_connections 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fa2b00a3f50 0x7fa2b00a6410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2cc19fe40 0x7fa2cc1a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2cc104ee0 0x7fa2cc19b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 --2- 172.21.15.28:0/3790598077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2cc109fc0 0x7fa2cc19bb00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 -- 172.21.15.28:0/3790598077 >> 172.21.15.28:0/3790598077 conn(0x7fa2cc100770 msgr2=0x7fa2cc100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 -- 172.21.15.28:0/3790598077 shutdown_connections 2024-04-26T01:44:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:44:28.952+0000 7fa2d2a00700 1 -- 172.21.15.28:0/3790598077 wait complete. 2024-04-26T01:44:29.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:29 smithi028 ceph-mon[28247]: pgmap v108: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2024-04-26T01:44:29.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:29 smithi169 ceph-mon[32851]: pgmap v108: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2024-04-26T01:44:29.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:29 smithi053 ceph-mon[32593]: pgmap v108: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2024-04-26T01:44:29.994 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a7997608', quiesce took 9.5 seconds. Will release after: 30.0 2024-04-26T01:44:31.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 155.0 s, 617.9 tps, lat 96.282 ms stddev 254.057 2024-04-26T01:44:31.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:31 smithi028 ceph-mon[28247]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 30 MiB/s wr, 1.38k op/s 2024-04-26T01:44:31.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:31 smithi169 ceph-mon[32851]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 30 MiB/s wr, 1.38k op/s 2024-04-26T01:44:31.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:31 smithi053 ceph-mon[32593]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 30 MiB/s wr, 1.38k op/s 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 -- 172.21.15.28:0/1546216174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9380102990 msgr2=0x7f93801069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 --2- 172.21.15.28:0/1546216174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9380102990 0x7f93801069d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f936c009900 tx=0x7f936c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 -- 172.21.15.28:0/1546216174 shutdown_connections 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 --2- 172.21.15.28:0/1546216174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93801070b0 0x7f938010ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 --2- 172.21.15.28:0/1546216174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9380102990 0x7f93801069d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 --2- 172.21.15.28:0/1546216174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9380101fe0 0x7f93801023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 -- 172.21.15.28:0/1546216174 >> 172.21.15.28:0/1546216174 conn(0x7f93800fd8a0 msgr2=0x7f93800ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 -- 172.21.15.28:0/1546216174 shutdown_connections 2024-04-26T01:44:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.049+0000 7f9386e00640 1 -- 172.21.15.28:0/1546216174 wait complete. 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.050+0000 7f9386e00640 1 Processor -- start 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.050+0000 7f9386e00640 1 -- start start 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.050+0000 7f9386e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9380101fe0 0x7f9380074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.050+0000 7f9386e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93801070b0 0x7f9380074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.050+0000 7f9386e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9380074cd0 0x7f93801a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9386e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9380111fd0 con 0x7f9380101fe0 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9386e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9380111e50 con 0x7f9380074cd0 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9386e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9380112150 con 0x7f93801070b0 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9380101fe0 0x7f9380074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9380101fe0 0x7f9380074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38390/0 (socket says 172.21.15.28:38390) 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385a00640 1 -- 172.21.15.28:0/3875799163 learned_addr learned my addr 172.21.15.28:0/3875799163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385000640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93801070b0 0x7f9380074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9386400640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9380074cd0 0x7f93801a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385a00640 1 -- 172.21.15.28:0/3875799163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93801070b0 msgr2=0x7f9380074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385a00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93801070b0 0x7f9380074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385a00640 1 -- 172.21.15.28:0/3875799163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9380074cd0 msgr2=0x7f93801a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385a00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9380074cd0 0x7f93801a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9385a00640 1 -- 172.21.15.28:0/3875799163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93801a6ea0 con 0x7f9380101fe0 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.051+0000 7f9386400640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9380074cd0 0x7f93801a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.052+0000 7f9385a00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9380101fe0 0x7f9380074250 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f937400ea60 tx=0x7f937400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.052+0000 7f937e000640 1 -- 172.21.15.28:0/3875799163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937400cde0 con 0x7f9380101fe0 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.052+0000 7f9386e00640 1 -- 172.21.15.28:0/3875799163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93801a7190 con 0x7f9380101fe0 2024-04-26T01:44:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.052+0000 7f937e000640 1 -- 172.21.15.28:0/3875799163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93740035f0 con 0x7f9380101fe0 2024-04-26T01:44:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.052+0000 7f937e000640 1 -- 172.21.15.28:0/3875799163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9374010690 con 0x7f9380101fe0 2024-04-26T01:44:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.052+0000 7f9386e00640 1 -- 172.21.15.28:0/3875799163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93801a76b0 con 0x7f9380101fe0 2024-04-26T01:44:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f938b254740 1 Processor -- start 2024-04-26T01:44:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f938b254740 1 -- start start 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f938b254740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a69020ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f938b254740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690219650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f938b254740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a69021bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f938b254740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a690174900 con 0x55a690219b90 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f938b254740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a69011aef0 con 0x55a69020e9b0 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f938b254740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55a6901708c0 con 0x55a69020f2d0 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.056+0000 7f937e000640 1 -- 172.21.15.28:0/3875799163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f93740037a0 con 0x7f9380101fe0 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9385000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a69021bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9386400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a69020ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9386400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a69020ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43834/0 (socket says 172.21.15.28:43834) 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9385000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a69021bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38406/0 (socket says 172.21.15.28:38406) 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9386400640 1 -- 172.21.15.28:0/3144006063 learned_addr learned my addr 172.21.15.28:0/3144006063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9385a00640 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690219650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9386400640 1 -- 172.21.15.28:0/3144006063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 msgr2=0x55a690219650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9386400640 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690219650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9386400640 1 -- 172.21.15.28:0/3144006063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 msgr2=0x55a69021bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9386400640 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a69021bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9386400640 1 -- 172.21.15.28:0/3144006063 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a68fe87210 con 0x55a69020e9b0 2024-04-26T01:44:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.057+0000 7f9385000640 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a69021bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.058+0000 7f9386400640 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a69020ed90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9378009fd0 tx=0x7f9378010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f001391800b8b32 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.058+0000 7f9372a00640 1 -- 172.21.15.28:0/3144006063 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9378011700 con 0x55a69020e9b0 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.058+0000 7f9372a00640 1 -- 172.21.15.28:0/3144006063 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9378011d50 con 0x55a69020e9b0 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.058+0000 7f937e000640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f93680a3d40 0x7f93680a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.058+0000 7f9372a00640 1 -- 172.21.15.28:0/3144006063 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937800e3c0 con 0x55a69020e9b0 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.058+0000 7f937e000640 1 -- 172.21.15.28:0/3875799163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f93740d0b40 con 0x7f9380101fe0 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.058+0000 7f9385a00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f93680a3d40 0x7f93680a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 -- 172.21.15.28:0/3144006063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 msgr2=0x55a69020ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a69020ed90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9378009fd0 tx=0x7f9378010860 comp rx=0 tx=0).stop 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 -- 172.21.15.28:0/3144006063 shutdown_connections 2024-04-26T01:44:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a69021bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690219650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 --2- 172.21.15.28:0/3144006063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a69020ed90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 -- 172.21.15.28:0/3144006063 >> 172.21.15.28:0/3144006063 conn(0x55a69020aec0 msgr2=0x55a69020b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f9385a00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f93680a3d40 0x7f93680a6200 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55a69020ff50 tx=0x7f93740d45f0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 -- 172.21.15.28:0/3144006063 shutdown_connections 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.059+0000 7f938b254740 1 -- 172.21.15.28:0/3144006063 wait complete. 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.060+0000 7f938b254740 1 -- start start 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.060+0000 7f938b254740 1 Processor -- start 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.060+0000 7f938b254740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690216b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.060+0000 7f938b254740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a6902170d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.060+0000 7f938b254740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a690219b90 0x55a690217610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.060+0000 7f938b254740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a69021c7c0 con 0x55a69020f2d0 2024-04-26T01:44:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.060+0000 7f938b254740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a69012e310 con 0x55a69020e9b0 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.060+0000 7f938b254740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55a69021c940 con 0x55a690219b90 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f938b254740 10 client.0 _ll_register_callbacks cb 0x55a6900fccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9386400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690216b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9385000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a6902170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9386400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690216b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43848/0 (socket says 172.21.15.28:43848) 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9386400640 1 -- 172.21.15.28:0/368100953 learned_addr learned my addr 172.21.15.28:0/368100953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9386400640 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a690219b90 0x55a690217610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9385000640 1 -- 172.21.15.28:0/368100953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a690219b90 msgr2=0x55a690217610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9385000640 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a690219b90 0x55a690217610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9385000640 1 -- 172.21.15.28:0/368100953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 msgr2=0x55a690216b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9385000640 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690216b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.061+0000 7f9385000640 1 -- 172.21.15.28:0/368100953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a68ff53620 con 0x55a69020f2d0 2024-04-26T01:44:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.062+0000 7f9385000640 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a6902170d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f936c006010 tx=0x7f936c032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4666147015f8872f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.062+0000 7f9367400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:44:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.062+0000 7f9367400640 1 -- 172.21.15.28:0/368100953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c035040 con 0x55a69020f2d0 2024-04-26T01:44:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.062+0000 7f9367400640 1 -- 172.21.15.28:0/368100953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f936c0316a0 con 0x55a69020f2d0 2024-04-26T01:44:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.062+0000 7f938b254740 1 -- 172.21.15.28:0/368100953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9378011d50 con 0x55a69020f2d0 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.062+0000 7f9367400640 1 -- 172.21.15.28:0/368100953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f936c0318a0 con 0x55a69020f2d0 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.062+0000 7f938b254740 10 client.15453 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.062+0000 7f938b254740 1 -- 172.21.15.28:0/368100953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a690228a80 con 0x55a69020f2d0 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f9367400640 1 -- 172.21.15.28:0/368100953 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9378011d50 con 0x55a69020f2d0 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f9367400640 1 -- 172.21.15.28:0/368100953 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f936c004750 con 0x55a69020f2d0 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f938b254740 10 client.15453 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f938b254740 10 client.15453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f938b254740 20 client.15453 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f938b254740 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a690229ef0 0x55a69022c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f938b254740 4 client.15453 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f938b254740 1 -- 172.21.15.28:0/368100953 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6901888b0 con 0x55a690229ef0 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.063+0000 7f9386400640 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a690229ef0 0x55a69022c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.064+0000 7f9386400640 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a690229ef0 0x55a69022c2e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8abdccbf3aa8184 server_cookie=617b8ecbe7cb158f in_seq=0 out_seq=0 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.064+0000 7f9367400640 10 client.15453 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f9367400640 1 -- 172.21.15.28:0/368100953 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6901888b0 con 0x55a690229ef0 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f9367400640 10 client.15453 handle_command_reply: tid=0 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 client.15453 shutdown 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 -- 172.21.15.28:0/368100953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 msgr2=0x55a6902170d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a6902170d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f936c006010 tx=0x7f936c032e90 comp rx=0 tx=0).stop 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 -- 172.21.15.28:0/368100953 shutdown_connections 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a690229ef0 0x55a69022c2e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a690219b90 0x55a690217610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a6902170d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 --2- 172.21.15.28:0/368100953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690216b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f9367400640 0 client.15453 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 -- 172.21.15.28:0/368100953 >> 172.21.15.28:0/368100953 conn(0x55a69020aec0 msgr2=0x55a690210bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 -- 172.21.15.28:0/368100953 shutdown_connections 2024-04-26T01:44:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.065+0000 7f938b254740 1 -- 172.21.15.28:0/368100953 wait complete. 2024-04-26T01:44:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.066+0000 7f938b254740 20 client.15453 trim_cache size 0 max 16384 2024-04-26T01:44:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.082+0000 7f938b254740 1 Processor -- start 2024-04-26T01:44:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.082+0000 7f938b254740 1 -- start start 2024-04-26T01:44:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.082+0000 7f938b254740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690225680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.082+0000 7f938b254740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690231bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.082+0000 7f938b254740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a690232130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.082+0000 7f938b254740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93801b2520 con 0x55a690219b90 2024-04-26T01:44:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.082+0000 7f938b254740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f93801afdf0 con 0x55a69020e9b0 2024-04-26T01:44:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.082+0000 7f938b254740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f93801adc40 con 0x55a69020f2d0 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9385000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690231bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9386400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a690232130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9385000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690231bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:53176/0 (socket says 172.21.15.28:53176) 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9385000640 1 -- 172.21.15.28:0/1317243433 learned_addr learned my addr 172.21.15.28:0/1317243433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9386400640 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690225680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9385000640 1 -- 172.21.15.28:0/1317243433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 msgr2=0x55a690225680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9385000640 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690225680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9385000640 1 -- 172.21.15.28:0/1317243433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 msgr2=0x55a690232130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9385000640 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a690232130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9385000640 1 -- 172.21.15.28:0/1317243433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69021c2f0 con 0x55a69020f2d0 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.083+0000 7f9386400640 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a690232130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.084+0000 7f9385000640 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690231bf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f936c02a910 tx=0x7f936c004f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f3f26a279080cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.084+0000 7f9372000640 1 -- 172.21.15.28:0/1317243433 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c035030 con 0x55a69020f2d0 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.084+0000 7f9372000640 1 -- 172.21.15.28:0/1317243433 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f936c0316a0 con 0x55a69020f2d0 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.084+0000 7f9372000640 1 -- 172.21.15.28:0/1317243433 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c046470 con 0x55a69020f2d0 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.084+0000 7f938b254740 1 -- 172.21.15.28:0/1317243433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 msgr2=0x55a690231bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.084+0000 7f938b254740 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690231bf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f936c02a910 tx=0x7f936c004f20 comp rx=0 tx=0).stop 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 -- 172.21.15.28:0/1317243433 shutdown_connections 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a690219b90 0x55a690232130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020f2d0 0x55a690231bf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 --2- 172.21.15.28:0/1317243433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a69020e9b0 0x55a690225680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 -- 172.21.15.28:0/1317243433 >> 172.21.15.28:0/1317243433 conn(0x55a69020aec0 msgr2=0x55a690211e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 -- 172.21.15.28:0/1317243433 shutdown_connections 2024-04-26T01:44:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 -- 172.21.15.28:0/1317243433 wait complete. 2024-04-26T01:44:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 -- start start 2024-04-26T01:44:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.085+0000 7f938b254740 1 Processor -- start 2024-04-26T01:44:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f938b254740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020e9b0 0x55a690222010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f938b254740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a69021f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f938b254740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a690219b90 0x55a69021f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f938b254740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93801ae2b0 con 0x55a69020f2d0 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f938b254740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f93801af720 con 0x55a690219b90 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f938b254740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f93801ae980 con 0x55a69020e9b0 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f938b254740 10 client.0 _ll_register_callbacks cb 0x7f936c032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f9385000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a69021f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f9385000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a69021f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38426/0 (socket says 172.21.15.28:38426) 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f9385000640 1 -- 172.21.15.28:0/2340856009 learned_addr learned my addr 172.21.15.28:0/2340856009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f9386400640 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020e9b0 0x55a690222010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.086+0000 7f9386400640 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a690219b90 0x55a69021f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9385000640 1 -- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020e9b0 msgr2=0x55a690222010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9385000640 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020e9b0 0x55a690222010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9385000640 1 -- 172.21.15.28:0/2340856009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a690219b90 msgr2=0x55a69021f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9385000640 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a690219b90 0x55a69021f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9385000640 1 -- 172.21.15.28:0/2340856009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a690220170 con 0x55a69020f2d0 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9386400640 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020e9b0 0x55a690222010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9385000640 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a69021f3a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f936c036040 tx=0x7f936c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2422a256ab15d8de server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9367400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9367400640 1 -- 172.21.15.28:0/2340856009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c004860 con 0x55a69020f2d0 2024-04-26T01:44:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f938b254740 1 -- 172.21.15.28:0/2340856009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f936c0316a0 con 0x55a69020f2d0 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.087+0000 7f9367400640 1 -- 172.21.15.28:0/2340856009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f936c004040 con 0x55a69020f2d0 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f9367400640 1 -- 172.21.15.28:0/2340856009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f936c004240 con 0x55a69020f2d0 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f938b254740 10 client.15459 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f938b254740 1 -- 172.21.15.28:0/2340856009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a690290550 con 0x55a69020f2d0 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f9367400640 1 -- 172.21.15.28:0/2340856009 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f936c0316a0 con 0x55a69020f2d0 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f9367400640 1 -- 172.21.15.28:0/2340856009 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f936c0037f0 con 0x55a69020f2d0 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f938b254740 10 client.15459 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f938b254740 10 client.15459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f938b254740 20 client.15459 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f938b254740 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a690290b50 0x55a690292f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f938b254740 4 client.15459 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:44:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.088+0000 7f938b254740 1 -- 172.21.15.28:0/2340856009 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a690182320 con 0x55a690290b50 2024-04-26T01:44:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.089+0000 7f9386400640 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a690290b50 0x55a690292f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.089+0000 7f9386400640 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a690290b50 0x55a690292f40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=528ad7cfe8932d64 server_cookie=449902002041e0f6 in_seq=0 out_seq=0 2024-04-26T01:44:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.089+0000 7f9367400640 10 client.15459 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.105+0000 7f9367400640 1 -- 172.21.15.28:0/2340856009 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a690182320 con 0x55a690290b50 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.105+0000 7f9367400640 10 client.15459 handle_command_reply: tid=0 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.105+0000 7f938b254740 1 client.15459 shutdown 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 -- 172.21.15.28:0/2340856009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 msgr2=0x55a69021f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a69021f3a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f936c036040 tx=0x7f936c000bc0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 -- 172.21.15.28:0/2340856009 shutdown_connections 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a690290b50 0x55a690292f40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a690219b90 0x55a69021f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69020f2d0 0x55a69021f3a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f9367400640 0 client.15459 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 --2- 172.21.15.28:0/2340856009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a69020e9b0 0x55a690222010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 -- 172.21.15.28:0/2340856009 >> 172.21.15.28:0/2340856009 conn(0x55a69020aec0 msgr2=0x55a690216b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 -- 172.21.15.28:0/2340856009 shutdown_connections 2024-04-26T01:44:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 1 -- 172.21.15.28:0/2340856009 wait complete. 2024-04-26T01:44:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.106+0000 7f938b254740 20 client.15459 trim_cache size 0 max 16384 2024-04-26T01:44:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:44:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:44:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:44:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 -- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f93680a3d40 msgr2=0x7f93680a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f93680a3d40 0x7f93680a6200 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55a69020ff50 tx=0x7f93740d45f0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 -- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9380101fe0 msgr2=0x7f9380074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9380101fe0 0x7f9380074250 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f937400ea60 tx=0x7f937400ef20 comp rx=0 tx=0).stop 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 -- 172.21.15.28:0/3875799163 shutdown_connections 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f93680a3d40 0x7f93680a6200 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9380074cd0 0x7f93801a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93801070b0 0x7f9380074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 --2- 172.21.15.28:0/3875799163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9380101fe0 0x7f9380074250 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 -- 172.21.15.28:0/3875799163 >> 172.21.15.28:0/3875799163 conn(0x7f93800fd8a0 msgr2=0x7f93800fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 -- 172.21.15.28:0/3875799163 shutdown_connections 2024-04-26T01:44:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:33.107+0000 7f9386e00640 1 -- 172.21.15.28:0/3875799163 wait complete. 2024-04-26T01:44:33.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:33 smithi028 ceph-mon[28247]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 29 MiB/s wr, 1.35k op/s 2024-04-26T01:44:33.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:33 smithi169 ceph-mon[32851]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 29 MiB/s wr, 1.35k op/s 2024-04-26T01:44:33.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:33 smithi053 ceph-mon[32593]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 29 MiB/s wr, 1.35k op/s 2024-04-26T01:44:35.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:35 smithi028 ceph-mon[28247]: pgmap v111: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 29 MiB/s wr, 1.38k op/s 2024-04-26T01:44:35.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:35 smithi169 ceph-mon[32851]: pgmap v111: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 29 MiB/s wr, 1.38k op/s 2024-04-26T01:44:35.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:35 smithi053 ceph-mon[32593]: pgmap v111: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 29 MiB/s wr, 1.38k op/s 2024-04-26T01:44:36.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 160.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:37 smithi028 ceph-mon[28247]: pgmap v112: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 1.24k op/s 2024-04-26T01:44:37.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:37 smithi169 ceph-mon[32851]: pgmap v112: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 1.24k op/s 2024-04-26T01:44:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:37 smithi053 ceph-mon[32593]: pgmap v112: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 1.24k op/s 2024-04-26T01:44:39.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:39 smithi028 ceph-mon[28247]: pgmap v113: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-04-26T01:44:39.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:39 smithi169 ceph-mon[32851]: pgmap v113: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-04-26T01:44:39.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:39 smithi053 ceph-mon[32593]: pgmap v113: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-04-26T01:44:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 165.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:41.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:41 smithi028 ceph-mon[28247]: pgmap v114: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-04-26T01:44:41.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:41 smithi169 ceph-mon[32851]: pgmap v114: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-04-26T01:44:41.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:41 smithi053 ceph-mon[32593]: pgmap v114: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 MiB/s wr, 1.19k op/s 2024-04-26T01:44:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:43 smithi028 ceph-mon[28247]: pgmap v115: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 202 KiB/s wr, 33 op/s 2024-04-26T01:44:43.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:43 smithi169 ceph-mon[32851]: pgmap v115: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 202 KiB/s wr, 33 op/s 2024-04-26T01:44:43.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:43 smithi053 ceph-mon[32593]: pgmap v115: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 202 KiB/s wr, 33 op/s 2024-04-26T01:44:45.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:45 smithi028 ceph-mon[28247]: pgmap v116: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.8 MiB/s wr, 335 op/s 2024-04-26T01:44:45.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:45 smithi169 ceph-mon[32851]: pgmap v116: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.8 MiB/s wr, 335 op/s 2024-04-26T01:44:45.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:45 smithi053 ceph-mon[32593]: pgmap v116: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.8 MiB/s wr, 335 op/s 2024-04-26T01:44:46.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 170.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1092639914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee0076480 msgr2=0x7f9ee0076860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1092639914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee0076480 0x7f9ee0076860 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009930 tx=0x7f9ed002a420 comp rx=0 tx=0).stop 2024-04-26T01:44:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1092639914 shutdown_connections 2024-04-26T01:44:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1092639914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee0112c90 0x7f9ee0115080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1092639914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee0076da0 0x7f9ee0112750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1092639914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee0076480 0x7f9ee0076860 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1092639914 >> 172.21.15.28:0/1092639914 conn(0x7f9ee00ffaa0 msgr2=0x7f9ee0101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1092639914 shutdown_connections 2024-04-26T01:44:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.259+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1092639914 wait complete. 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.260+0000 7f9ee4e00640 1 Processor -- start 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.260+0000 7f9ee4e00640 1 -- start start 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9ee4e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee0076480 0x7f9ee01a4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9ee4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee0076da0 0x7f9ee01a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9ee4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee0112c90 0x7f9ee019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9ee4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee0118600 con 0x7f9ee0112c90 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9ee4e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee0118480 con 0x7f9ee0076da0 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9ee4e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee0118780 con 0x7f9ee0076480 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9edfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee0112c90 0x7f9ee019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9edfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee0112c90 0x7f9ee019ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34820/0 (socket says 172.21.15.28:34820) 2024-04-26T01:44:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9edfe00640 1 -- 172.21.15.28:0/1610425386 learned_addr learned my addr 172.21.15.28:0/1610425386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9edf400640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee0076480 0x7f9ee01a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.261+0000 7f9edea00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee0076da0 0x7f9ee01a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9edfe00640 1 -- 172.21.15.28:0/1610425386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee0076480 msgr2=0x7f9ee01a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9edfe00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee0076480 0x7f9ee01a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9edfe00640 1 -- 172.21.15.28:0/1610425386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee0076da0 msgr2=0x7f9ee01a4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9edfe00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee0076da0 0x7f9ee01a4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9edfe00640 1 -- 172.21.15.28:0/1610425386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee019f1f0 con 0x7f9ee0112c90 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9edfe00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee0112c90 0x7f9ee019ea70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800eb40 tx=0x7f9ed800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9ed7e00640 1 -- 172.21.15.28:0/1610425386 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8004b20 con 0x7f9ee0112c90 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9ed7e00640 1 -- 172.21.15.28:0/1610425386 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed8004cc0 con 0x7f9ee0112c90 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1610425386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee019f4e0 con 0x7f9ee0112c90 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.262+0000 7f9ed7e00640 1 -- 172.21.15.28:0/1610425386 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8010720 con 0x7f9ee0112c90 2024-04-26T01:44:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.263+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1610425386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee01ab730 con 0x7f9ee0112c90 2024-04-26T01:44:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.266+0000 7f9ee9234740 1 Processor -- start 2024-04-26T01:44:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.266+0000 7f9ed7e00640 1 -- 172.21.15.28:0/1610425386 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9ed8005560 con 0x7f9ee0112c90 2024-04-26T01:44:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.266+0000 7f9ee9234740 1 -- start start 2024-04-26T01:44:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc119b0 0x5614bbc11d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc122d0 0x5614bbc1c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc1cb90 0x5614bbc1ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9ee9234740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614bbb77900 con 0x5614bbc122d0 2024-04-26T01:44:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9ee9234740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5614bbb1def0 con 0x5614bbc119b0 2024-04-26T01:44:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9ee9234740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5614bbb738c0 con 0x5614bbc1cb90 2024-04-26T01:44:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9edf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc122d0 0x5614bbc1c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9edf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc122d0 0x5614bbc1c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34834/0 (socket says 172.21.15.28:34834) 2024-04-26T01:44:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9edea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc1cb90 0x5614bbc1ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9edf400640 1 -- 172.21.15.28:0/577392151 learned_addr learned my addr 172.21.15.28:0/577392151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.267+0000 7f9edea00640 1 --2- 172.21.15.28:0/577392151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc119b0 0x5614bbc11d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.268+0000 7f9edf400640 1 -- 172.21.15.28:0/577392151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc1cb90 msgr2=0x5614bbc1ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.268+0000 7f9edf400640 1 --2- 172.21.15.28:0/577392151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc1cb90 0x5614bbc1ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.268+0000 7f9edf400640 1 -- 172.21.15.28:0/577392151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc119b0 msgr2=0x5614bbc11d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.268+0000 7f9edf400640 1 --2- 172.21.15.28:0/577392151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc119b0 0x5614bbc11d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.268+0000 7f9edf400640 1 -- 172.21.15.28:0/577392151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614bb88a210 con 0x5614bbc122d0 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.268+0000 7f9ed7e00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9ec40a3d10 0x7f9ec40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.268+0000 7f9edf400640 1 --2- 172.21.15.28:0/577392151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc122d0 0x5614bbc1c650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009900 tx=0x7f9ed0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3c031c14f1dfec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.268+0000 7f9ed7e00640 1 -- 172.21.15.28:0/1610425386 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f9ed80d0f10 con 0x7f9ee0112c90 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.269+0000 7f9edea00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9ec40a3d10 0x7f9ec40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.269+0000 7f9ed4c00640 1 -- 172.21.15.28:0/577392151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0040070 con 0x5614bbc122d0 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.269+0000 7f9ed4c00640 1 -- 172.21.15.28:0/577392151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed0004030 con 0x5614bbc122d0 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.269+0000 7f9ed4c00640 1 -- 172.21.15.28:0/577392151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0035040 con 0x5614bbc122d0 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 -- 172.21.15.28:0/577392151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc122d0 msgr2=0x5614bbc1c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 --2- 172.21.15.28:0/577392151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc122d0 0x5614bbc1c650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009900 tx=0x7f9ed0005c00 comp rx=0 tx=0).stop 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9edea00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9ec40a3d10 0x7f9ec40a61d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009fd0 tx=0x7f9ecc005f90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 -- 172.21.15.28:0/577392151 shutdown_connections 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 --2- 172.21.15.28:0/577392151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc1cb90 0x5614bbc1ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 --2- 172.21.15.28:0/577392151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc122d0 0x5614bbc1c650 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 --2- 172.21.15.28:0/577392151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc119b0 0x5614bbc11d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 -- 172.21.15.28:0/577392151 >> 172.21.15.28:0/577392151 conn(0x5614bbc0dec0 msgr2=0x5614bbc0e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 -- 172.21.15.28:0/577392151 shutdown_connections 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 -- 172.21.15.28:0/577392151 wait complete. 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.270+0000 7f9ee9234740 1 -- start start 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9ee9234740 1 Processor -- start 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc19a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc19f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc1a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9ee9234740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614bbc1f760 con 0x5614bbc119b0 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9ee9234740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5614bbb31310 con 0x5614bbc1cb90 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9ee9234740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5614bbc1f8e0 con 0x5614bbc122d0 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9ee9234740 10 client.0 _ll_register_callbacks cb 0x5614bbaffcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9edf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc19a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9edf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc19a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34846/0 (socket says 172.21.15.28:34846) 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9edf400640 1 -- 172.21.15.28:0/2147485846 learned_addr learned my addr 172.21.15.28:0/2147485846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9edea00640 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc19f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.271+0000 7f9edfe00640 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc1a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.272+0000 7f9edf400640 1 -- 172.21.15.28:0/2147485846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 msgr2=0x5614bbc19f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.272+0000 7f9edf400640 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc19f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.272+0000 7f9edf400640 1 -- 172.21.15.28:0/2147485846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 msgr2=0x5614bbc1a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.272+0000 7f9edf400640 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc1a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.272+0000 7f9edf400640 1 -- 172.21.15.28:0/2147485846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614bb956620 con 0x5614bbc119b0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.272+0000 7f9edf400640 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc19a30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0005100 tx=0x7f9ed002ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3b6c3ea593f510 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.272+0000 7f9ec9600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.272+0000 7f9ec9600640 1 -- 172.21.15.28:0/2147485846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0040070 con 0x5614bbc119b0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.273+0000 7f9ec9600640 1 -- 172.21.15.28:0/2147485846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f9ed00043f0 con 0x5614bbc119b0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.273+0000 7f9ee9234740 1 -- 172.21.15.28:0/2147485846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ed0004030 con 0x5614bbc119b0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.273+0000 7f9ec9600640 1 -- 172.21.15.28:0/2147485846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9ed0035050 con 0x5614bbc119b0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.273+0000 7f9ee9234740 10 client.15471 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.273+0000 7f9ee9234740 1 -- 172.21.15.28:0/2147485846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614bbc2b950 con 0x5614bbc119b0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.273+0000 7f9ec9600640 1 -- 172.21.15.28:0/2147485846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed0004030 con 0x5614bbc119b0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.273+0000 7f9ec9600640 1 -- 172.21.15.28:0/2147485846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f9ed0004890 con 0x5614bbc119b0 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.274+0000 7f9ee9234740 10 client.15471 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.274+0000 7f9ee9234740 10 client.15471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.274+0000 7f9ee9234740 20 client.15471 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:46.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.274+0000 7f9ee9234740 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5614bbc2cca0 0x5614bbc2f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.274+0000 7f9ee9234740 4 client.15471 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:46.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.274+0000 7f9ee9234740 1 -- 172.21.15.28:0/2147485846 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614bbb8b8b0 con 0x5614bbc2cca0 2024-04-26T01:44:46.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.274+0000 7f9edea00640 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5614bbc2cca0 0x5614bbc2f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.275+0000 7f9edea00640 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5614bbc2cca0 0x5614bbc2f090 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da68a5d7af20e8f6 server_cookie=564f3c93fee89de9 in_seq=0 out_seq=0 2024-04-26T01:44:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.275+0000 7f9ec9600640 10 client.15471 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.275+0000 7f9ec9600640 1 -- 172.21.15.28:0/2147485846 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614bbb8b8b0 con 0x5614bbc2cca0 2024-04-26T01:44:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ec9600640 10 client.15471 handle_command_reply: tid=0 2024-04-26T01:44:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 client.15471 shutdown 2024-04-26T01:44:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 -- 172.21.15.28:0/2147485846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 msgr2=0x5614bbc19a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc19a30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0005100 tx=0x7f9ed002ab70 comp rx=0 tx=0).stop 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 -- 172.21.15.28:0/2147485846 shutdown_connections 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5614bbc2cca0 0x5614bbc2f090 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc1a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc19f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 --2- 172.21.15.28:0/2147485846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc19a30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ec9600640 0 client.15471 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 -- 172.21.15.28:0/2147485846 >> 172.21.15.28:0/2147485846 conn(0x5614bbc0dec0 msgr2=0x5614bbc13c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 -- 172.21.15.28:0/2147485846 shutdown_connections 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 1 -- 172.21.15.28:0/2147485846 wait complete. 2024-04-26T01:44:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.276+0000 7f9ee9234740 20 client.15471 trim_cache size 0 max 16384 2024-04-26T01:44:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.293+0000 7f9ee9234740 1 Processor -- start 2024-04-26T01:44:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.293+0000 7f9ee9234740 1 -- start start 2024-04-26T01:44:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.293+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc28650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.293+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc34750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.293+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc34c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.293+0000 7f9ee9234740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01b6800 con 0x5614bbc119b0 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.293+0000 7f9ee9234740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01b40d0 con 0x5614bbc1cb90 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.293+0000 7f9ee9234740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01b1f20 con 0x5614bbc122d0 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc34750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc28650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc34c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc34750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51298/0 (socket says 172.21.15.28:51298) 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc34c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50448/0 (socket says 172.21.15.28:50448) 2024-04-26T01:44:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edea00640 1 -- 172.21.15.28:0/91207023 learned_addr learned my addr 172.21.15.28:0/91207023 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edea00640 1 -- 172.21.15.28:0/91207023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 msgr2=0x5614bbc34c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edea00640 1 --2- 172.21.15.28:0/91207023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc34c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edea00640 1 -- 172.21.15.28:0/91207023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 msgr2=0x5614bbc28650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edea00640 1 --2- 172.21.15.28:0/91207023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc28650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edea00640 1 -- 172.21.15.28:0/91207023 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614bbc14820 con 0x5614bbc122d0 2024-04-26T01:44:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.294+0000 7f9edf400640 1 --2- 172.21.15.28:0/91207023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc28650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.295+0000 7f9edea00640 1 --2- 172.21.15.28:0/91207023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc34750 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0114c10 tx=0x7f9ecc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be005d221bc6638f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.295+0000 7f9ecbe00640 1 -- 172.21.15.28:0/91207023 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc011be0 con 0x5614bbc122d0 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.295+0000 7f9ecbe00640 1 -- 172.21.15.28:0/91207023 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ecc011d80 con 0x5614bbc122d0 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.295+0000 7f9ecbe00640 1 -- 172.21.15.28:0/91207023 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc02d7d0 con 0x5614bbc122d0 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 -- 172.21.15.28:0/91207023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 msgr2=0x5614bbc34750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 --2- 172.21.15.28:0/91207023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc34750 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0114c10 tx=0x7f9ecc0046c0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 -- 172.21.15.28:0/91207023 shutdown_connections 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 --2- 172.21.15.28:0/91207023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc34c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 --2- 172.21.15.28:0/91207023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc34750 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 --2- 172.21.15.28:0/91207023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc28650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 -- 172.21.15.28:0/91207023 >> 172.21.15.28:0/91207023 conn(0x5614bbc0dec0 msgr2=0x5614bbc0e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 -- 172.21.15.28:0/91207023 shutdown_connections 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 -- 172.21.15.28:0/91207023 wait complete. 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 -- start start 2024-04-26T01:44:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.296+0000 7f9ee9234740 1 Processor -- start 2024-04-26T01:44:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc21b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc22070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9ee9234740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc225b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9ee9234740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01b2590 con 0x5614bbc119b0 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9ee9234740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01b3a00 con 0x5614bbc1cb90 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9ee9234740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01b2c60 con 0x5614bbc122d0 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9ee9234740 10 client.0 _ll_register_callbacks cb 0x7f9ed0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9edf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc21b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9edf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc21b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34860/0 (socket says 172.21.15.28:34860) 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9edf400640 1 -- 172.21.15.28:0/381168938 learned_addr learned my addr 172.21.15.28:0/381168938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9edfe00640 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc225b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.297+0000 7f9edea00640 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc22070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9edea00640 1 -- 172.21.15.28:0/381168938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 msgr2=0x5614bbc225b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9edea00640 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc225b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9edea00640 1 -- 172.21.15.28:0/381168938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 msgr2=0x5614bbc21b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9edea00640 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc21b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9edea00640 1 -- 172.21.15.28:0/381168938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614bbc91210 con 0x5614bbc122d0 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9edf400640 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc21b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9edea00640 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc22070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x5614bbc122a0 tx=0x7f9ecc011f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce1cd033d2e22ef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9ec9600640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.298+0000 7f9ec9600640 1 -- 172.21.15.28:0/381168938 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc011be0 con 0x5614bbc122d0 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.299+0000 7f9ee9234740 1 -- 172.21.15.28:0/381168938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ecc011d80 con 0x5614bbc122d0 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.299+0000 7f9ec9600640 1 -- 172.21.15.28:0/381168938 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f9ecc030120 con 0x5614bbc122d0 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.299+0000 7f9ec9600640 1 -- 172.21.15.28:0/381168938 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9ecc030320 con 0x5614bbc122d0 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.299+0000 7f9ee9234740 10 client.25648 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.299+0000 7f9ee9234740 1 -- 172.21.15.28:0/381168938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614bbc92500 con 0x5614bbc122d0 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.299+0000 7f9ec9600640 1 -- 172.21.15.28:0/381168938 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ecc011d80 con 0x5614bbc122d0 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.300+0000 7f9ec9600640 1 -- 172.21.15.28:0/381168938 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f9ecc004960 con 0x5614bbc122d0 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.300+0000 7f9ee9234740 10 client.25648 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.300+0000 7f9ee9234740 10 client.25648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.300+0000 7f9ee9234740 20 client.25648 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.300+0000 7f9ee9234740 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5614bbc92c00 0x5614bbc95020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.300+0000 7f9ee9234740 4 client.25648 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.300+0000 7f9ee9234740 1 -- 172.21.15.28:0/381168938 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614bbb85320 con 0x5614bbc92c00 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.300+0000 7f9edf400640 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5614bbc92c00 0x5614bbc95020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.301+0000 7f9edf400640 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5614bbc92c00 0x5614bbc95020 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d80996da5afad8c0 server_cookie=36506a6ad9b43ef1 in_seq=0 out_seq=0 2024-04-26T01:44:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.301+0000 7f9ec9600640 10 client.25648 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.315+0000 7f9ec9600640 1 -- 172.21.15.28:0/381168938 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614bbb85320 con 0x5614bbc92c00 2024-04-26T01:44:46.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.315+0000 7f9ec9600640 10 client.25648 handle_command_reply: tid=0 2024-04-26T01:44:46.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.315+0000 7f9ee9234740 1 client.25648 shutdown 2024-04-26T01:44:46.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.315+0000 7f9ee9234740 1 -- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 msgr2=0x5614bbc22070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.315+0000 7f9ee9234740 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc22070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x5614bbc122a0 tx=0x7f9ecc011f30 comp rx=0 tx=0).stop 2024-04-26T01:44:46.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 1 -- 172.21.15.28:0/381168938 shutdown_connections 2024-04-26T01:44:46.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5614bbc92c00 0x5614bbc95020 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614bbc1cb90 0x5614bbc225b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5614bbc122d0 0x5614bbc22070 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 1 --2- 172.21.15.28:0/381168938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614bbc119b0 0x5614bbc21b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ec9600640 0 client.25648 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:46.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 1 -- 172.21.15.28:0/381168938 >> 172.21.15.28:0/381168938 conn(0x5614bbc0dec0 msgr2=0x5614bbc18300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:46.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 1 -- 172.21.15.28:0/381168938 shutdown_connections 2024-04-26T01:44:46.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 1 -- 172.21.15.28:0/381168938 wait complete. 2024-04-26T01:44:46.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.316+0000 7f9ee9234740 20 client.25648 trim_cache size 0 max 16384 2024-04-26T01:44:46.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:44:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:44:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:44:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:44:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9ec40a3d10 msgr2=0x7f9ec40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9ec40a3d10 0x7f9ec40a61d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009fd0 tx=0x7f9ecc005f90 comp rx=0 tx=0).stop 2024-04-26T01:44:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee0112c90 msgr2=0x7f9ee019ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee0112c90 0x7f9ee019ea70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800eb40 tx=0x7f9ed800c2d0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1610425386 shutdown_connections 2024-04-26T01:44:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f9ec40a3d10 0x7f9ec40a61d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee0112c90 0x7f9ee019ea70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee0076da0 0x7f9ee01a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 --2- 172.21.15.28:0/1610425386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee0076480 0x7f9ee01a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1610425386 >> 172.21.15.28:0/1610425386 conn(0x7f9ee00ffaa0 msgr2=0x7f9ee0101e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1610425386 shutdown_connections 2024-04-26T01:44:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:46.317+0000 7f9ee4e00640 1 -- 172.21.15.28:0/1610425386 wait complete. 2024-04-26T01:44:47.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:47 smithi169 ceph-mon[32851]: pgmap v117: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 MiB/s wr, 304 op/s 2024-04-26T01:44:47.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:47 smithi053 ceph-mon[32593]: pgmap v117: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 MiB/s wr, 304 op/s 2024-04-26T01:44:48.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:47 smithi028 ceph-mon[28247]: pgmap v117: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 MiB/s wr, 304 op/s 2024-04-26T01:44:49.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:49 smithi169 ceph-mon[32851]: pgmap v118: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 MiB/s wr, 304 op/s 2024-04-26T01:44:49.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:49 smithi053 ceph-mon[32593]: pgmap v118: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 MiB/s wr, 304 op/s 2024-04-26T01:44:50.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:49 smithi028 ceph-mon[28247]: pgmap v118: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 MiB/s wr, 304 op/s 2024-04-26T01:44:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 175.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:51.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:51 smithi169 ceph-mon[32851]: pgmap v119: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.6 MiB/s wr, 310 op/s 2024-04-26T01:44:51.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:51 smithi053 ceph-mon[32593]: pgmap v119: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.6 MiB/s wr, 310 op/s 2024-04-26T01:44:52.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:51 smithi028 ceph-mon[28247]: pgmap v119: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.6 MiB/s wr, 310 op/s 2024-04-26T01:44:53.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:53 smithi169 ceph-mon[32851]: pgmap v120: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.6 MiB/s wr, 307 op/s 2024-04-26T01:44:53.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:53 smithi053 ceph-mon[32593]: pgmap v120: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.6 MiB/s wr, 307 op/s 2024-04-26T01:44:54.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:53 smithi028 ceph-mon[28247]: pgmap v120: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.6 MiB/s wr, 307 op/s 2024-04-26T01:44:55.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:55 smithi169 ceph-mon[32851]: pgmap v121: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-04-26T01:44:55.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:55 smithi053 ceph-mon[32593]: pgmap v121: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-04-26T01:44:56.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:55 smithi028 ceph-mon[28247]: pgmap v121: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-04-26T01:44:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 180.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:44:57.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:57 smithi169 ceph-mon[32851]: pgmap v122: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:44:57.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:57 smithi053 ceph-mon[32593]: pgmap v122: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:44:58.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:57 smithi028 ceph-mon[28247]: pgmap v122: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.465+0000 7fd7a9e00640 1 -- 172.21.15.28:0/1955262648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4103b00 msgr2=0x7fd7a410bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.465+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/1955262648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4103b00 0x7fd7a410bfe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009860 tx=0x7fd78c02a460 comp rx=0 tx=0).stop 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.466+0000 7fd7a9e00640 1 -- 172.21.15.28:0/1955262648 shutdown_connections 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.466+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/1955262648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7a410c520 0x7fd7a410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.466+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/1955262648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4103b00 0x7fd7a410bfe0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.466+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/1955262648 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7a41031e0 0x7fd7a41035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.466+0000 7fd7a9e00640 1 -- 172.21.15.28:0/1955262648 >> 172.21.15.28:0/1955262648 conn(0x7fd7a4077fd0 msgr2=0x7fd7a40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.466+0000 7fd7a9e00640 1 -- 172.21.15.28:0/1955262648 shutdown_connections 2024-04-26T01:44:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.466+0000 7fd7a9e00640 1 -- 172.21.15.28:0/1955262648 wait complete. 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.467+0000 7fd7a9e00640 1 Processor -- start 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.467+0000 7fd7a9e00640 1 -- start start 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.467+0000 7fd7a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41031e0 0x7fd7a4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.467+0000 7fd7a9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7a4103b00 0x7fd7a4198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.467+0000 7fd7a9e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7a410c520 0x7fd7a419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.467+0000 7fd7a9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4112200 con 0x7fd7a41031e0 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.467+0000 7fd7a9e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4112080 con 0x7fd7a4103b00 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.467+0000 7fd7a9e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4112380 con 0x7fd7a410c520 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41031e0 0x7fd7a4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7a410c520 0x7fd7a419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41031e0 0x7fd7a4198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46920/0 (socket says 172.21.15.28:46920) 2024-04-26T01:44:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a2a00640 1 -- 172.21.15.28:0/2563148949 learned_addr learned my addr 172.21.15.28:0/2563148949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd79be00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7a4103b00 0x7fd7a4198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a3400640 1 -- 172.21.15.28:0/2563148949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7a4103b00 msgr2=0x7fd7a4198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7a4103b00 0x7fd7a4198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a3400640 1 -- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41031e0 msgr2=0x7fd7a4198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41031e0 0x7fd7a4198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a3400640 1 -- 172.21.15.28:0/2563148949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a419d730 con 0x7fd7a410c520 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.468+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41031e0 0x7fd7a4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.469+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7a410c520 0x7fd7a419d000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00efc0 tx=0x7fd79c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.469+0000 7fd7a8a00640 1 -- 172.21.15.28:0/2563148949 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c003970 con 0x7fd7a410c520 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.469+0000 7fd7a8a00640 1 -- 172.21.15.28:0/2563148949 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd79c004020 con 0x7fd7a410c520 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.469+0000 7fd7a9e00640 1 -- 172.21.15.28:0/2563148949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a419da20 con 0x7fd7a410c520 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.469+0000 7fd7a8a00640 1 -- 172.21.15.28:0/2563148949 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c00b940 con 0x7fd7a410c520 2024-04-26T01:44:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.469+0000 7fd7a9e00640 1 -- 172.21.15.28:0/2563148949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a41a52b0 con 0x7fd7a410c520 2024-04-26T01:44:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.472+0000 7fd7acc0f740 1 Processor -- start 2024-04-26T01:44:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7a8a00640 1 -- 172.21.15.28:0/2563148949 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd79c01d020 con 0x7fd7a410c520 2024-04-26T01:44:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7acc0f740 1 -- start start 2024-04-26T01:44:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 0x55b3ce410b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce422200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b3ce376900 con 0x55b3ce41fe10 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b3ce31cef0 con 0x55b3ce410740 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55b3ce3728c0 con 0x55b3ce411060 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7a2a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd79be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 0x55b3ce410b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7a2a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:43162/0 (socket says 172.21.15.28:43162) 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.473+0000 7fd7a2a00640 1 -- 172.21.15.28:0/973777995 learned_addr learned my addr 172.21.15.28:0/973777995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd79be00640 1 --2- 172.21.15.28:0/973777995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce422200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd7a2a00640 1 -- 172.21.15.28:0/973777995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 msgr2=0x55b3ce410b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/973777995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 0x55b3ce410b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd7a2a00640 1 -- 172.21.15.28:0/973777995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 msgr2=0x55b3ce422200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/973777995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce422200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd7a2a00640 1 -- 172.21.15.28:0/973777995 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ce089210 con 0x55b3ce411060 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/973777995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41f8d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd794011630 tx=0x7fd794011af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8064bddaaa47ffe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd7a8a00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd7880a3dc0 0x7fd7880a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd793e00640 1 -- 172.21.15.28:0/973777995 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79401e070 con 0x55b3ce411060 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd793e00640 1 -- 172.21.15.28:0/973777995 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd79400fba0 con 0x55b3ce411060 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.474+0000 7fd793e00640 1 -- 172.21.15.28:0/973777995 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794019c80 con 0x55b3ce411060 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd7a8a00640 1 -- 172.21.15.28:0/2563148949 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fd79c0d0650 con 0x7fd7a410c520 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd79be00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd7880a3dc0 0x7fd7880a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd7acc0f740 1 -- 172.21.15.28:0/973777995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 msgr2=0x55b3ce41f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/973777995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41f8d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd794011630 tx=0x7fd794011af0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd7acc0f740 1 -- 172.21.15.28:0/973777995 shutdown_connections 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/973777995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce422200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/973777995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41f8d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/973777995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 0x55b3ce410b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd7acc0f740 1 -- 172.21.15.28:0/973777995 >> 172.21.15.28:0/973777995 conn(0x55b3ce40cd30 msgr2=0x55b3ce40d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.475+0000 7fd79be00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd7880a3dc0 0x7fd7880a6280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7a410e2c0 tx=0x7fd78c005c90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.476+0000 7fd7acc0f740 1 -- 172.21.15.28:0/973777995 shutdown_connections 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.476+0000 7fd7acc0f740 1 -- 172.21.15.28:0/973777995 wait complete. 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.476+0000 7fd7acc0f740 1 -- start start 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.476+0000 7fd7acc0f740 1 Processor -- start 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 0x55b3ce41cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce41d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b3ce422a40 con 0x55b3ce41fe10 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b3ce330310 con 0x55b3ce410740 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55b3ce422bc0 con 0x55b3ce411060 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7acc0f740 10 client.0 _ll_register_callbacks cb 0x55b3ce2fecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce41d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce41d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46932/0 (socket says 172.21.15.28:46932) 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7a3400640 1 -- 172.21.15.28:0/267955553 learned_addr learned my addr 172.21.15.28:0/267955553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd79be00640 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.477+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 0x55b3ce41cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7a3400640 1 -- 172.21.15.28:0/267955553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 msgr2=0x55b3ce41d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7a3400640 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7a3400640 1 -- 172.21.15.28:0/267955553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 msgr2=0x55b3ce41cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7a3400640 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 0x55b3ce41cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7a3400640 1 -- 172.21.15.28:0/267955553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ce155620 con 0x55b3ce41fe10 2024-04-26T01:44:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7a3400640 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce41d830 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd79c09c2c0 tx=0x7fd79c09cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac278807fe15637 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd790c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:44:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd790c00640 1 -- 172.21.15.28:0/267955553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0a5230 con 0x55b3ce41fe10 2024-04-26T01:44:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd790c00640 1 -- 172.21.15.28:0/267955553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fd79c098c30 con 0x55b3ce41fe10 2024-04-26T01:44:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7acc0f740 1 -- 172.21.15.28:0/267955553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd79400fba0 con 0x55b3ce41fe10 2024-04-26T01:44:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd790c00640 1 -- 172.21.15.28:0/267955553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd79c0a4da0 con 0x55b3ce41fe10 2024-04-26T01:44:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7acc0f740 10 client.15489 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.478+0000 7fd7acc0f740 1 -- 172.21.15.28:0/267955553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b3ce42ecc0 con 0x55b3ce41fe10 2024-04-26T01:44:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd790c00640 1 -- 172.21.15.28:0/267955553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd79400fba0 con 0x55b3ce41fe10 2024-04-26T01:44:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd790c00640 1 -- 172.21.15.28:0/267955553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fd79c00b040 con 0x55b3ce41fe10 2024-04-26T01:44:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd7acc0f740 10 client.15489 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd7acc0f740 10 client.15489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd7acc0f740 20 client.15489 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b3ce430180 0x55b3ce432590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd7acc0f740 4 client.15489 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd7acc0f740 1 -- 172.21.15.28:0/267955553 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3ce38a8b0 con 0x55b3ce430180 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.479+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b3ce430180 0x55b3ce432590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.480+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b3ce430180 0x55b3ce432590 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fb44b248c921a48 server_cookie=4de905ab43437af1 in_seq=0 out_seq=0 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.480+0000 7fd790c00640 10 client.15489 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd790c00640 1 -- 172.21.15.28:0/267955553 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3ce38a8b0 con 0x55b3ce430180 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd790c00640 10 client.15489 handle_command_reply: tid=0 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 client.15489 shutdown 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 -- 172.21.15.28:0/267955553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 msgr2=0x55b3ce41d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce41d830 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd79c09c2c0 tx=0x7fd79c09cd80 comp rx=0 tx=0).stop 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 -- 172.21.15.28:0/267955553 shutdown_connections 2024-04-26T01:44:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b3ce430180 0x55b3ce432590 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce41fe10 0x55b3ce41d830 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce41d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/267955553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce410740 0x55b3ce41cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd790c00640 0 client.15489 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.481+0000 7fd7acc0f740 1 -- 172.21.15.28:0/267955553 >> 172.21.15.28:0/267955553 conn(0x55b3ce40cd30 msgr2=0x55b3ce412ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.482+0000 7fd7acc0f740 1 -- 172.21.15.28:0/267955553 shutdown_connections 2024-04-26T01:44:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.482+0000 7fd7acc0f740 1 -- 172.21.15.28:0/267955553 wait complete. 2024-04-26T01:44:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.482+0000 7fd7acc0f740 20 client.15489 trim_cache size 0 max 16384 2024-04-26T01:44:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.498+0000 7fd7acc0f740 1 Processor -- start 2024-04-26T01:44:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.498+0000 7fd7acc0f740 1 -- start start 2024-04-26T01:44:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.498+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce410740 0x55b3ce42b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.498+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce437ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.498+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 0x55b3ce4383e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.498+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a41ae220 con 0x55b3ce410740 2024-04-26T01:44:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.498+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a41abaf0 con 0x55b3ce41fe10 2024-04-26T01:44:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.498+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a41a9940 con 0x55b3ce411060 2024-04-26T01:44:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce410740 0x55b3ce42b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce410740 0x55b3ce42b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46944/0 (socket says 172.21.15.28:46944) 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 -- 172.21.15.28:0/2438777170 learned_addr learned my addr 172.21.15.28:0/2438777170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd79be00640 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce437ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 0x55b3ce4383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 -- 172.21.15.28:0/2438777170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 msgr2=0x55b3ce437ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce437ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 -- 172.21.15.28:0/2438777170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 msgr2=0x55b3ce4383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 0x55b3ce4383e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 -- 172.21.15.28:0/2438777170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ce41b300 con 0x55b3ce410740 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.499+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce410740 0x55b3ce42b900 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd794005fd0 tx=0x7fd79400f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742ba2b5e8e1ea04 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd793400640 1 -- 172.21.15.28:0/2438777170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79401e040 con 0x55b3ce410740 2024-04-26T01:44:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd793400640 1 -- 172.21.15.28:0/2438777170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd79402f070 con 0x55b3ce410740 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd793400640 1 -- 172.21.15.28:0/2438777170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79402a640 con 0x55b3ce410740 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd7acc0f740 1 -- 172.21.15.28:0/2438777170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce410740 msgr2=0x55b3ce42b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce410740 0x55b3ce42b900 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd794005fd0 tx=0x7fd79400f470 comp rx=0 tx=0).stop 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd7acc0f740 1 -- 172.21.15.28:0/2438777170 shutdown_connections 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 0x55b3ce4383e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce411060 0x55b3ce437ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/2438777170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce410740 0x55b3ce42b900 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.500+0000 7fd7acc0f740 1 -- 172.21.15.28:0/2438777170 >> 172.21.15.28:0/2438777170 conn(0x55b3ce40cd30 msgr2=0x55b3ce40d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.501+0000 7fd7acc0f740 1 -- 172.21.15.28:0/2438777170 shutdown_connections 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.501+0000 7fd7acc0f740 1 -- 172.21.15.28:0/2438777170 wait complete. 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.501+0000 7fd7acc0f740 1 -- start start 2024-04-26T01:44:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.501+0000 7fd7acc0f740 1 Processor -- start 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce410740 0x55b3ce4282c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce411060 0x55b3ce416080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7acc0f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 0x55b3ce416600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a41a9fb0 con 0x55b3ce411060 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a41ab420 con 0x55b3ce41fe10 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7acc0f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a41aa680 con 0x55b3ce410740 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7acc0f740 10 client.0 _ll_register_callbacks cb 0x7fd79c09bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd79be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce411060 0x55b3ce416080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd79be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce411060 0x55b3ce416080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46948/0 (socket says 172.21.15.28:46948) 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd79be00640 1 -- 172.21.15.28:0/22557316 learned_addr learned my addr 172.21.15.28:0/22557316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce410740 0x55b3ce4282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd7a3400640 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 0x55b3ce416600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd79be00640 1 -- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce410740 msgr2=0x55b3ce4282c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd79be00640 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce410740 0x55b3ce4282c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd79be00640 1 -- 172.21.15.28:0/22557316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 msgr2=0x55b3ce416600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.502+0000 7fd79be00640 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 0x55b3ce416600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.503+0000 7fd79be00640 1 -- 172.21.15.28:0/22557316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ce416e30 con 0x55b3ce411060 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.503+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce410740 0x55b3ce4282c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.503+0000 7fd79be00640 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce411060 0x55b3ce416080 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd78c02a980 tx=0x7fd78c005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82e88426d312b05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.503+0000 7fd790c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.503+0000 7fd790c00640 1 -- 172.21.15.28:0/22557316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c005250 con 0x55b3ce411060 2024-04-26T01:44:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.503+0000 7fd790c00640 1 -- 172.21.15.28:0/22557316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fd78c045070 con 0x55b3ce411060 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.503+0000 7fd7acc0f740 1 -- 172.21.15.28:0/22557316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd79402f070 con 0x55b3ce411060 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd790c00640 1 -- 172.21.15.28:0/22557316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd78c0053f0 con 0x55b3ce411060 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd7acc0f740 10 client.15495 fetch_fsmap learned FSMap version 29 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd7acc0f740 1 -- 172.21.15.28:0/22557316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b3ce49aff0 con 0x55b3ce411060 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd790c00640 1 -- 172.21.15.28:0/22557316 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd79402f070 con 0x55b3ce411060 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd790c00640 1 -- 172.21.15.28:0/22557316 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fd78c048040 con 0x55b3ce411060 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd7acc0f740 10 client.15495 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd7acc0f740 10 client.15495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd7acc0f740 20 client.15495 populate_metadata read hostname 'smithi028' 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b3ce49b800 0x55b3ce49dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd7acc0f740 4 client.15495 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.504+0000 7fd7acc0f740 1 -- 172.21.15.28:0/22557316 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3ce384320 con 0x55b3ce49b800 2024-04-26T01:44:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.505+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b3ce49b800 0x55b3ce49dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.505+0000 7fd7a2a00640 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b3ce49b800 0x55b3ce49dc20 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=426b0155f49e6df0 server_cookie=57029ac1fbf361ed in_seq=0 out_seq=0 2024-04-26T01:44:59.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.505+0000 7fd790c00640 10 client.15495 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.516+0000 7fd790c00640 1 -- 172.21.15.28:0/22557316 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b3ce384320 con 0x55b3ce49b800 2024-04-26T01:44:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.516+0000 7fd790c00640 10 client.15495 handle_command_reply: tid=0 2024-04-26T01:44:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.516+0000 7fd7acc0f740 1 client.15495 shutdown 2024-04-26T01:44:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 -- 172.21.15.28:0/22557316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce411060 msgr2=0x55b3ce416080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce411060 0x55b3ce416080 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd78c02a980 tx=0x7fd78c005a90 comp rx=0 tx=0).stop 2024-04-26T01:44:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 -- 172.21.15.28:0/22557316 shutdown_connections 2024-04-26T01:44:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55b3ce49b800 0x55b3ce49dc20 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b3ce41fe10 0x55b3ce416600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ce411060 0x55b3ce416080 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd790c00640 0 client.15495 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:44:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 --2- 172.21.15.28:0/22557316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b3ce410740 0x55b3ce4282c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 -- 172.21.15.28:0/22557316 >> 172.21.15.28:0/22557316 conn(0x55b3ce40cd30 msgr2=0x55b3ce41cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 -- 172.21.15.28:0/22557316 shutdown_connections 2024-04-26T01:44:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 1 -- 172.21.15.28:0/22557316 wait complete. 2024-04-26T01:44:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.517+0000 7fd7acc0f740 20 client.15495 trim_cache size 0 max 16384 2024-04-26T01:44:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:44:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:44:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:44:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:44:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 -- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd7880a3dc0 msgr2=0x7fd7880a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd7880a3dc0 0x7fd7880a6280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7a410e2c0 tx=0x7fd78c005c90 comp rx=0 tx=0).stop 2024-04-26T01:44:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 -- 172.21.15.28:0/2563148949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7a410c520 msgr2=0x7fd7a419d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7a410c520 0x7fd7a419d000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00efc0 tx=0x7fd79c00c800 comp rx=0 tx=0).stop 2024-04-26T01:44:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 -- 172.21.15.28:0/2563148949 shutdown_connections 2024-04-26T01:44:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd7880a3dc0 0x7fd7880a6280 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7a410c520 0x7fd7a419d000 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7a4103b00 0x7fd7a4198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 --2- 172.21.15.28:0/2563148949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41031e0 0x7fd7a4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 -- 172.21.15.28:0/2563148949 >> 172.21.15.28:0/2563148949 conn(0x7fd7a4077fd0 msgr2=0x7fd7a40fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 -- 172.21.15.28:0/2563148949 shutdown_connections 2024-04-26T01:44:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:44:59.518+0000 7fd7a9e00640 1 -- 172.21.15.28:0/2563148949 wait complete. 2024-04-26T01:44:59.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:44:59 smithi169 ceph-mon[32851]: pgmap v123: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:44:59.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:44:59 smithi053 ceph-mon[32593]: pgmap v123: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:44:59.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:45:00.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:44:59 smithi028 ceph-mon[28247]: pgmap v123: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:45:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:01.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 185.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:45:01.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:01 smithi169 ceph-mon[32851]: pgmap v124: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-26T01:45:01.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:01 smithi053 ceph-mon[32593]: pgmap v124: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-26T01:45:02.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:01 smithi028 ceph-mon[28247]: pgmap v124: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-26T01:45:03.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 -- 172.21.15.28:0/2096545494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc10f440 msgr2=0x7f75bc111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 --2- 172.21.15.28:0/2096545494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc10f440 0x7f75bc111830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f75b400b0a0 tx=0x7f75b400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 -- 172.21.15.28:0/2096545494 shutdown_connections 2024-04-26T01:45:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 --2- 172.21.15.28:0/2096545494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc10f440 0x7f75bc111830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 --2- 172.21.15.28:0/2096545494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75bc074cb0 0x7f75bc075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 --2- 172.21.15.28:0/2096545494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75bc0755d0 0x7f75bc10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 -- 172.21.15.28:0/2096545494 >> 172.21.15.28:0/2096545494 conn(0x7f75bc079c30 msgr2=0x7f75bc07a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 -- 172.21.15.28:0/2096545494 shutdown_connections 2024-04-26T01:45:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.153+0000 7f75c3a00700 1 -- 172.21.15.28:0/2096545494 wait complete. 2024-04-26T01:45:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.154+0000 7f75c3a00700 1 Processor -- start 2024-04-26T01:45:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.154+0000 7f75c3a00700 1 -- start start 2024-04-26T01:45:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75c3a00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75bc074cb0 0x7f75bc19b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75c3a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc0755d0 0x7f75bc19b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75c3a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75bc10f440 0x7f75bc19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75c3a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc114eb0 con 0x7f75bc0755d0 2024-04-26T01:45:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75c3a00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc114d30 con 0x7f75bc074cb0 2024-04-26T01:45:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75c3a00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc115030 con 0x7f75bc10f440 2024-04-26T01:45:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75bbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc0755d0 0x7f75bc19b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75bbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc0755d0 0x7f75bc19b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46954/0 (socket says 172.21.15.28:46954) 2024-04-26T01:45:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75bbe00700 1 -- 172.21.15.28:0/3184322263 learned_addr learned my addr 172.21.15.28:0/3184322263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75c1000700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75bc074cb0 0x7f75bc19b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.155+0000 7f75c1a00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75bc10f440 0x7f75bc19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.156+0000 7f75c1000700 1 -- 172.21.15.28:0/3184322263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75bc10f440 msgr2=0x7f75bc19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.156+0000 7f75c1000700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75bc10f440 0x7f75bc19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.156+0000 7f75c1000700 1 -- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc0755d0 msgr2=0x7f75bc19b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.156+0000 7f75c1000700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc0755d0 0x7f75bc19b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.156+0000 7f75c1000700 1 -- 172.21.15.28:0/3184322263 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75bc1a03c0 con 0x7f75bc074cb0 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.156+0000 7f75bbe00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc0755d0 0x7f75bc19b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.156+0000 7f75c1000700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75bc074cb0 0x7f75bc19b400 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e9f0 tx=0x7f75b000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.157+0000 7f75b9600700 1 -- 172.21.15.28:0/3184322263 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b000c9c0 con 0x7f75bc074cb0 2024-04-26T01:45:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.157+0000 7f75b9600700 1 -- 172.21.15.28:0/3184322263 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b000cb60 con 0x7f75bc074cb0 2024-04-26T01:45:03.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.157+0000 7f75b9600700 1 -- 172.21.15.28:0/3184322263 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0018680 con 0x7f75bc074cb0 2024-04-26T01:45:03.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.157+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75bc1a06b0 con 0x7f75bc074cb0 2024-04-26T01:45:03.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.157+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75bc1a7f30 con 0x7f75bc074cb0 2024-04-26T01:45:03.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.158+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75bc069730 con 0x7f75bc074cb0 2024-04-26T01:45:03.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.160+0000 7f75b9600700 1 -- 172.21.15.28:0/3184322263 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f75b0010430 con 0x7f75bc074cb0 2024-04-26T01:45:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.161+0000 7f75b9600700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f75a40a3e30 0x7f75a40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.161+0000 7f75b9600700 1 -- 172.21.15.28:0/3184322263 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f75b00d0960 con 0x7f75bc074cb0 2024-04-26T01:45:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.162+0000 7f75bbe00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f75a40a3e30 0x7f75a40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.162+0000 7f75bbe00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f75a40a3e30 0x7f75a40a62f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75bc1025f0 tx=0x7f75a8009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.166+0000 7f75b9600700 1 -- 172.21.15.28:0/3184322263 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75b009cfd0 con 0x7f75bc074cb0 2024-04-26T01:45:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.451+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75bc02df80 con 0x7f75bc074cb0 2024-04-26T01:45:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.453+0000 7f75b9600700 1 -- 172.21.15.28:0/3184322263 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11864 (secure 0 0 0) 0x7f75b009c8d0 con 0x7f75bc074cb0 2024-04-26T01:45:03.450 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:45:03.450 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:44:00.873936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24509,24523]},"id":1}]} 2024-04-26T01:45:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f75a40a3e30 msgr2=0x7f75a40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f75a40a3e30 0x7f75a40a62f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75bc1025f0 tx=0x7f75a8009260 comp rx=0 tx=0).stop 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75bc074cb0 msgr2=0x7f75bc19b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75bc074cb0 0x7f75bc19b400 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e9f0 tx=0x7f75b000ed00 comp rx=0 tx=0).stop 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 shutdown_connections 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f75a40a3e30 0x7f75a40a62f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75bc10f440 0x7f75bc19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75bc074cb0 0x7f75bc19b400 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 --2- 172.21.15.28:0/3184322263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75bc0755d0 0x7f75bc19b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.456+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 >> 172.21.15.28:0/3184322263 conn(0x7f75bc079c30 msgr2=0x7f75bc0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.457+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 shutdown_connections 2024-04-26T01:45:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:03.457+0000 7f75c3a00700 1 -- 172.21.15.28:0/3184322263 wait complete. 2024-04-26T01:45:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-04-26T01:45:03.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:03 smithi028 ceph-mon[28247]: pgmap v125: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-26T01:45:03.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:03 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3184322263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:03 smithi169 ceph-mon[32851]: pgmap v125: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-26T01:45:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:03 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3184322263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:03.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:03 smithi053 ceph-mon[32593]: pgmap v125: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-26T01:45:03.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:03 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3184322263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:04.513 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id a7997608 --release --await' 2024-04-26T01:45:04.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id a7997608 --release --await 2024-04-26T01:45:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:05.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:05 smithi028 ceph-mon[28247]: pgmap v126: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 43 KiB/s wr, 30 op/s 2024-04-26T01:45:05.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:05 smithi169 ceph-mon[32851]: pgmap v126: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 43 KiB/s wr, 30 op/s 2024-04-26T01:45:05.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:05 smithi053 ceph-mon[32593]: pgmap v126: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 43 KiB/s wr, 30 op/s 2024-04-26T01:45:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 190.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:45:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 -- 172.21.15.28:0/2926885121 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 msgr2=0x7f10b4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 --2- 172.21.15.28:0/2926885121 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 0x7f10b4102f10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f10a00098f0 tx=0x7f10a0009c00 comp rx=0 tx=0).stop 2024-04-26T01:45:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 -- 172.21.15.28:0/2926885121 shutdown_connections 2024-04-26T01:45:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 --2- 172.21.15.28:0/2926885121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10b4103450 0x7f10b410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 --2- 172.21.15.28:0/2926885121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10b4108a50 0x7f10b4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 --2- 172.21.15.28:0/2926885121 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 0x7f10b4102f10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 -- 172.21.15.28:0/2926885121 >> 172.21.15.28:0/2926885121 conn(0x7f10b4079ba0 msgr2=0x7f10b4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 -- 172.21.15.28:0/2926885121 shutdown_connections 2024-04-26T01:45:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.368+0000 7f10b9000700 1 -- 172.21.15.28:0/2926885121 wait complete. 2024-04-26T01:45:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.369+0000 7f10b9000700 1 Processor -- start 2024-04-26T01:45:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.370+0000 7f10b9000700 1 -- start start 2024-04-26T01:45:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.370+0000 7f10b9000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 0x7f10b419b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.370+0000 7f10b9000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10b4103450 0x7f10b419b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.370+0000 7f10b9000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10b4108a50 0x7f10b419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.370+0000 7f10b9000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10b4112c50 con 0x7f10b4108a50 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.370+0000 7f10b9000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f10b4112ad0 con 0x7f10b4103450 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.370+0000 7f10b9000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f10b4112dd0 con 0x7f10b4102a90 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 0x7f10b419b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 0x7f10b419b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:43202/0 (socket says 172.21.15.28:43202) 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10b4103450 0x7f10b419b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1e00700 1 -- 172.21.15.28:0/3418400028 learned_addr learned my addr 172.21.15.28:0/3418400028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b2800700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10b4108a50 0x7f10b419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1e00700 1 -- 172.21.15.28:0/3418400028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10b4103450 msgr2=0x7f10b419b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1e00700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10b4103450 0x7f10b419b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1e00700 1 -- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10b4108a50 msgr2=0x7f10b419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1e00700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10b4108a50 0x7f10b419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.371+0000 7f10b1e00700 1 -- 172.21.15.28:0/3418400028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b41a03f0 con 0x7f10b4102a90 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.372+0000 7f10b1e00700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 0x7f10b419b3e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f10a400e9a0 tx=0x7f10a400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.372+0000 7f10aaa00700 1 -- 172.21.15.28:0/3418400028 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a400cbd0 con 0x7f10b4102a90 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.372+0000 7f10aaa00700 1 -- 172.21.15.28:0/3418400028 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a400cd70 con 0x7f10b4102a90 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.372+0000 7f10aaa00700 1 -- 172.21.15.28:0/3418400028 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a4018680 con 0x7f10b4102a90 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.372+0000 7f10b9000700 1 -- 172.21.15.28:0/3418400028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b41a06e0 con 0x7f10b4102a90 2024-04-26T01:45:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.372+0000 7f10b9000700 1 -- 172.21.15.28:0/3418400028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b41a7f60 con 0x7f10b4102a90 2024-04-26T01:45:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.376+0000 7f10aaa00700 1 -- 172.21.15.28:0/3418400028 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f10a4010430 con 0x7f10b4102a90 2024-04-26T01:45:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.377+0000 7f10c391db80 1 Processor -- start 2024-04-26T01:45:07.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.377+0000 7f10aaa00700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f10980a3e30 0x7f10980a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.377+0000 7f10c391db80 1 -- start start 2024-04-26T01:45:07.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.377+0000 7f10aaa00700 1 -- 172.21.15.28:0/3418400028 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f10a40d0970 con 0x7f10b4102a90 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.377+0000 7f10b2800700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f10980a3e30 0x7f10980a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.377+0000 7f10c391db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582cd40 0x559e0582d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10c391db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05837f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10c391db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e0583a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10c391db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559e0577ff90 con 0x559e058384d0 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10c391db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559e05726830 con 0x559e0582cd40 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10c391db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x559e0577bf50 con 0x559e0582d660 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05837f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582cd40 0x559e0582d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05837f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:43204/0 (socket says 172.21.15.28:43204) 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1400700 1 -- 172.21.15.28:0/2938670619 learned_addr learned my addr 172.21.15.28:0/2938670619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b2800700 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e0583a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1e00700 1 -- 172.21.15.28:0/2938670619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 msgr2=0x559e05837f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1e00700 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05837f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1e00700 1 -- 172.21.15.28:0/2938670619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 msgr2=0x559e0583a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1e00700 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e0583a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.378+0000 7f10b1e00700 1 -- 172.21.15.28:0/2938670619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e0571c780 con 0x559e0582cd40 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.379+0000 7f10b2800700 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e0583a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.379+0000 7f10b1e00700 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582cd40 0x559e0582d120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f10a40ce090 tx=0x7f10a409cfe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4838b6a165dabfdd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.379+0000 7f109f400700 1 -- 172.21.15.28:0/2938670619 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40a5550 con 0x559e0582cd40 2024-04-26T01:45:07.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.379+0000 7f109f400700 1 -- 172.21.15.28:0/2938670619 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a40870f0 con 0x559e0582cd40 2024-04-26T01:45:07.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.379+0000 7f10b2800700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f10980a3e30 0x7f10980a62f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x559e0582e710 tx=0x7f10ac0099e0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:07.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.379+0000 7f109f400700 1 -- 172.21.15.28:0/2938670619 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40a5320 con 0x559e0582cd40 2024-04-26T01:45:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 -- 172.21.15.28:0/2938670619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582cd40 msgr2=0x559e0582d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582cd40 0x559e0582d120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f10a40ce090 tx=0x7f10a409cfe0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 -- 172.21.15.28:0/2938670619 shutdown_connections 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e0583a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582cd40 0x559e0582d120 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 --2- 172.21.15.28:0/2938670619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05837f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 -- 172.21.15.28:0/2938670619 >> 172.21.15.28:0/2938670619 conn(0x559e0575c050 msgr2=0x559e05829250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 -- 172.21.15.28:0/2938670619 shutdown_connections 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.380+0000 7f10c391db80 1 -- 172.21.15.28:0/2938670619 wait complete. 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.381+0000 7f10c391db80 1 -- start start 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.381+0000 7f10c391db80 1 Processor -- start 2024-04-26T01:45:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.381+0000 7f10c391db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582cd40 0x559e05837cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10c391db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05834fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10c391db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05835500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10c391db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559e0583aea0 con 0x559e0582cd40 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10c391db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559e05739960 con 0x559e058384d0 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10c391db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x559e0583b020 con 0x559e0582d660 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10c391db80 10 client.0 _ll_register_callbacks cb 0x559e05718820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582cd40 0x559e05837cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582cd40 0x559e05837cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46992/0 (socket says 172.21.15.28:46992) 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1e00700 1 -- 172.21.15.28:0/363727220 learned_addr learned my addr 172.21.15.28:0/363727220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1400700 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05834fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b2800700 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05835500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1e00700 1 -- 172.21.15.28:0/363727220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 msgr2=0x559e05834fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1e00700 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05834fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1e00700 1 -- 172.21.15.28:0/363727220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 msgr2=0x559e05835500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1e00700 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05835500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.382+0000 7f10b1e00700 1 -- 172.21.15.28:0/363727220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e056cd9d0 con 0x559e0582cd40 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.383+0000 7f10b1e00700 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582cd40 0x559e05837cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f10a409c5e0 tx=0x7f10a40a5370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317b30c857936983 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.383+0000 7f1093e00700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.383+0000 7f1093e00700 1 -- 172.21.15.28:0/363727220 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40872a0 con 0x559e0582cd40 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.383+0000 7f1093e00700 1 -- 172.21.15.28:0/363727220 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f10a4010930 con 0x559e0582cd40 2024-04-26T01:45:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.383+0000 7f10c391db80 1 -- 172.21.15.28:0/363727220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10a40870f0 con 0x559e0582cd40 2024-04-26T01:45:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.383+0000 7f1093e00700 1 -- 172.21.15.28:0/363727220 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f10a40a5550 con 0x559e0582cd40 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.383+0000 7f10c391db80 10 client.15513 fetch_fsmap learned FSMap version 29 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.383+0000 7f10c391db80 1 -- 172.21.15.28:0/363727220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e05716bf0 con 0x559e0582cd40 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.384+0000 7f1093e00700 1 -- 172.21.15.28:0/363727220 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a40870f0 con 0x559e0582cd40 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.384+0000 7f1093e00700 1 -- 172.21.15.28:0/363727220 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f10a40bbaa0 con 0x559e0582cd40 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.384+0000 7f10c391db80 10 client.15513 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.384+0000 7f10c391db80 10 client.15513 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.384+0000 7f10c391db80 20 client.15513 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.384+0000 7f10c391db80 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559e05847a90 0x559e05849e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.384+0000 7f10c391db80 4 client.15513 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.384+0000 7f10c391db80 1 -- 172.21.15.28:0/363727220 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e0578d780 con 0x559e05847a90 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.385+0000 7f10b1400700 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559e05847a90 0x559e05849e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.385+0000 7f10b1400700 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559e05847a90 0x559e05849e80 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4e0860dad86064 server_cookie=e50baf68b026eac9 in_seq=0 out_seq=0 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.385+0000 7f1093e00700 10 client.15513 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.386+0000 7f1093e00700 1 -- 172.21.15.28:0/363727220 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e0578d780 con 0x559e05847a90 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.386+0000 7f1093e00700 10 client.15513 handle_command_reply: tid=0 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.386+0000 7f10c391db80 1 client.15513 shutdown 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 -- 172.21.15.28:0/363727220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582cd40 msgr2=0x559e05837cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582cd40 0x559e05837cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f10a409c5e0 tx=0x7f10a40a5370 comp rx=0 tx=0).stop 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 -- 172.21.15.28:0/363727220 shutdown_connections 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05835500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582cd40 0x559e05837cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559e05847a90 0x559e05849e80 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 --2- 172.21.15.28:0/363727220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582d660 0x559e05834fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 -- 172.21.15.28:0/363727220 >> 172.21.15.28:0/363727220 conn(0x559e0575c050 msgr2=0x559e05828e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:07.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f1093e00700 0 client.15513 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:07.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 -- 172.21.15.28:0/363727220 shutdown_connections 2024-04-26T01:45:07.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 1 -- 172.21.15.28:0/363727220 wait complete. 2024-04-26T01:45:07.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.387+0000 7f10c391db80 20 client.15513 trim_cache size 0 max 16384 2024-04-26T01:45:07.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.409+0000 7f10c391db80 1 Processor -- start 2024-04-26T01:45:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.409+0000 7f10c391db80 1 -- start start 2024-04-26T01:45:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.409+0000 7f10c391db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e0584ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.409+0000 7f10c391db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582d660 0x559e0584d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.409+0000 7f10c391db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05828b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.409+0000 7f10c391db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10b41b1e40 con 0x559e0582d660 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.409+0000 7f10c391db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f10b41af6d0 con 0x559e058384d0 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.409+0000 7f10c391db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f10b41ad520 con 0x559e0582cd40 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e0584ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e0584ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:43216/0 (socket says 172.21.15.28:43216) 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b1e00700 1 -- 172.21.15.28:0/1033320004 learned_addr learned my addr 172.21.15.28:0/1033320004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b2800700 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05828b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b1400700 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582d660 0x559e0584d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b2800700 1 -- 172.21.15.28:0/1033320004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 msgr2=0x559e0584ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b2800700 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e0584ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b2800700 1 -- 172.21.15.28:0/1033320004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582d660 msgr2=0x559e0584d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b2800700 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582d660 0x559e0584d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b2800700 1 -- 172.21.15.28:0/1033320004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e0583af20 con 0x559e058384d0 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.410+0000 7f10b1400700 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582d660 0x559e0584d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.411+0000 7f10b2800700 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05828b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x559e058384a0 tx=0x7f10ac004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9dab633c47864ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.411+0000 7f109ea00700 1 -- 172.21.15.28:0/1033320004 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac017d90 con 0x559e058384d0 2024-04-26T01:45:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.411+0000 7f109ea00700 1 -- 172.21.15.28:0/1033320004 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10ac02a070 con 0x559e058384d0 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.411+0000 7f109ea00700 1 -- 172.21.15.28:0/1033320004 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac025650 con 0x559e058384d0 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 -- 172.21.15.28:0/1033320004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 msgr2=0x559e05828b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05828b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x559e058384a0 tx=0x7f10ac004b40 comp rx=0 tx=0).stop 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 -- 172.21.15.28:0/1033320004 shutdown_connections 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e058384d0 0x559e05828b10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e0584ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 --2- 172.21.15.28:0/1033320004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e0582d660 0x559e0584d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 -- 172.21.15.28:0/1033320004 >> 172.21.15.28:0/1033320004 conn(0x559e0575c050 msgr2=0x559e05828400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 -- 172.21.15.28:0/1033320004 shutdown_connections 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 -- 172.21.15.28:0/1033320004 wait complete. 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.412+0000 7f10c391db80 1 -- start start 2024-04-26T01:45:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10c391db80 1 Processor -- start 2024-04-26T01:45:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10c391db80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e058a8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10c391db80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582d660 0x559e058a8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10c391db80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e058a2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10c391db80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10b41adb90 con 0x559e058384d0 2024-04-26T01:45:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10c391db80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f10b41af000 con 0x559e0582d660 2024-04-26T01:45:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10c391db80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f10b41ae260 con 0x559e0582cd40 2024-04-26T01:45:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10c391db80 10 client.0 _ll_register_callbacks cb 0x559e05718820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:07.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10b2800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e058a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10b1400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582d660 0x559e058a8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.413+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e058a8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b2800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e058a2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47018/0 (socket says 172.21.15.28:47018) 2024-04-26T01:45:07.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b1400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582d660 0x559e058a8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36130/0 (socket says 172.21.15.28:36130) 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b2800700 1 -- 172.21.15.28:0/89726985 learned_addr learned my addr 172.21.15.28:0/89726985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b1e00700 1 -- 172.21.15.28:0/89726985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582d660 msgr2=0x559e058a8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b1e00700 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582d660 0x559e058a8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b1e00700 1 -- 172.21.15.28:0/89726985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 msgr2=0x559e058a2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b1e00700 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e058a2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b1e00700 1 -- 172.21.15.28:0/89726985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e0582d370 con 0x559e0582cd40 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b1400700 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582d660 0x559e058a8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f10b1e00700 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e058a8350 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009230 tx=0x7f10a4010bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3140a2284fd5a5a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.414+0000 7f1093e00700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f1093e00700 1 -- 172.21.15.28:0/89726985 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40870f0 con 0x559e0582cd40 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f1093e00700 1 -- 172.21.15.28:0/89726985 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f10a40bbaa0 con 0x559e0582cd40 2024-04-26T01:45:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 1 -- 172.21.15.28:0/89726985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10ac02a070 con 0x559e0582cd40 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f1093e00700 1 -- 172.21.15.28:0/89726985 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f10a40bbca0 con 0x559e0582cd40 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 10 client.25690 fetch_fsmap learned FSMap version 29 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 1 -- 172.21.15.28:0/89726985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e058af9e0 con 0x559e0582cd40 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f1093e00700 1 -- 172.21.15.28:0/89726985 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10ac02a070 con 0x559e0582cd40 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f1093e00700 1 -- 172.21.15.28:0/89726985 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f10a409b7a0 con 0x559e0582cd40 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 10 client.25690 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 10 client.25690 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 20 client.25690 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559e058afed0 0x559e058b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 4 client.25690 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a7997608", "release": true, "await": true}] 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.415+0000 7f10c391db80 1 -- 172.21.15.28:0/89726985 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a7997608", "release": true, "await": true}) -- 0x559e05787820 con 0x559e058afed0 2024-04-26T01:45:07.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.416+0000 7f10b1400700 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559e058afed0 0x559e058b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.416+0000 7f10b1400700 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559e058afed0 0x559e058b22f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=530e70bee5dfb001 server_cookie=d5790bcc260a386d in_seq=0 out_seq=0 2024-04-26T01:45:07.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.416+0000 7f1093e00700 10 client.25690 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:07.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f1093e00700 1 -- 172.21.15.28:0/89726985 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x559e05787820 con 0x559e058afed0 2024-04-26T01:45:07.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f1093e00700 10 client.25690 handle_command_reply: tid=0 2024-04-26T01:45:07.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 client.25690 shutdown 2024-04-26T01:45:07.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 -- 172.21.15.28:0/89726985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 msgr2=0x559e058a8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e058a8350 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009230 tx=0x7f10a4010bb0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 -- 172.21.15.28:0/89726985 shutdown_connections 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e058384d0 0x559e058a2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559e058afed0 0x559e058b22f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e0582cd40 0x559e058a8350 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 --2- 172.21.15.28:0/89726985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e0582d660 0x559e058a8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 -- 172.21.15.28:0/89726985 >> 172.21.15.28:0/89726985 conn(0x559e0575c050 msgr2=0x559e05843ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f1093e00700 0 client.25690 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 -- 172.21.15.28:0/89726985 shutdown_connections 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.423+0000 7f10c391db80 1 -- 172.21.15.28:0/89726985 wait complete. 2024-04-26T01:45:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.424+0000 7f10c391db80 20 client.25690 trim_cache size 0 max 16384 2024-04-26T01:45:07.422 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:45:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 29, 2024-04-26T01:45:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:45:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 12, 2024-04-26T01:45:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-26T01:45:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "a7997608": { 2024-04-26T01:45:07.422 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 12, 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 60.0, 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 90.0, 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:45:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-26T01:45:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:45:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:45:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:45:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:45:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:45:07.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:45:07.424 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.424+0000 7f10b9000700 1 -- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f10980a3e30 msgr2=0x7f10980a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.424+0000 7f10b9000700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f10980a3e30 0x7f10980a62f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x559e0582e710 tx=0x7f10ac0099e0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.424+0000 7f10b9000700 1 -- 172.21.15.28:0/3418400028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 msgr2=0x7f10b419b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.424+0000 7f10b9000700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 0x7f10b419b3e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f10a400e9a0 tx=0x7f10a400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.425+0000 7f10b9000700 1 -- 172.21.15.28:0/3418400028 shutdown_connections 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.425+0000 7f10b9000700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f10980a3e30 0x7f10980a62f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.425+0000 7f10b9000700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10b4108a50 0x7f10b419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.425+0000 7f10b9000700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b4102a90 0x7f10b419b3e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.425+0000 7f10b9000700 1 --2- 172.21.15.28:0/3418400028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10b4103450 0x7f10b419b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.425+0000 7f10b9000700 1 -- 172.21.15.28:0/3418400028 >> 172.21.15.28:0/3418400028 conn(0x7f10b4079ba0 msgr2=0x7f10b410df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:07.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.425+0000 7f10b9000700 1 -- 172.21.15.28:0/3418400028 shutdown_connections 2024-04-26T01:45:07.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:45:07.425+0000 7f10b9000700 1 -- 172.21.15.28:0/3418400028 wait complete. 2024-04-26T01:45:07.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:07 smithi028 ceph-mon[28247]: pgmap v127: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 KiB/s wr, 26 op/s 2024-04-26T01:45:07.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:07 smithi169 ceph-mon[32851]: pgmap v127: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 KiB/s wr, 26 op/s 2024-04-26T01:45:07.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:07 smithi053 ceph-mon[32593]: pgmap v127: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 KiB/s wr, 26 op/s 2024-04-26T01:45:08.421 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a7997608', seconds elapsed: 47.9 2024-04-26T01:45:08.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:08 smithi169 ceph-mon[32851]: pgmap v128: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 KiB/s wr, 26 op/s 2024-04-26T01:45:08.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:08 smithi053 ceph-mon[32593]: pgmap v128: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 KiB/s wr, 26 op/s 2024-04-26T01:45:09.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:08 smithi028 ceph-mon[28247]: pgmap v128: 97 pgs: 97 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 KiB/s wr, 26 op/s 2024-04-26T01:45:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 195.0 s, 2124.1 tps, lat 399.877 ms stddev 3742.867 2024-04-26T01:45:11.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:11 smithi169 ceph-mon[32851]: pgmap v129: 97 pgs: 97 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 904 KiB/s wr, 121 op/s 2024-04-26T01:45:11.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:11 smithi053 ceph-mon[32593]: pgmap v129: 97 pgs: 97 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 904 KiB/s wr, 121 op/s 2024-04-26T01:45:11.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:11 smithi028 ceph-mon[28247]: pgmap v129: 97 pgs: 97 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 904 KiB/s wr, 121 op/s 2024-04-26T01:45:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.712+0000 7f2951200640 1 -- 172.21.15.28:0/3113922101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f294c10e980 msgr2=0x7f294c10ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.712+0000 7f2951200640 1 --2- 172.21.15.28:0/3113922101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f294c10e980 0x7f294c10ed60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f293c00aa10 tx=0x7f293c02b720 comp rx=0 tx=0).stop 2024-04-26T01:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.713+0000 7f2951200640 1 -- 172.21.15.28:0/3113922101 shutdown_connections 2024-04-26T01:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.713+0000 7f2951200640 1 --2- 172.21.15.28:0/3113922101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c1098f0 0x7f294c115840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.713+0000 7f2951200640 1 --2- 172.21.15.28:0/3113922101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f294c108f50 0x7f294c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.713+0000 7f2951200640 1 --2- 172.21.15.28:0/3113922101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f294c10e980 0x7f294c10ed60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.713+0000 7f2951200640 1 -- 172.21.15.28:0/3113922101 >> 172.21.15.28:0/3113922101 conn(0x7f294c06d4b0 msgr2=0x7f294c06d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.713+0000 7f2951200640 1 -- 172.21.15.28:0/3113922101 shutdown_connections 2024-04-26T01:45:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.713+0000 7f2951200640 1 -- 172.21.15.28:0/3113922101 wait complete. 2024-04-26T01:45:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.715+0000 7f2951200640 1 Processor -- start 2024-04-26T01:45:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.716+0000 7f2951200640 1 -- start start 2024-04-26T01:45:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.717+0000 7f2951200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c108f50 0x7f294c1a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.717+0000 7f2951200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f294c1098f0 0x7f294c1a34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.717+0000 7f2951200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f294c10e980 0x7f294c1a7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.717+0000 7f2951200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f294c118870 con 0x7f294c108f50 2024-04-26T01:45:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.717+0000 7f2951200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1186f0 con 0x7f294c1098f0 2024-04-26T01:45:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.717+0000 7f2951200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1189f0 con 0x7f294c10e980 2024-04-26T01:45:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.718+0000 7f2949e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c108f50 0x7f294c1a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.718+0000 7f294a800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f294c10e980 0x7f294c1a7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.718+0000 7f2949e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c108f50 0x7f294c1a2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45588/0 (socket says 172.21.15.28:45588) 2024-04-26T01:45:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.718+0000 7f2949e00640 1 -- 172.21.15.28:0/871938864 learned_addr learned my addr 172.21.15.28:0/871938864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.718+0000 7f2949400640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f294c1098f0 0x7f294c1a34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.718+0000 7f2949e00640 1 -- 172.21.15.28:0/871938864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f294c10e980 msgr2=0x7f294c1a7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.719+0000 7f2949e00640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f294c10e980 0x7f294c1a7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.719+0000 7f2949e00640 1 -- 172.21.15.28:0/871938864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f294c1098f0 msgr2=0x7f294c1a34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.719+0000 7f2949e00640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f294c1098f0 0x7f294c1a34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.719+0000 7f2949e00640 1 -- 172.21.15.28:0/871938864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c1a7f50 con 0x7f294c108f50 2024-04-26T01:45:12.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.719+0000 7f2949e00640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c108f50 0x7f294c1a2f60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f293c02bc90 tx=0x7f293c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:12.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.720+0000 7f2942a00640 1 -- 172.21.15.28:0/871938864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c03f070 con 0x7f294c108f50 2024-04-26T01:45:12.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.720+0000 7f2942a00640 1 -- 172.21.15.28:0/871938864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f293c030e00 con 0x7f294c108f50 2024-04-26T01:45:12.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.720+0000 7f2951200640 1 -- 172.21.15.28:0/871938864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c1a81e0 con 0x7f294c108f50 2024-04-26T01:45:12.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.721+0000 7f2951200640 1 -- 172.21.15.28:0/871938864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c1afad0 con 0x7f294c108f50 2024-04-26T01:45:12.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.721+0000 7f2942a00640 1 -- 172.21.15.28:0/871938864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c036040 con 0x7f294c108f50 2024-04-26T01:45:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.728+0000 7f2954017740 1 Processor -- start 2024-04-26T01:45:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.729+0000 7f2954017740 1 -- start start 2024-04-26T01:45:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.730+0000 7f2954017740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e741cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.730+0000 7f2954017740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.730+0000 7f2954017740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e7432620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.730+0000 7f2954017740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d6e7382900 con 0x55d6e741c7e0 2024-04-26T01:45:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.730+0000 7f2954017740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d6e7328ef0 con 0x55d6e741d100 2024-04-26T01:45:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.730+0000 7f2954017740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d6e737e8c0 con 0x55d6e7430230 2024-04-26T01:45:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.730+0000 7f2942a00640 1 -- 172.21.15.28:0/871938864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f293c042070 con 0x7f294c108f50 2024-04-26T01:45:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.731+0000 7f2949e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.731+0000 7f2949e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:40218/0 (socket says 172.21.15.28:40218) 2024-04-26T01:45:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.731+0000 7f2949e00640 1 -- 172.21.15.28:0/3051067011 learned_addr learned my addr 172.21.15.28:0/3051067011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.731+0000 7f2949400640 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e7432620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.731+0000 7f294a800640 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e741cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.732+0000 7f2949400640 1 -- 172.21.15.28:0/3051067011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 msgr2=0x55d6e742fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.732+0000 7f2949400640 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.732+0000 7f2949400640 1 -- 172.21.15.28:0/3051067011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 msgr2=0x55d6e741cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.732+0000 7f2949400640 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e741cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.732+0000 7f2949400640 1 -- 172.21.15.28:0/3051067011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6e7095210 con 0x55d6e7430230 2024-04-26T01:45:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.732+0000 7f2949e00640 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.732+0000 7f294a800640 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e741cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.732+0000 7f2949400640 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e7432620 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f293800abc0 tx=0x7f293800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=722c28721977c44f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.733+0000 7f2937400640 1 -- 172.21.15.28:0/3051067011 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938005220 con 0x55d6e7430230 2024-04-26T01:45:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.733+0000 7f2937400640 1 -- 172.21.15.28:0/3051067011 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2938004c50 con 0x55d6e7430230 2024-04-26T01:45:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.733+0000 7f2937400640 1 -- 172.21.15.28:0/3051067011 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938018800 con 0x55d6e7430230 2024-04-26T01:45:12.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.735+0000 7f2954017740 1 -- 172.21.15.28:0/3051067011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 msgr2=0x55d6e7432620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.735+0000 7f2954017740 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e7432620 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f293800abc0 tx=0x7f293800f790 comp rx=0 tx=0).stop 2024-04-26T01:45:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.736+0000 7f2942a00640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f29300a3e90 0x7f29300a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.738+0000 7f294a800640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f29300a3e90 0x7f29300a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.738+0000 7f2954017740 1 -- 172.21.15.28:0/3051067011 shutdown_connections 2024-04-26T01:45:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.738+0000 7f2954017740 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e7432620 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.738+0000 7f2954017740 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.738+0000 7f2954017740 1 --2- 172.21.15.28:0/3051067011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e741cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.738+0000 7f2954017740 1 -- 172.21.15.28:0/3051067011 >> 172.21.15.28:0/3051067011 conn(0x55d6e7418c70 msgr2=0x55d6e7419050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.738+0000 7f2954017740 1 -- 172.21.15.28:0/3051067011 shutdown_connections 2024-04-26T01:45:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.739+0000 7f2942a00640 1 -- 172.21.15.28:0/871938864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f293c0f0a10 con 0x7f294c108f50 2024-04-26T01:45:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.739+0000 7f2954017740 1 -- 172.21.15.28:0/3051067011 wait complete. 2024-04-26T01:45:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.740+0000 7f294a800640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f29300a3e90 0x7f29300a6350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2944007e10 tx=0x7f2944007520 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.741+0000 7f2954017740 1 -- start start 2024-04-26T01:45:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.742+0000 7f2954017740 1 Processor -- start 2024-04-26T01:45:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.744+0000 7f2954017740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e742d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.744+0000 7f2954017740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.744+0000 7f2954017740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e742db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.744+0000 7f2954017740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d6e7432e00 con 0x55d6e741c7e0 2024-04-26T01:45:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.744+0000 7f2954017740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d6e733c310 con 0x55d6e741d100 2024-04-26T01:45:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.744+0000 7f2954017740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d6e7432f80 con 0x55d6e7430230 2024-04-26T01:45:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.745+0000 7f2954017740 10 client.0 _ll_register_callbacks cb 0x55d6e730acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.745+0000 7f294a800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e742db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.745+0000 7f2949e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e742d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.745+0000 7f294a800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e742db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:57336/0 (socket says 172.21.15.28:57336) 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.745+0000 7f294a800640 1 -- 172.21.15.28:0/1328167147 learned_addr learned my addr 172.21.15.28:0/1328167147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.745+0000 7f2949400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.746+0000 7f2949e00640 1 -- 172.21.15.28:0/1328167147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 msgr2=0x55d6e742db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.746+0000 7f2949e00640 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e742db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.746+0000 7f2949e00640 1 -- 172.21.15.28:0/1328167147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 msgr2=0x55d6e742d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.746+0000 7f2949e00640 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.746+0000 7f2949e00640 1 -- 172.21.15.28:0/1328167147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6e7161620 con 0x55d6e741c7e0 2024-04-26T01:45:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.746+0000 7f2949e00640 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e742d0f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f293c0ee790 tx=0x7f293c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4795d0aa5b3fa6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.747+0000 7f292be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:45:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.747+0000 7f292be00640 1 -- 172.21.15.28:0/1328167147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c049030 con 0x55d6e741c7e0 2024-04-26T01:45:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.747+0000 7f292be00640 1 -- 172.21.15.28:0/1328167147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f293c0c51c0 con 0x55d6e741c7e0 2024-04-26T01:45:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.748+0000 7f2954017740 1 -- 172.21.15.28:0/1328167147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2938004c50 con 0x55d6e741c7e0 2024-04-26T01:45:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.748+0000 7f292be00640 1 -- 172.21.15.28:0/1328167147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f293c0f9070 con 0x55d6e741c7e0 2024-04-26T01:45:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.748+0000 7f2954017740 10 client.15528 fetch_fsmap learned FSMap version 29 2024-04-26T01:45:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.748+0000 7f2954017740 1 -- 172.21.15.28:0/1328167147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6e743f060 con 0x55d6e741c7e0 2024-04-26T01:45:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.749+0000 7f292be00640 1 -- 172.21.15.28:0/1328167147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2938004c50 con 0x55d6e741c7e0 2024-04-26T01:45:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.749+0000 7f292be00640 1 -- 172.21.15.28:0/1328167147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f293c04c060 con 0x55d6e741c7e0 2024-04-26T01:45:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.750+0000 7f2954017740 10 client.15528 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:45:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.750+0000 7f2954017740 10 client.15528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:45:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.750+0000 7f2954017740 20 client.15528 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.750+0000 7f2954017740 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d6e7440570 0x55d6e74429d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.750+0000 7f2954017740 4 client.15528 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.750+0000 7f2954017740 1 -- 172.21.15.28:0/1328167147 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6e73968b0 con 0x55d6e7440570 2024-04-26T01:45:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.751+0000 7f2949400640 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d6e7440570 0x55d6e74429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.753+0000 7f292be00640 10 client.15528 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.753+0000 7f2949400640 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d6e7440570 0x55d6e74429d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f4b32a506d519e server_cookie=fb3f8c52a530dcd9 in_seq=0 out_seq=0 2024-04-26T01:45:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.756+0000 7f292be00640 1 -- 172.21.15.28:0/1328167147 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d6e73968b0 con 0x55d6e7440570 2024-04-26T01:45:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.756+0000 7f292be00640 10 client.15528 handle_command_reply: tid=0 2024-04-26T01:45:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.756+0000 7f2954017740 1 client.15528 shutdown 2024-04-26T01:45:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.757+0000 7f2954017740 1 -- 172.21.15.28:0/1328167147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 msgr2=0x55d6e742d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.757+0000 7f2954017740 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e742d0f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f293c0ee790 tx=0x7f293c004040 comp rx=0 tx=0).stop 2024-04-26T01:45:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.759+0000 7f2954017740 1 -- 172.21.15.28:0/1328167147 shutdown_connections 2024-04-26T01:45:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.759+0000 7f2954017740 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d6e7440570 0x55d6e74429d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.759+0000 7f2954017740 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e742db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.759+0000 7f2954017740 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e742d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.759+0000 7f2954017740 1 --2- 172.21.15.28:0/1328167147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e742d0f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.759+0000 7f2954017740 1 -- 172.21.15.28:0/1328167147 >> 172.21.15.28:0/1328167147 conn(0x55d6e7418c70 msgr2=0x55d6e7417e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.759+0000 7f292be00640 0 client.15528 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.760+0000 7f2954017740 1 -- 172.21.15.28:0/1328167147 shutdown_connections 2024-04-26T01:45:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.760+0000 7f2954017740 1 -- 172.21.15.28:0/1328167147 wait complete. 2024-04-26T01:45:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.760+0000 7f2954017740 20 client.15528 trim_cache size 0 max 16384 2024-04-26T01:45:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.808+0000 7f2954017740 1 Processor -- start 2024-04-26T01:45:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.809+0000 7f2954017740 1 -- start start 2024-04-26T01:45:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.810+0000 7f2954017740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e743bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2954017740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e741d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2954017740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e74482e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2954017740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f292c011000 con 0x55d6e741c7e0 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2954017740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f292c00e8d0 con 0x55d6e741d100 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2954017740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f292c00c720 con 0x55d6e7430230 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2949400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e741d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2949400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e741d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:40244/0 (socket says 172.21.15.28:40244) 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2949400640 1 -- 172.21.15.28:0/3290350902 learned_addr learned my addr 172.21.15.28:0/3290350902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f294a800640 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e74482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.811+0000 7f2949e00640 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e743bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.812+0000 7f2949400640 1 -- 172.21.15.28:0/3290350902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 msgr2=0x55d6e74482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.812+0000 7f2949400640 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e74482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.812+0000 7f2949400640 1 -- 172.21.15.28:0/3290350902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 msgr2=0x55d6e743bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.812+0000 7f2949400640 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e743bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.812+0000 7f2949400640 1 -- 172.21.15.28:0/3290350902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6e742c0a0 con 0x55d6e741d100 2024-04-26T01:45:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.812+0000 7f2949e00640 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e743bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.812+0000 7f2949400640 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e741d7b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f293800a310 tx=0x7f293800fd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78afdafc5fe64687 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.813+0000 7f2936a00640 1 -- 172.21.15.28:0/3290350902 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293802bb50 con 0x55d6e741d100 2024-04-26T01:45:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.813+0000 7f2936a00640 1 -- 172.21.15.28:0/3290350902 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f293802f030 con 0x55d6e741d100 2024-04-26T01:45:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.814+0000 7f2936a00640 1 -- 172.21.15.28:0/3290350902 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938005130 con 0x55d6e741d100 2024-04-26T01:45:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.815+0000 7f2954017740 1 -- 172.21.15.28:0/3290350902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 msgr2=0x55d6e741d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.815+0000 7f2954017740 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e741d7b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f293800a310 tx=0x7f293800fd40 comp rx=0 tx=0).stop 2024-04-26T01:45:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.816+0000 7f2954017740 1 -- 172.21.15.28:0/3290350902 shutdown_connections 2024-04-26T01:45:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.816+0000 7f2954017740 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e74482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.816+0000 7f2954017740 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e741d7b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.816+0000 7f2954017740 1 --2- 172.21.15.28:0/3290350902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e743bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.816+0000 7f2954017740 1 -- 172.21.15.28:0/3290350902 >> 172.21.15.28:0/3290350902 conn(0x55d6e7418c70 msgr2=0x55d6e742d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.816+0000 7f2954017740 1 -- 172.21.15.28:0/3290350902 shutdown_connections 2024-04-26T01:45:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.817+0000 7f2954017740 1 -- 172.21.15.28:0/3290350902 wait complete. 2024-04-26T01:45:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.818+0000 7f2954017740 1 -- start start 2024-04-26T01:45:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.818+0000 7f2954017740 1 Processor -- start 2024-04-26T01:45:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.818+0000 7f2954017740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e7435c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2954017740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e7436170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2954017740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e74366b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2954017740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f292c00cd90 con 0x55d6e741c7e0 2024-04-26T01:45:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2954017740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f292c00e200 con 0x55d6e741d100 2024-04-26T01:45:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2954017740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f292c00d460 con 0x55d6e7430230 2024-04-26T01:45:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2954017740 10 client.0 _ll_register_callbacks cb 0x7f293c0f4250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f294a800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e74366b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2949400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e7436170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2949400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e7436170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:40248/0 (socket says 172.21.15.28:40248) 2024-04-26T01:45:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.819+0000 7f2949400640 1 -- 172.21.15.28:0/2424014391 learned_addr learned my addr 172.21.15.28:0/2424014391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:12.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.820+0000 7f2949e00640 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e7435c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.820+0000 7f2949400640 1 -- 172.21.15.28:0/2424014391 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 msgr2=0x55d6e74366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:12.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.820+0000 7f2949400640 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e74366b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.820+0000 7f2949400640 1 -- 172.21.15.28:0/2424014391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 msgr2=0x55d6e7435c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:12.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.820+0000 7f2949400640 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e7435c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.820+0000 7f2949400640 1 -- 172.21.15.28:0/2424014391 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6e741f480 con 0x55d6e741d100 2024-04-26T01:45:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.821+0000 7f2949400640 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e7436170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2938000c00 tx=0x7f293802cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa472ba123e8bb64 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.821+0000 7f292be00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:45:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.822+0000 7f292be00640 1 -- 172.21.15.28:0/2424014391 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293802b2d0 con 0x55d6e741d100 2024-04-26T01:45:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.822+0000 7f292be00640 1 -- 172.21.15.28:0/2424014391 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f293802b470 con 0x55d6e741d100 2024-04-26T01:45:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.822+0000 7f2954017740 1 -- 172.21.15.28:0/2424014391 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f293802f030 con 0x55d6e741d100 2024-04-26T01:45:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.823+0000 7f292be00640 1 -- 172.21.15.28:0/2424014391 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f293802b670 con 0x55d6e741d100 2024-04-26T01:45:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.823+0000 7f2954017740 10 client.25685 fetch_fsmap learned FSMap version 29 2024-04-26T01:45:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.823+0000 7f2954017740 1 -- 172.21.15.28:0/2424014391 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6e74206e0 con 0x55d6e741d100 2024-04-26T01:45:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.824+0000 7f292be00640 1 -- 172.21.15.28:0/2424014391 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f293802f030 con 0x55d6e741d100 2024-04-26T01:45:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.824+0000 7f292be00640 1 -- 172.21.15.28:0/2424014391 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f2938004a30 con 0x55d6e741d100 2024-04-26T01:45:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.824+0000 7f2954017740 10 client.25685 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:45:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.824+0000 7f2954017740 10 client.25685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:45:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.825+0000 7f2954017740 20 client.25685 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.825+0000 7f2954017740 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d6e7420d50 0x55d6e7423170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.825+0000 7f2954017740 4 client.25685 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:45:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.825+0000 7f2954017740 1 -- 172.21.15.28:0/2424014391 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6e741de90 con 0x55d6e7420d50 2024-04-26T01:45:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.826+0000 7f2949e00640 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d6e7420d50 0x55d6e7423170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:12.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.827+0000 7f2949e00640 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d6e7420d50 0x55d6e7423170 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a70a28adfbc384d4 server_cookie=ec451e632a8b361b in_seq=0 out_seq=0 2024-04-26T01:45:12.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.828+0000 7f292be00640 10 client.25685 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:12.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.842+0000 7f292be00640 1 -- 172.21.15.28:0/2424014391 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d6e741de90 con 0x55d6e7420d50 2024-04-26T01:45:12.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.842+0000 7f292be00640 10 client.25685 handle_command_reply: tid=0 2024-04-26T01:45:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.842+0000 7f2954017740 1 client.25685 shutdown 2024-04-26T01:45:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f2954017740 1 -- 172.21.15.28:0/2424014391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 msgr2=0x55d6e7436170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f2954017740 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e7436170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2938000c00 tx=0x7f293802cbe0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f2954017740 1 -- 172.21.15.28:0/2424014391 shutdown_connections 2024-04-26T01:45:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f2954017740 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d6e7420d50 0x55d6e7423170 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f2954017740 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d6e7430230 0x55d6e74366b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f2954017740 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d6e741d100 0x55d6e7436170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f292be00640 0 client.25685 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f2954017740 1 --2- 172.21.15.28:0/2424014391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6e741c7e0 0x55d6e7435c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.843+0000 7f2954017740 1 -- 172.21.15.28:0/2424014391 >> 172.21.15.28:0/2424014391 conn(0x55d6e7418c70 msgr2=0x55d6e741e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.844+0000 7f2954017740 1 -- 172.21.15.28:0/2424014391 shutdown_connections 2024-04-26T01:45:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.844+0000 7f2954017740 1 -- 172.21.15.28:0/2424014391 wait complete. 2024-04-26T01:45:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.844+0000 7f2954017740 20 client.25685 trim_cache size 0 max 16384 2024-04-26T01:45:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:45:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:45:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:45:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.845+0000 7f2951200640 1 -- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f29300a3e90 msgr2=0x7f29300a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.845+0000 7f2951200640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f29300a3e90 0x7f29300a6350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2944007e10 tx=0x7f2944007520 comp rx=0 tx=0).stop 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 -- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c108f50 msgr2=0x7f294c1a2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c108f50 0x7f294c1a2f60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f293c02bc90 tx=0x7f293c003cd0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 -- 172.21.15.28:0/871938864 shutdown_connections 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f29300a3e90 0x7f29300a6350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f294c10e980 0x7f294c1a7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f294c1098f0 0x7f294c1a34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 --2- 172.21.15.28:0/871938864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c108f50 0x7f294c1a2f60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 -- 172.21.15.28:0/871938864 >> 172.21.15.28:0/871938864 conn(0x7f294c06d4b0 msgr2=0x7f294c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 -- 172.21.15.28:0/871938864 shutdown_connections 2024-04-26T01:45:12.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:12.846+0000 7f2951200640 1 -- 172.21.15.28:0/871938864 wait complete. 2024-04-26T01:45:13.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:13 smithi169 ceph-mon[32851]: pgmap v130: 97 pgs: 97 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 897 KiB/s wr, 115 op/s 2024-04-26T01:45:13.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:13 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:13.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:13 smithi053 ceph-mon[32593]: pgmap v130: 97 pgs: 97 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 897 KiB/s wr, 115 op/s 2024-04-26T01:45:13.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:13 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:13.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:13 smithi028 ceph-mon[28247]: pgmap v130: 97 pgs: 97 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 897 KiB/s wr, 115 op/s 2024-04-26T01:45:13.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:13 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:15.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:15 smithi169 ceph-mon[32851]: pgmap v131: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 3.9 MiB/s wr, 427 op/s 2024-04-26T01:45:15.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:15 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:15.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:15 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:15.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:15 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:45:15.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:15 smithi053 ceph-mon[32593]: pgmap v131: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 3.9 MiB/s wr, 427 op/s 2024-04-26T01:45:15.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:15 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:15.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:15 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:15.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:15 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:45:15.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:15 smithi028 ceph-mon[28247]: pgmap v131: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 3.9 MiB/s wr, 427 op/s 2024-04-26T01:45:15.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:15 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:15.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:15 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:15.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:15 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:45:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 200.0 s, 3031.6 tps, lat 32.754 ms stddev 36.139 2024-04-26T01:45:17.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:17 smithi053 ceph-mon[32593]: pgmap v132: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 406 op/s 2024-04-26T01:45:17.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:17 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:17.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:17 smithi028 ceph-mon[28247]: pgmap v132: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 406 op/s 2024-04-26T01:45:17.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:17 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:17 smithi169 ceph-mon[32851]: pgmap v132: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 406 op/s 2024-04-26T01:45:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:17 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:18.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:18 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:18.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:18 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:18.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:18 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:19.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:19 smithi053 ceph-mon[32593]: pgmap v133: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 406 op/s 2024-04-26T01:45:19.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:19 smithi028 ceph-mon[28247]: pgmap v133: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 406 op/s 2024-04-26T01:45:19.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:19 smithi169 ceph-mon[32851]: pgmap v133: 97 pgs: 97 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 406 op/s 2024-04-26T01:45:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 205.0 s, 455.2 tps, lat 209.960 ms stddev 901.199 2024-04-26T01:45:21.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:21 smithi053 ceph-mon[32593]: pgmap v134: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 5.9 MiB/s wr, 633 op/s 2024-04-26T01:45:21.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:21 smithi028 ceph-mon[28247]: pgmap v134: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 5.9 MiB/s wr, 633 op/s 2024-04-26T01:45:21.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:21 smithi169 ceph-mon[32851]: pgmap v134: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 5.9 MiB/s wr, 633 op/s 2024-04-26T01:45:23.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:23 smithi053 ceph-mon[32593]: pgmap v135: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:45:23.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:23 smithi028 ceph-mon[28247]: pgmap v135: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:45:23.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:23 smithi169 ceph-mon[32851]: pgmap v135: 97 pgs: 97 active+clean; 1.9 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:45:25.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:25 smithi053 ceph-mon[32593]: pgmap v136: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.4 MiB/s wr, 751 op/s 2024-04-26T01:45:25.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:25 smithi028 ceph-mon[28247]: pgmap v136: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.4 MiB/s wr, 751 op/s 2024-04-26T01:45:25.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:25 smithi169 ceph-mon[32851]: pgmap v136: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.4 MiB/s wr, 751 op/s 2024-04-26T01:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.049+0000 7fd836400640 1 -- 172.21.15.28:0/1923984103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301092c0 msgr2=0x7fd830110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.049+0000 7fd836400640 1 --2- 172.21.15.28:0/1923984103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301092c0 0x7fd830110e90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd828009ca0 tx=0x7fd82802a8c0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.051+0000 7fd836400640 1 -- 172.21.15.28:0/1923984103 shutdown_connections 2024-04-26T01:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.052+0000 7fd836400640 1 --2- 172.21.15.28:0/1923984103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301092c0 0x7fd830110e90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.052+0000 7fd836400640 1 --2- 172.21.15.28:0/1923984103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd830104ba0 0x7fd830108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.052+0000 7fd836400640 1 --2- 172.21.15.28:0/1923984103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8301041f0 0x7fd8301045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.052+0000 7fd836400640 1 -- 172.21.15.28:0/1923984103 >> 172.21.15.28:0/1923984103 conn(0x7fd8300ffa30 msgr2=0x7fd830101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.053+0000 7fd836400640 1 -- 172.21.15.28:0/1923984103 shutdown_connections 2024-04-26T01:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.053+0000 7fd836400640 1 -- 172.21.15.28:0/1923984103 wait complete. 2024-04-26T01:45:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.056+0000 7fd836400640 1 Processor -- start 2024-04-26T01:45:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.056+0000 7fd836400640 1 -- start start 2024-04-26T01:45:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd836400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301041f0 0x7fd830110d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd836400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8301092c0 0x7fd83010de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd836400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83010e3b0 0x7fd83010e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd835000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301041f0 0x7fd830110d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd835000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301041f0 0x7fd830110d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:56780/0 (socket says 172.21.15.28:56780) 2024-04-26T01:45:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd835000640 1 -- 172.21.15.28:0/1050943660 learned_addr learned my addr 172.21.15.28:0/1050943660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd830114210 con 0x7fd83010e3b0 2024-04-26T01:45:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd830114090 con 0x7fd8301041f0 2024-04-26T01:45:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.057+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd830114390 con 0x7fd8301092c0 2024-04-26T01:45:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.058+0000 7fd82fe00640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8301092c0 0x7fd83010de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.058+0000 7fd835a00640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83010e3b0 0x7fd83010e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.058+0000 7fd835000640 1 -- 172.21.15.28:0/1050943660 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8301092c0 msgr2=0x7fd83010de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.058+0000 7fd835000640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8301092c0 0x7fd83010de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.058+0000 7fd835000640 1 -- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83010e3b0 msgr2=0x7fd83010e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.058+0000 7fd835000640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83010e3b0 0x7fd83010e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.058+0000 7fd835000640 1 -- 172.21.15.28:0/1050943660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8301ab190 con 0x7fd8301041f0 2024-04-26T01:45:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.059+0000 7fd835000640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301041f0 0x7fd830110d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c920 tx=0x7fd82400cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.059+0000 7fd82d600640 1 -- 172.21.15.28:0/1050943660 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824007c70 con 0x7fd8301041f0 2024-04-26T01:45:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.059+0000 7fd82d600640 1 -- 172.21.15.28:0/1050943660 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd824007e10 con 0x7fd8301041f0 2024-04-26T01:45:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.059+0000 7fd82d600640 1 -- 172.21.15.28:0/1050943660 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824018820 con 0x7fd8301041f0 2024-04-26T01:45:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.060+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8301ab480 con 0x7fd8301041f0 2024-04-26T01:45:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.060+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8301ab980 con 0x7fd8301041f0 2024-04-26T01:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.068+0000 7fd83a78c740 1 Processor -- start 2024-04-26T01:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.069+0000 7fd83a78c740 1 -- start start 2024-04-26T01:45:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd82d600640 1 -- 172.21.15.28:0/1050943660 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd82400f450 con 0x7fd8301041f0 2024-04-26T01:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fcdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fce2d0 0x55d304fd8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 0x55d304fdaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd83a78c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d304f33900 con 0x55d304fd8b90 2024-04-26T01:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd83a78c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d304ed9ef0 con 0x55d304fce2d0 2024-04-26T01:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd83a78c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d304f2f8c0 con 0x55d304fcd9b0 2024-04-26T01:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd835a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fcdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd835a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fcdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55480/0 (socket says 172.21.15.28:55480) 2024-04-26T01:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.071+0000 7fd835a00640 1 -- 172.21.15.28:0/266412472 learned_addr learned my addr 172.21.15.28:0/266412472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.072+0000 7fd835000640 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fce2d0 0x55d304fd8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.072+0000 7fd835000640 1 -- 172.21.15.28:0/266412472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 msgr2=0x55d304fcdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.072+0000 7fd835000640 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fcdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.072+0000 7fd835000640 1 -- 172.21.15.28:0/266412472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 msgr2=0x55d304fdaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:26.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.072+0000 7fd835000640 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 0x55d304fdaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.072+0000 7fd835000640 1 -- 172.21.15.28:0/266412472 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d304c46210 con 0x55d304fce2d0 2024-04-26T01:45:26.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.072+0000 7fd835a00640 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fcdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:26.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.073+0000 7fd835000640 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fce2d0 0x55d304fd8650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd8240ce0e0 tx=0x7fd8240d6cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ece3e0b115e437d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:26.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.074+0000 7fd82d600640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd8180a3e60 0x7fd8180a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.074+0000 7fd822000640 1 -- 172.21.15.28:0/266412472 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8240df070 con 0x55d304fce2d0 2024-04-26T01:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.074+0000 7fd82d600640 1 -- 172.21.15.28:0/1050943660 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fd8240d09c0 con 0x7fd8301041f0 2024-04-26T01:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.074+0000 7fd822000640 1 -- 172.21.15.28:0/266412472 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8240d8c60 con 0x55d304fce2d0 2024-04-26T01:45:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.075+0000 7fd83a78c740 1 -- 172.21.15.28:0/266412472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fce2d0 msgr2=0x55d304fd8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.075+0000 7fd835a00640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd8180a3e60 0x7fd8180a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.075+0000 7fd83a78c740 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fce2d0 0x55d304fd8650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd8240ce0e0 tx=0x7fd8240d6cb0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.077+0000 7fd835a00640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd8180a3e60 0x7fd8180a6320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd828035000 tx=0x7fd82802f440 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.077+0000 7fd83a78c740 1 -- 172.21.15.28:0/266412472 shutdown_connections 2024-04-26T01:45:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.077+0000 7fd83a78c740 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 0x55d304fdaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.077+0000 7fd83a78c740 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fce2d0 0x55d304fd8650 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.077+0000 7fd83a78c740 1 --2- 172.21.15.28:0/266412472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fcdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.077+0000 7fd83a78c740 1 -- 172.21.15.28:0/266412472 >> 172.21.15.28:0/266412472 conn(0x55d304fc9ec0 msgr2=0x55d304fca2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.077+0000 7fd83a78c740 1 -- 172.21.15.28:0/266412472 shutdown_connections 2024-04-26T01:45:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.077+0000 7fd83a78c740 1 -- 172.21.15.28:0/266412472 wait complete. 2024-04-26T01:45:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.078+0000 7fd83a78c740 1 -- start start 2024-04-26T01:45:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.078+0000 7fd83a78c740 1 Processor -- start 2024-04-26T01:45:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fd59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fd5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fd8b90 0x55d304fd6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd83a78c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d304fdb810 con 0x55d304fce2d0 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd83a78c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d304eed310 con 0x55d304fd8b90 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd83a78c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d304fdb990 con 0x55d304fcd9b0 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd83a78c740 10 client.0 _ll_register_callbacks cb 0x55d304ebbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd82fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fd5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd835000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fd59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd82fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fd5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52140/0 (socket says 172.21.15.28:52140) 2024-04-26T01:45:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd82fe00640 1 -- 172.21.15.28:0/3168653573 learned_addr learned my addr 172.21.15.28:0/3168653573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.079+0000 7fd835a00640 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fd8b90 0x55d304fd6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.080+0000 7fd835a00640 1 -- 172.21.15.28:0/3168653573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 msgr2=0x55d304fd59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.080+0000 7fd835a00640 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fd59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.080+0000 7fd835a00640 1 -- 172.21.15.28:0/3168653573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 msgr2=0x55d304fd5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.080+0000 7fd835a00640 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fd5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.080+0000 7fd835a00640 1 -- 172.21.15.28:0/3168653573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d304d12620 con 0x55d304fd8b90 2024-04-26T01:45:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.080+0000 7fd835000640 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fd59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.080+0000 7fd82fe00640 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fd5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.081+0000 7fd835a00640 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fd8b90 0x55d304fd6470 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd82802aeb0 tx=0x7fd8280095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c04a7765b0b6054 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.081+0000 7fd816a00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:45:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.082+0000 7fd816a00640 1 -- 172.21.15.28:0/3168653573 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82804f070 con 0x55d304fd8b90 2024-04-26T01:45:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.082+0000 7fd83a78c740 1 -- 172.21.15.28:0/3168653573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8240d8c60 con 0x55d304fd8b90 2024-04-26T01:45:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.083+0000 7fd816a00640 1 -- 172.21.15.28:0/3168653573 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fd82802f610 con 0x55d304fd8b90 2024-04-26T01:45:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.083+0000 7fd816a00640 1 -- 172.21.15.28:0/3168653573 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd82805b050 con 0x55d304fd8b90 2024-04-26T01:45:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.084+0000 7fd83a78c740 10 client.25697 fetch_fsmap learned FSMap version 31 2024-04-26T01:45:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.084+0000 7fd83a78c740 1 -- 172.21.15.28:0/3168653573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d304fe79d0 con 0x55d304fd8b90 2024-04-26T01:45:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.085+0000 7fd816a00640 1 -- 172.21.15.28:0/3168653573 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8240d8c60 con 0x55d304fd8b90 2024-04-26T01:45:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.085+0000 7fd816a00640 1 -- 172.21.15.28:0/3168653573 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7fd828044040 con 0x55d304fd8b90 2024-04-26T01:45:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.085+0000 7fd83a78c740 10 client.25697 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:45:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.085+0000 7fd83a78c740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:45:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.085+0000 7fd83a78c740 20 client.25697 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.085+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d304fe8f30 0x55d304feb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.085+0000 7fd83a78c740 4 client.25697 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.085+0000 7fd83a78c740 1 -- 172.21.15.28:0/3168653573 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d304f478b0 con 0x55d304fe8f30 2024-04-26T01:45:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.086+0000 7fd835000640 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d304fe8f30 0x55d304feb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.087+0000 7fd835000640 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d304fe8f30 0x55d304feb350 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=807e5099c7c42837 server_cookie=d63f5d3c5b0a553e in_seq=0 out_seq=0 2024-04-26T01:45:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.087+0000 7fd816a00640 10 client.25697 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:26.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.088+0000 7fd816a00640 1 -- 172.21.15.28:0/3168653573 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d304f478b0 con 0x55d304fe8f30 2024-04-26T01:45:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.088+0000 7fd816a00640 10 client.25697 handle_command_reply: tid=0 2024-04-26T01:45:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.088+0000 7fd83a78c740 1 client.25697 shutdown 2024-04-26T01:45:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd83a78c740 1 -- 172.21.15.28:0/3168653573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fd8b90 msgr2=0x55d304fd6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fd8b90 0x55d304fd6470 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd82802aeb0 tx=0x7fd8280095c0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd83a78c740 1 -- 172.21.15.28:0/3168653573 shutdown_connections 2024-04-26T01:45:26.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d304fe8f30 0x55d304feb350 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fd8b90 0x55d304fd6470 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fd5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3168653573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fcd9b0 0x55d304fd59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd83a78c740 1 -- 172.21.15.28:0/3168653573 >> 172.21.15.28:0/3168653573 conn(0x55d304fc9ec0 msgr2=0x55d304fcfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.090+0000 7fd816a00640 0 client.25697 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.091+0000 7fd83a78c740 1 -- 172.21.15.28:0/3168653573 shutdown_connections 2024-04-26T01:45:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.091+0000 7fd83a78c740 1 -- 172.21.15.28:0/3168653573 wait complete. 2024-04-26T01:45:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.091+0000 7fd83a78c740 20 client.25697 trim_cache size 0 max 16384 2024-04-26T01:45:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.119+0000 7fd83a78c740 1 Processor -- start 2024-04-26T01:45:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.119+0000 7fd83a78c740 1 -- start start 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.120+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 0x55d304fe4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.120+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fce2d0 0x55d304feaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.120+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 0x55d304fd4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.120+0000 7fd83a78c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301b6a70 con 0x55d304fd8b90 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.120+0000 7fd83a78c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301b42e0 con 0x55d304fcd9b0 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.120+0000 7fd83a78c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301b2130 con 0x55d304fce2d0 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd82fe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fce2d0 0x55d304feaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd82fe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fce2d0 0x55d304feaf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55498/0 (socket says 172.21.15.28:55498) 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd82fe00640 1 -- 172.21.15.28:0/3092156539 learned_addr learned my addr 172.21.15.28:0/3092156539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd835000640 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 0x55d304fe4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd835a00640 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 0x55d304fd4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd82fe00640 1 -- 172.21.15.28:0/3092156539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 msgr2=0x55d304fe4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd82fe00640 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 0x55d304fe4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd82fe00640 1 -- 172.21.15.28:0/3092156539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 msgr2=0x55d304fd4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd82fe00640 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 0x55d304fd4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd82fe00640 1 -- 172.21.15.28:0/3092156539 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d304fd3f40 con 0x55d304fce2d0 2024-04-26T01:45:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.121+0000 7fd835a00640 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 0x55d304fd4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.122+0000 7fd82fe00640 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fce2d0 0x55d304feaf40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009f50 tx=0x7fd81c0128e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce4e92d0d7dac786 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.122+0000 7fd821600640 1 -- 172.21.15.28:0/3092156539 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c004020 con 0x55d304fce2d0 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.122+0000 7fd821600640 1 -- 172.21.15.28:0/3092156539 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd81c0041c0 con 0x55d304fce2d0 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.122+0000 7fd821600640 1 -- 172.21.15.28:0/3092156539 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c01c930 con 0x55d304fce2d0 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 -- 172.21.15.28:0/3092156539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fce2d0 msgr2=0x55d304feaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fce2d0 0x55d304feaf40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009f50 tx=0x7fd81c0128e0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 -- 172.21.15.28:0/3092156539 shutdown_connections 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fd8b90 0x55d304fd4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fce2d0 0x55d304feaf40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3092156539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 0x55d304fe4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 -- 172.21.15.28:0/3092156539 >> 172.21.15.28:0/3092156539 conn(0x55d304fc9ec0 msgr2=0x55d304fcf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 -- 172.21.15.28:0/3092156539 shutdown_connections 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 -- 172.21.15.28:0/3092156539 wait complete. 2024-04-26T01:45:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.123+0000 7fd83a78c740 1 -- start start 2024-04-26T01:45:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.124+0000 7fd83a78c740 1 Processor -- start 2024-04-26T01:45:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 0x55d304fdbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fdc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd83a78c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fd8b90 0x55d30504d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd835a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fd8b90 0x55d30504d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd83a78c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301b27a0 con 0x55d304fce2d0 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd835000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 0x55d304fdbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd835a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fd8b90 0x55d30504d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55502/0 (socket says 172.21.15.28:55502) 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd835a00640 1 -- 172.21.15.28:0/3568597846 learned_addr learned my addr 172.21.15.28:0/3568597846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.125+0000 7fd82fe00640 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fdc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.126+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301b3c10 con 0x55d304fcd9b0 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.126+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301b2e70 con 0x55d304fd8b90 2024-04-26T01:45:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.126+0000 7fd83a78c740 10 client.0 _ll_register_callbacks cb 0x7fd828005f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.127+0000 7fd835a00640 1 -- 172.21.15.28:0/3568597846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 msgr2=0x55d304fdbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.127+0000 7fd835a00640 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 0x55d304fdbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.127+0000 7fd835a00640 1 -- 172.21.15.28:0/3568597846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 msgr2=0x55d304fdc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.127+0000 7fd835a00640 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fdc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.127+0000 7fd835a00640 1 -- 172.21.15.28:0/3568597846 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d30504da90 con 0x55d304fd8b90 2024-04-26T01:45:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.128+0000 7fd816a00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:45:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.128+0000 7fd835a00640 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fd8b90 0x55d30504d480 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd828042040 tx=0x7fd828004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5b789e692a22b6d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.129+0000 7fd816a00640 1 -- 172.21.15.28:0/3568597846 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82804f070 con 0x55d304fd8b90 2024-04-26T01:45:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.129+0000 7fd816a00640 1 -- 172.21.15.28:0/3568597846 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fd828004b40 con 0x55d304fd8b90 2024-04-26T01:45:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.129+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd81c0041c0 con 0x55d304fd8b90 2024-04-26T01:45:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.129+0000 7fd816a00640 1 -- 172.21.15.28:0/3568597846 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd82802af40 con 0x55d304fd8b90 2024-04-26T01:45:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.130+0000 7fd83a78c740 10 client.25720 fetch_fsmap learned FSMap version 31 2024-04-26T01:45:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.130+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d30504ed10 con 0x55d304fd8b90 2024-04-26T01:45:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.130+0000 7fd816a00640 1 -- 172.21.15.28:0/3568597846 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd81c0041c0 con 0x55d304fd8b90 2024-04-26T01:45:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.130+0000 7fd816a00640 1 -- 172.21.15.28:0/3568597846 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7fd828044070 con 0x55d304fd8b90 2024-04-26T01:45:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.131+0000 7fd83a78c740 10 client.25720 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:45:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.131+0000 7fd83a78c740 10 client.25720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:45:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.131+0000 7fd83a78c740 20 client.25720 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.131+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d30504f4c0 0x55d3050518e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.131+0000 7fd83a78c740 4 client.25720 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:45:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.131+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d304f41320 con 0x55d30504f4c0 2024-04-26T01:45:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.132+0000 7fd835000640 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d30504f4c0 0x55d3050518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.133+0000 7fd835000640 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d30504f4c0 0x55d3050518e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df252fcf083a75dd server_cookie=9858eb818a3b49b in_seq=0 out_seq=0 2024-04-26T01:45:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.133+0000 7fd816a00640 10 client.25720 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.149+0000 7fd816a00640 1 -- 172.21.15.28:0/3568597846 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d304f41320 con 0x55d30504f4c0 2024-04-26T01:45:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd816a00640 10 client.25720 handle_command_reply: tid=0 2024-04-26T01:45:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 client.25720 shutdown 2024-04-26T01:45:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fd8b90 msgr2=0x55d30504d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fd8b90 0x55d30504d480 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd828042040 tx=0x7fd828004ab0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 shutdown_connections 2024-04-26T01:45:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d30504f4c0 0x55d3050518e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d304fd8b90 0x55d30504d480 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d304fce2d0 0x55d304fdc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 --2- 172.21.15.28:0/3568597846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d304fcd9b0 0x55d304fdbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 >> 172.21.15.28:0/3568597846 conn(0x55d304fc9ec0 msgr2=0x55d304fd42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.150+0000 7fd816a00640 0 client.25720 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.151+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 shutdown_connections 2024-04-26T01:45:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.151+0000 7fd83a78c740 1 -- 172.21.15.28:0/3568597846 wait complete. 2024-04-26T01:45:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.152+0000 7fd83a78c740 20 client.25720 trim_cache size 0 max 16384 2024-04-26T01:45:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:45:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:45:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:45:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:45:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd8180a3e60 msgr2=0x7fd8180a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd8180a3e60 0x7fd8180a6320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd828035000 tx=0x7fd82802f440 comp rx=0 tx=0).stop 2024-04-26T01:45:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301041f0 msgr2=0x7fd830110d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301041f0 0x7fd830110d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c920 tx=0x7fd82400cde0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 shutdown_connections 2024-04-26T01:45:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd8180a3e60 0x7fd8180a6320 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83010e3b0 0x7fd83010e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8301092c0 0x7fd83010de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 --2- 172.21.15.28:0/1050943660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8301041f0 0x7fd830110d80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.153+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 >> 172.21.15.28:0/1050943660 conn(0x7fd8300ffa30 msgr2=0x7fd830101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.154+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 shutdown_connections 2024-04-26T01:45:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:26.154+0000 7fd836400640 1 -- 172.21.15.28:0/1050943660 wait complete. 2024-04-26T01:45:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 210.0 s, 3464.8 tps, lat 30.278 ms stddev 86.878 2024-04-26T01:45:27.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:27 smithi053 ceph-mon[32593]: pgmap v137: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 439 op/s 2024-04-26T01:45:27.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:27 smithi028 ceph-mon[28247]: pgmap v137: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 439 op/s 2024-04-26T01:45:27.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:27 smithi169 ceph-mon[32851]: pgmap v137: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 439 op/s 2024-04-26T01:45:29.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:29 smithi053 ceph-mon[32593]: pgmap v138: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 439 op/s 2024-04-26T01:45:29.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:29 smithi028 ceph-mon[28247]: pgmap v138: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 439 op/s 2024-04-26T01:45:29.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:29 smithi169 ceph-mon[32851]: pgmap v138: 97 pgs: 97 active+clean; 1.9 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 439 op/s 2024-04-26T01:45:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 215.0 s, 3170.3 tps, lat 31.434 ms stddev 35.551 2024-04-26T01:45:31.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:31 smithi053 ceph-mon[32593]: pgmap v139: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.5 MiB/s wr, 763 op/s 2024-04-26T01:45:31.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:31 smithi028 ceph-mon[28247]: pgmap v139: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.5 MiB/s wr, 763 op/s 2024-04-26T01:45:31.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:31 smithi169 ceph-mon[32851]: pgmap v139: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.5 MiB/s wr, 763 op/s 2024-04-26T01:45:33.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:33 smithi053 ceph-mon[32593]: pgmap v140: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 536 op/s 2024-04-26T01:45:33.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:33 smithi028 ceph-mon[28247]: pgmap v140: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 536 op/s 2024-04-26T01:45:33.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:33 smithi169 ceph-mon[32851]: pgmap v140: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 536 op/s 2024-04-26T01:45:35.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:35 smithi053 ceph-mon[32593]: pgmap v141: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.3 MiB/s wr, 831 op/s 2024-04-26T01:45:35.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:35 smithi028 ceph-mon[28247]: pgmap v141: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.3 MiB/s wr, 831 op/s 2024-04-26T01:45:35.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:35 smithi169 ceph-mon[32851]: pgmap v141: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.3 MiB/s wr, 831 op/s 2024-04-26T01:45:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 220.0 s, 2767.7 tps, lat 35.870 ms stddev 43.257 2024-04-26T01:45:37.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:37 smithi028 ceph-mon[28247]: pgmap v142: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 618 op/s 2024-04-26T01:45:37.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:37 smithi169 ceph-mon[32851]: pgmap v142: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 618 op/s 2024-04-26T01:45:37.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:37 smithi053 ceph-mon[32593]: pgmap v142: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 618 op/s 2024-04-26T01:45:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.344+0000 7f547e000640 1 -- 172.21.15.28:0/3445737354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f547810e600 msgr2=0x7f547810e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.344+0000 7f547e000640 1 --2- 172.21.15.28:0/3445737354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f547810e600 0x7f547810e9e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5460009930 tx=0x7f546002a420 comp rx=0 tx=0).stop 2024-04-26T01:45:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.346+0000 7f547e000640 1 -- 172.21.15.28:0/3445737354 shutdown_connections 2024-04-26T01:45:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.346+0000 7f547e000640 1 --2- 172.21.15.28:0/3445737354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5478109750 0x7f5478115410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.346+0000 7f547e000640 1 --2- 172.21.15.28:0/3445737354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478108d90 0x7f5478109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.346+0000 7f547e000640 1 --2- 172.21.15.28:0/3445737354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f547810e600 0x7f547810e9e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.346+0000 7f547e000640 1 -- 172.21.15.28:0/3445737354 >> 172.21.15.28:0/3445737354 conn(0x7f5478077cb0 msgr2=0x7f547807a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.346+0000 7f547e000640 1 -- 172.21.15.28:0/3445737354 shutdown_connections 2024-04-26T01:45:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.346+0000 7f547e000640 1 -- 172.21.15.28:0/3445737354 wait complete. 2024-04-26T01:45:39.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.347+0000 7f547e000640 1 Processor -- start 2024-04-26T01:45:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.348+0000 7f547e000640 1 -- start start 2024-04-26T01:45:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.349+0000 7f547e000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5478108d90 0x7f54781a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.349+0000 7f547e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478109750 0x7f54781a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.349+0000 7f547e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f547810e600 0x7f547819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.349+0000 7f547e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54781187d0 con 0x7f5478109750 2024-04-26T01:45:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.349+0000 7f547e000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5478118650 con 0x7f547810e600 2024-04-26T01:45:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.349+0000 7f547e000640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5478118950 con 0x7f5478108d90 2024-04-26T01:45:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5478108d90 0x7f54781a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f547810e600 0x7f547819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5478108d90 0x7f54781a48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58012/0 (socket says 172.21.15.28:58012) 2024-04-26T01:45:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f547810e600 0x7f547819ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36928/0 (socket says 172.21.15.28:36928) 2024-04-26T01:45:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477400640 1 -- 172.21.15.28:0/3320460653 learned_addr learned my addr 172.21.15.28:0/3320460653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5476a00640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478109750 0x7f54781a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477e00640 1 -- 172.21.15.28:0/3320460653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5478108d90 msgr2=0x7f54781a48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477e00640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5478108d90 0x7f54781a48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477e00640 1 -- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478109750 msgr2=0x7f54781a4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477e00640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478109750 0x7f54781a4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5477e00640 1 -- 172.21.15.28:0/3320460653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547819f1d0 con 0x7f547810e600 2024-04-26T01:45:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.350+0000 7f5476a00640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478109750 0x7f54781a4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.352+0000 7f5477e00640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f547810e600 0x7f547819ea50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f547000e940 tx=0x7f547000ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.352+0000 7f546fe00640 1 -- 172.21.15.28:0/3320460653 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470004d20 con 0x7f547810e600 2024-04-26T01:45:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.353+0000 7f546fe00640 1 -- 172.21.15.28:0/3320460653 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5470005020 con 0x7f547810e600 2024-04-26T01:45:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.353+0000 7f546fe00640 1 -- 172.21.15.28:0/3320460653 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54700116d0 con 0x7f547810e600 2024-04-26T01:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.353+0000 7f547e000640 1 -- 172.21.15.28:0/3320460653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547819f4c0 con 0x7f547810e600 2024-04-26T01:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.353+0000 7f547e000640 1 -- 172.21.15.28:0/3320460653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54781ab760 con 0x7f547810e600 2024-04-26T01:45:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.359+0000 7f5480e4f740 1 Processor -- start 2024-04-26T01:45:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.359+0000 7f546fe00640 1 -- 172.21.15.28:0/3320460653 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5470005460 con 0x7f547810e600 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.359+0000 7f5480e4f740 1 -- start start 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.360+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 0x55fc521e1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.360+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.360+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.360+0000 7f5480e4f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fc52147900 con 0x55fc521e2140 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.360+0000 7f5480e4f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fc520edef0 con 0x55fc521e1820 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.360+0000 7f5480e4f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55fc521438c0 con 0x55fc521eca00 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.360+0000 7f5476a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 0x55fc521e1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.360+0000 7f5476a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.361+0000 7f5476a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521eedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58024/0 (socket says 172.21.15.28:58024) 2024-04-26T01:45:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.361+0000 7f5476a00640 1 -- 172.21.15.28:0/978250140 learned_addr learned my addr 172.21.15.28:0/978250140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.361+0000 7f5476a00640 1 -- 172.21.15.28:0/978250140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 msgr2=0x55fc521e1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.361+0000 7f5476a00640 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 0x55fc521e1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.361+0000 7f5476a00640 1 -- 172.21.15.28:0/978250140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 msgr2=0x55fc521ec4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.361+0000 7f5477400640 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.362+0000 7f5476a00640 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521ec4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.362+0000 7f5476a00640 1 -- 172.21.15.28:0/978250140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc51e5a210 con 0x55fc521eca00 2024-04-26T01:45:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.362+0000 7f5477400640 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521ec4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.363+0000 7f5476a00640 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521eedf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5468015930 tx=0x7f5468004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fcf91186943740 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.363+0000 7f546fe00640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f545c0a3e40 0x7f545c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.363+0000 7f546fe00640 1 -- 172.21.15.28:0/3320460653 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f54700d0880 con 0x7f547810e600 2024-04-26T01:45:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.363+0000 7f5467e00640 1 -- 172.21.15.28:0/978250140 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468004220 con 0x55fc521eca00 2024-04-26T01:45:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.364+0000 7f5467e00640 1 -- 172.21.15.28:0/978250140 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54680043c0 con 0x55fc521eca00 2024-04-26T01:45:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.364+0000 7f5467e00640 1 -- 172.21.15.28:0/978250140 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468004d40 con 0x55fc521eca00 2024-04-26T01:45:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.365+0000 7f5477400640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f545c0a3e40 0x7f545c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.365+0000 7f5480e4f740 1 -- 172.21.15.28:0/978250140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 msgr2=0x55fc521eedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.365+0000 7f5480e4f740 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521eedf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5468015930 tx=0x7f5468004730 comp rx=0 tx=0).stop 2024-04-26T01:45:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.366+0000 7f5477400640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f545c0a3e40 0x7f545c0a6300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5460009900 tx=0x7f54600058d0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.366+0000 7f5480e4f740 1 -- 172.21.15.28:0/978250140 shutdown_connections 2024-04-26T01:45:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.367+0000 7f5480e4f740 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521eedf0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.367+0000 7f5480e4f740 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521ec4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.367+0000 7f5480e4f740 1 --2- 172.21.15.28:0/978250140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 0x55fc521e1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.368+0000 7f5480e4f740 1 -- 172.21.15.28:0/978250140 >> 172.21.15.28:0/978250140 conn(0x55fc521ddd30 msgr2=0x55fc521de110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.368+0000 7f5480e4f740 1 -- 172.21.15.28:0/978250140 shutdown_connections 2024-04-26T01:45:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.368+0000 7f5480e4f740 1 -- 172.21.15.28:0/978250140 wait complete. 2024-04-26T01:45:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.368+0000 7f5480e4f740 1 -- start start 2024-04-26T01:45:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.369+0000 7f5480e4f740 1 Processor -- start 2024-04-26T01:45:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.369+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 0x55fc521e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.369+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521e9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.369+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.369+0000 7f5480e4f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fc521ef5d0 con 0x55fc521e2140 2024-04-26T01:45:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.369+0000 7f5480e4f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fc52101310 con 0x55fc521e1820 2024-04-26T01:45:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5480e4f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55fc521ef750 con 0x55fc521eca00 2024-04-26T01:45:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5480e4f740 10 client.0 _ll_register_callbacks cb 0x55fc520cfcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 0x55fc521e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521ea4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58040/0 (socket says 172.21.15.28:58040) 2024-04-26T01:45:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477e00640 1 -- 172.21.15.28:0/3004688631 learned_addr learned my addr 172.21.15.28:0/3004688631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5476a00640 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521e9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477e00640 1 -- 172.21.15.28:0/3004688631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 msgr2=0x55fc521e9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477e00640 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 0x55fc521e9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477e00640 1 -- 172.21.15.28:0/3004688631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 msgr2=0x55fc521e9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477e00640 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521e9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.370+0000 7f5477e00640 1 -- 172.21.15.28:0/3004688631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc51f26620 con 0x55fc521eca00 2024-04-26T01:45:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.371+0000 7f5477e00640 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521ea4a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5470099200 tx=0x7f547009c250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b350c4df2a19ca95 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.372+0000 7f5464c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:45:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.372+0000 7f5464c00640 1 -- 172.21.15.28:0/3004688631 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547001f040 con 0x55fc521eca00 2024-04-26T01:45:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.372+0000 7f5464c00640 1 -- 172.21.15.28:0/3004688631 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f5470098ea0 con 0x55fc521eca00 2024-04-26T01:45:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.372+0000 7f5480e4f740 1 -- 172.21.15.28:0/3004688631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54680043c0 con 0x55fc521eca00 2024-04-26T01:45:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.372+0000 7f5464c00640 1 -- 172.21.15.28:0/3004688631 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f54700a54a0 con 0x55fc521eca00 2024-04-26T01:45:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.372+0000 7f5480e4f740 10 client.25732 fetch_fsmap learned FSMap version 31 2024-04-26T01:45:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.372+0000 7f5480e4f740 1 -- 172.21.15.28:0/3004688631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fc5220c8a0 con 0x55fc521eca00 2024-04-26T01:45:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.373+0000 7f5464c00640 1 -- 172.21.15.28:0/3004688631 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54680043c0 con 0x55fc521eca00 2024-04-26T01:45:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.374+0000 7f5464c00640 1 -- 172.21.15.28:0/3004688631 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f547000f040 con 0x55fc521eca00 2024-04-26T01:45:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.374+0000 7f5480e4f740 10 client.25732 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:45:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.374+0000 7f5480e4f740 10 client.25732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:45:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.374+0000 7f5480e4f740 20 client.25732 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.374+0000 7f5480e4f740 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fc5220e1c0 0x55fc522105d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.374+0000 7f5480e4f740 4 client.25732 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.374+0000 7f5480e4f740 1 -- 172.21.15.28:0/3004688631 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc5215b8b0 con 0x55fc5220e1c0 2024-04-26T01:45:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.375+0000 7f5477400640 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fc5220e1c0 0x55fc522105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.376+0000 7f5477400640 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fc5220e1c0 0x55fc522105d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c08d9bc59fa86d97 server_cookie=4b2f4539ec18047c in_seq=0 out_seq=0 2024-04-26T01:45:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.376+0000 7f5464c00640 10 client.25732 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.376+0000 7f5464c00640 1 -- 172.21.15.28:0/3004688631 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc5215b8b0 con 0x55fc5220e1c0 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.376+0000 7f5464c00640 10 client.25732 handle_command_reply: tid=0 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 client.25732 shutdown 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 -- 172.21.15.28:0/3004688631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 msgr2=0x55fc521ea4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521ea4a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5470099200 tx=0x7f547009c250 comp rx=0 tx=0).stop 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 -- 172.21.15.28:0/3004688631 shutdown_connections 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fc5220e1c0 0x55fc522105d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521eca00 0x55fc521ea4a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521e9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 --2- 172.21.15.28:0/3004688631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521e1820 0x55fc521e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5464c00640 0 client.25732 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 -- 172.21.15.28:0/3004688631 >> 172.21.15.28:0/3004688631 conn(0x55fc521ddd30 msgr2=0x55fc521e3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 -- 172.21.15.28:0/3004688631 shutdown_connections 2024-04-26T01:45:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 1 -- 172.21.15.28:0/3004688631 wait complete. 2024-04-26T01:45:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.377+0000 7f5480e4f740 20 client.25732 trim_cache size 0 max 16384 2024-04-26T01:45:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.405+0000 7f5480e4f740 1 Processor -- start 2024-04-26T01:45:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.405+0000 7f5480e4f740 1 -- start start 2024-04-26T01:45:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.405+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.405+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521dd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.405+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc52215ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.405+0000 7f5480e4f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f547807f0f0 con 0x55fc521e2140 2024-04-26T01:45:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.405+0000 7f5480e4f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f547807c960 con 0x55fc521eca00 2024-04-26T01:45:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.405+0000 7f5480e4f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5478072170 con 0x55fc521e1820 2024-04-26T01:45:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.406+0000 7f5477e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc52215ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.406+0000 7f5477e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc52215ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36940/0 (socket says 172.21.15.28:36940) 2024-04-26T01:45:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.406+0000 7f5477e00640 1 -- 172.21.15.28:0/1789435953 learned_addr learned my addr 172.21.15.28:0/1789435953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.406+0000 7f5477400640 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.406+0000 7f5477e00640 1 -- 172.21.15.28:0/1789435953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 msgr2=0x55fc521f8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.407+0000 7f5477e00640 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.407+0000 7f5477e00640 1 -- 172.21.15.28:0/1789435953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 msgr2=0x55fc521dd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.407+0000 7f5476a00640 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521dd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.407+0000 7f5477e00640 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521dd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.407+0000 7f5477e00640 1 -- 172.21.15.28:0/1789435953 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc521ddb50 con 0x55fc521eca00 2024-04-26T01:45:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.407+0000 7f5477400640 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.407+0000 7f5476a00640 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521dd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.408+0000 7f5477e00640 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc52215ee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f547009c530 tx=0x7f547009ad40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bd9750253bc7871 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.408+0000 7f5467400640 1 -- 172.21.15.28:0/1789435953 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547001f070 con 0x55fc521eca00 2024-04-26T01:45:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.408+0000 7f5467400640 1 -- 172.21.15.28:0/1789435953 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54700a5010 con 0x55fc521eca00 2024-04-26T01:45:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.409+0000 7f5467400640 1 -- 172.21.15.28:0/1789435953 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547009bfb0 con 0x55fc521eca00 2024-04-26T01:45:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.410+0000 7f5480e4f740 1 -- 172.21.15.28:0/1789435953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 msgr2=0x55fc52215ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.410+0000 7f5480e4f740 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc52215ee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f547009c530 tx=0x7f547009ad40 comp rx=0 tx=0).stop 2024-04-26T01:45:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.410+0000 7f5480e4f740 1 -- 172.21.15.28:0/1789435953 shutdown_connections 2024-04-26T01:45:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.410+0000 7f5480e4f740 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc52215ee0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.410+0000 7f5480e4f740 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521dd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.410+0000 7f5480e4f740 1 --2- 172.21.15.28:0/1789435953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.410+0000 7f5480e4f740 1 -- 172.21.15.28:0/1789435953 >> 172.21.15.28:0/1789435953 conn(0x55fc521ddd30 msgr2=0x55fc521e99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.411+0000 7f5480e4f740 1 -- 172.21.15.28:0/1789435953 shutdown_connections 2024-04-26T01:45:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.411+0000 7f5480e4f740 1 -- 172.21.15.28:0/1789435953 wait complete. 2024-04-26T01:45:39.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.412+0000 7f5480e4f740 1 -- start start 2024-04-26T01:45:39.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.413+0000 7f5480e4f740 1 Processor -- start 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.413+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.413+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521f2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.413+0000 7f5480e4f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc521f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5480e4f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54780727e0 con 0x55fc521e2140 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5480e4f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5478073c50 con 0x55fc521eca00 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5480e4f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5478072eb0 con 0x55fc521e1820 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5480e4f740 10 client.0 _ll_register_callbacks cb 0x7f547009c690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58050/0 (socket says 172.21.15.28:58050) 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477400640 1 -- 172.21.15.28:0/601959863 learned_addr learned my addr 172.21.15.28:0/601959863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477e00640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc521f2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5476a00640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477400640 1 -- 172.21.15.28:0/601959863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 msgr2=0x55fc521f2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:39.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477400640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc521f2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477400640 1 -- 172.21.15.28:0/601959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 msgr2=0x55fc521f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:39.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477400640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477400640 1 -- 172.21.15.28:0/601959863 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc52273760 con 0x55fc521e1820 2024-04-26T01:45:39.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5477e00640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc521f2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.414+0000 7f5476a00640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.415+0000 7f5477400640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f22d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5460031440 tx=0x7f5460009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c3471083d00c703 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.415+0000 7f5464c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.415+0000 7f5464c00640 1 -- 172.21.15.28:0/601959863 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460035040 con 0x55fc521e1820 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.416+0000 7f5464c00640 1 -- 172.21.15.28:0/601959863 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f5460043040 con 0x55fc521e1820 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.416+0000 7f5480e4f740 1 -- 172.21.15.28:0/601959863 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54700a5010 con 0x55fc521e1820 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.416+0000 7f5464c00640 1 -- 172.21.15.28:0/601959863 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f546005f050 con 0x55fc521e1820 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.416+0000 7f5480e4f740 10 client.25735 fetch_fsmap learned FSMap version 31 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.416+0000 7f5480e4f740 1 -- 172.21.15.28:0/601959863 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fc522748c0 con 0x55fc521e1820 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.416+0000 7f5464c00640 1 -- 172.21.15.28:0/601959863 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54700a5010 con 0x55fc521e1820 2024-04-26T01:45:39.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.417+0000 7f5464c00640 1 -- 172.21.15.28:0/601959863 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f5460003710 con 0x55fc521e1820 2024-04-26T01:45:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.417+0000 7f5480e4f740 10 client.25735 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:45:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.417+0000 7f5480e4f740 10 client.25735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:45:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.417+0000 7f5480e4f740 20 client.25735 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.417+0000 7f5480e4f740 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fc52274ff0 0x55fc52277450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.417+0000 7f5480e4f740 4 client.25735 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:45:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.417+0000 7f5480e4f740 1 -- 172.21.15.28:0/601959863 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fc52155320 con 0x55fc52274ff0 2024-04-26T01:45:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.418+0000 7f5476a00640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fc52274ff0 0x55fc52277450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.419+0000 7f5476a00640 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fc52274ff0 0x55fc52277450 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48975bcaefbababd server_cookie=f44956de24aa82e2 in_seq=0 out_seq=0 2024-04-26T01:45:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.419+0000 7f5464c00640 10 client.25735 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.433+0000 7f5464c00640 1 -- 172.21.15.28:0/601959863 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fc52155320 con 0x55fc52274ff0 2024-04-26T01:45:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.433+0000 7f5464c00640 10 client.25735 handle_command_reply: tid=0 2024-04-26T01:45:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.433+0000 7f5480e4f740 1 client.25735 shutdown 2024-04-26T01:45:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.434+0000 7f5480e4f740 1 -- 172.21.15.28:0/601959863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 msgr2=0x55fc521f22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.434+0000 7f5480e4f740 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f22d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5460031440 tx=0x7f5460009260 comp rx=0 tx=0).stop 2024-04-26T01:45:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.435+0000 7f5480e4f740 1 -- 172.21.15.28:0/601959863 shutdown_connections 2024-04-26T01:45:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.435+0000 7f5480e4f740 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fc52274ff0 0x55fc52277450 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.435+0000 7f5480e4f740 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc521eca00 0x55fc521f2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.435+0000 7f5480e4f740 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc521e2140 0x55fc521f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.435+0000 7f5480e4f740 1 --2- 172.21.15.28:0/601959863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fc521e1820 0x55fc521f22d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.435+0000 7f5480e4f740 1 -- 172.21.15.28:0/601959863 >> 172.21.15.28:0/601959863 conn(0x55fc521ddd30 msgr2=0x55fc521e99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.435+0000 7f5464c00640 0 client.25735 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.435+0000 7f5480e4f740 1 -- 172.21.15.28:0/601959863 shutdown_connections 2024-04-26T01:45:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.436+0000 7f5480e4f740 1 -- 172.21.15.28:0/601959863 wait complete. 2024-04-26T01:45:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.436+0000 7f5480e4f740 20 client.25735 trim_cache size 0 max 16384 2024-04-26T01:45:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:45:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:45:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:45:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.437+0000 7f547e000640 1 -- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f545c0a3e40 msgr2=0x7f545c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.437+0000 7f547e000640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f545c0a3e40 0x7f545c0a6300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5460009900 tx=0x7f54600058d0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.437+0000 7f547e000640 1 -- 172.21.15.28:0/3320460653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f547810e600 msgr2=0x7f547819ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.437+0000 7f547e000640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f547810e600 0x7f547819ea50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f547000e940 tx=0x7f547000ee00 comp rx=0 tx=0).stop 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.438+0000 7f547e000640 1 -- 172.21.15.28:0/3320460653 shutdown_connections 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.438+0000 7f547e000640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f545c0a3e40 0x7f545c0a6300 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.438+0000 7f547e000640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f547810e600 0x7f547819ea50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.438+0000 7f547e000640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478109750 0x7f54781a4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.438+0000 7f547e000640 1 --2- 172.21.15.28:0/3320460653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5478108d90 0x7f54781a48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.438+0000 7f547e000640 1 -- 172.21.15.28:0/3320460653 >> 172.21.15.28:0/3320460653 conn(0x7f5478077cb0 msgr2=0x7f547807a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:39.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.438+0000 7f547e000640 1 -- 172.21.15.28:0/3320460653 shutdown_connections 2024-04-26T01:45:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:39.438+0000 7f547e000640 1 -- 172.21.15.28:0/3320460653 wait complete. 2024-04-26T01:45:39.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:39 smithi028 ceph-mon[28247]: pgmap v143: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 618 op/s 2024-04-26T01:45:39.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:39 smithi169 ceph-mon[32851]: pgmap v143: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 618 op/s 2024-04-26T01:45:39.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:39 smithi053 ceph-mon[32593]: pgmap v143: 97 pgs: 97 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 618 op/s 2024-04-26T01:45:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 225.0 s, 3047.4 tps, lat 33.050 ms stddev 38.670 2024-04-26T01:45:41.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:41 smithi028 ceph-mon[28247]: pgmap v144: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 961 op/s 2024-04-26T01:45:41.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:41 smithi169 ceph-mon[32851]: pgmap v144: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 961 op/s 2024-04-26T01:45:41.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:41 smithi053 ceph-mon[32593]: pgmap v144: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 961 op/s 2024-04-26T01:45:43.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:43 smithi028 ceph-mon[28247]: pgmap v145: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.5 MiB/s wr, 637 op/s 2024-04-26T01:45:43.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:43 smithi169 ceph-mon[32851]: pgmap v145: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.5 MiB/s wr, 637 op/s 2024-04-26T01:45:43.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:43 smithi053 ceph-mon[32593]: pgmap v145: 97 pgs: 97 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.5 MiB/s wr, 637 op/s 2024-04-26T01:45:45.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:45 smithi028 ceph-mon[28247]: pgmap v146: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 946 op/s 2024-04-26T01:45:45.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:45 smithi169 ceph-mon[32851]: pgmap v146: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 946 op/s 2024-04-26T01:45:45.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:45 smithi053 ceph-mon[32593]: pgmap v146: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 946 op/s 2024-04-26T01:45:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 230.0 s, 2757.0 tps, lat 36.126 ms stddev 42.181 2024-04-26T01:45:47.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:47 smithi028 ceph-mon[28247]: pgmap v147: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-04-26T01:45:47.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:47 smithi169 ceph-mon[32851]: pgmap v147: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-04-26T01:45:47.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:47 smithi053 ceph-mon[32593]: pgmap v147: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-04-26T01:45:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:48 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:48.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:48 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:48.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:48 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:49 smithi028 ceph-mon[28247]: pgmap v148: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-04-26T01:45:49.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:49 smithi169 ceph-mon[32851]: pgmap v148: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-04-26T01:45:49.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:49 smithi053 ceph-mon[32593]: pgmap v148: 97 pgs: 97 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-04-26T01:45:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 235.0 s, 2853.2 tps, lat 35.117 ms stddev 42.929 2024-04-26T01:45:51.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:51 smithi028 ceph-mon[28247]: pgmap v149: 97 pgs: 97 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 948 op/s 2024-04-26T01:45:51.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:51 smithi169 ceph-mon[32851]: pgmap v149: 97 pgs: 97 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 948 op/s 2024-04-26T01:45:51.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:51 smithi053 ceph-mon[32593]: pgmap v149: 97 pgs: 97 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 948 op/s 2024-04-26T01:45:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.654+0000 7f1786000640 1 -- 172.21.15.28:0/1331544379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 msgr2=0x7f1780107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.654+0000 7f1786000640 1 --2- 172.21.15.28:0/1331544379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 0x7f1780107310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f176c00dcc0 tx=0x7f176c02e390 comp rx=0 tx=0).stop 2024-04-26T01:45:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.655+0000 7f1786000640 1 -- 172.21.15.28:0/1331544379 shutdown_connections 2024-04-26T01:45:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.655+0000 7f1786000640 1 --2- 172.21.15.28:0/1331544379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1780100e10 0x7f178010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.655+0000 7f1786000640 1 --2- 172.21.15.28:0/1331544379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1780100450 0x7f17801008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.655+0000 7f1786000640 1 --2- 172.21.15.28:0/1331544379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 0x7f1780107310 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.655+0000 7f1786000640 1 -- 172.21.15.28:0/1331544379 >> 172.21.15.28:0/1331544379 conn(0x7f17800773e0 msgr2=0x7f17800fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.655+0000 7f1786000640 1 -- 172.21.15.28:0/1331544379 shutdown_connections 2024-04-26T01:45:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.655+0000 7f1786000640 1 -- 172.21.15.28:0/1331544379 wait complete. 2024-04-26T01:45:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.657+0000 7f1786000640 1 Processor -- start 2024-04-26T01:45:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.657+0000 7f1786000640 1 -- start start 2024-04-26T01:45:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.657+0000 7f1786000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1780100450 0x7f178019d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.657+0000 7f1786000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1780100e10 0x7f178019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.657+0000 7f1786000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 0x7f1780196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.657+0000 7f1786000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1780110130 con 0x7f1780100450 2024-04-26T01:45:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.657+0000 7f1786000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f178010ffb0 con 0x7f1780100e10 2024-04-26T01:45:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.657+0000 7f1786000640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17801102b0 con 0x7f1780106f30 2024-04-26T01:45:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1780100e10 0x7f178019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1780100e10 0x7f178019d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37130/0 (socket says 172.21.15.28:37130) 2024-04-26T01:45:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177ea00640 1 -- 172.21.15.28:0/524983459 learned_addr learned my addr 172.21.15.28:0/524983459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177fe00640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 0x7f1780196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177f400640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1780100450 0x7f178019d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177ea00640 1 -- 172.21.15.28:0/524983459 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 msgr2=0x7f1780196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177ea00640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 0x7f1780196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177ea00640 1 -- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1780100450 msgr2=0x7f178019d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177ea00640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1780100450 0x7f178019d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177ea00640 1 -- 172.21.15.28:0/524983459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1780196dd0 con 0x7f1780100e10 2024-04-26T01:45:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.658+0000 7f177fe00640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 0x7f1780196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.662+0000 7f177ea00640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1780100e10 0x7f178019d7f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f177800eab0 tx=0x7f177800ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.662+0000 7f1777e00640 1 -- 172.21.15.28:0/524983459 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177800ce40 con 0x7f1780100e10 2024-04-26T01:45:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.662+0000 7f1777e00640 1 -- 172.21.15.28:0/524983459 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1778011920 con 0x7f1780100e10 2024-04-26T01:45:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.662+0000 7f1777e00640 1 -- 172.21.15.28:0/524983459 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778010a30 con 0x7f1780100e10 2024-04-26T01:45:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.662+0000 7f1786000640 1 -- 172.21.15.28:0/524983459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17801970c0 con 0x7f1780100e10 2024-04-26T01:45:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.663+0000 7f1786000640 1 -- 172.21.15.28:0/524983459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f178019afb0 con 0x7f1780100e10 2024-04-26T01:45:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.669+0000 7f1788f8f740 1 Processor -- start 2024-04-26T01:45:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.670+0000 7f1777e00640 1 -- 172.21.15.28:0/524983459 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1778010bd0 con 0x7f1780100e10 2024-04-26T01:45:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.671+0000 7f1788f8f740 1 -- start start 2024-04-26T01:45:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.672+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f48820 0x5602c5f48c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.672+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 0x5602c5f534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.672+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f53a00 0x5602c5f55df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.672+0000 7f1788f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602c5eae900 con 0x5602c5f49140 2024-04-26T01:45:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.672+0000 7f1788f8f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5602c5e54ef0 con 0x5602c5f53a00 2024-04-26T01:45:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.672+0000 7f1788f8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5602c5eaa8c0 con 0x5602c5f48820 2024-04-26T01:45:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.673+0000 7f1777e00640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f17640a3e40 0x7f17640a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.673+0000 7f1777e00640 1 -- 172.21.15.28:0/524983459 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f17780d0e20 con 0x7f1780100e10 2024-04-26T01:45:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.674+0000 7f177f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 0x5602c5f534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.674+0000 7f177fe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f48820 0x5602c5f48c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f53a00 0x5602c5f55df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f53a00 0x5602c5f55df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37142/0 (socket says 172.21.15.28:37142) 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 -- 172.21.15.28:0/3395173658 learned_addr learned my addr 172.21.15.28:0/3395173658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.674+0000 7f177fe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f48820 0x5602c5f48c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55274/0 (socket says 172.21.15.28:55274) 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177fe00640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f17640a3e40 0x7f17640a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 -- 172.21.15.28:0/3395173658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f48820 msgr2=0x5602c5f48c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 --2- 172.21.15.28:0/3395173658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f48820 0x5602c5f48c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 -- 172.21.15.28:0/3395173658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 msgr2=0x5602c5f534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 --2- 172.21.15.28:0/3395173658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 0x5602c5f534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 -- 172.21.15.28:0/3395173658 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602c5bc1210 con 0x5602c5f53a00 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.675+0000 7f177ea00640 1 --2- 172.21.15.28:0/3395173658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f53a00 0x5602c5f55df0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f177809caf0 tx=0x7f1778011e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5836ae2b7aeffc28 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.676+0000 7f176be00640 1 -- 172.21.15.28:0/3395173658 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17780a11a0 con 0x5602c5f53a00 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.676+0000 7f176be00640 1 -- 172.21.15.28:0/3395173658 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17780a5a60 con 0x5602c5f53a00 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.676+0000 7f176be00640 1 -- 172.21.15.28:0/3395173658 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778099460 con 0x5602c5f53a00 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.676+0000 7f177fe00640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f17640a3e40 0x7f17640a6300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17700146b0 tx=0x7f1770004730 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.678+0000 7f1788f8f740 1 -- 172.21.15.28:0/3395173658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f53a00 msgr2=0x5602c5f55df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.678+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3395173658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f53a00 0x5602c5f55df0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f177809caf0 tx=0x7f1778011e50 comp rx=0 tx=0).stop 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.678+0000 7f1788f8f740 1 -- 172.21.15.28:0/3395173658 shutdown_connections 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.678+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3395173658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f53a00 0x5602c5f55df0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.678+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3395173658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 0x5602c5f534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.678+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3395173658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f48820 0x5602c5f48c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.678+0000 7f1788f8f740 1 -- 172.21.15.28:0/3395173658 >> 172.21.15.28:0/3395173658 conn(0x5602c5f44d30 msgr2=0x5602c5f45110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.679+0000 7f1788f8f740 1 -- 172.21.15.28:0/3395173658 shutdown_connections 2024-04-26T01:45:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.680+0000 7f1788f8f740 1 -- 172.21.15.28:0/3395173658 wait complete. 2024-04-26T01:45:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.681+0000 7f1788f8f740 1 -- start start 2024-04-26T01:45:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.681+0000 7f1788f8f740 1 Processor -- start 2024-04-26T01:45:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.682+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f509b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.682+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 0x5602c5f50ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.682+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f53a00 0x5602c5f51430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.682+0000 7f1788f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602c5f67640 con 0x5602c5f49140 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.682+0000 7f1788f8f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5602c5e68310 con 0x5602c5f48820 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.682+0000 7f1788f8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5602c5f677c0 con 0x5602c5f53a00 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.682+0000 7f1788f8f740 10 client.0 _ll_register_callbacks cb 0x5602c5e36cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.682+0000 7f177fe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f53a00 0x5602c5f51430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.683+0000 7f177fe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f53a00 0x5602c5f51430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55284/0 (socket says 172.21.15.28:55284) 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.683+0000 7f177fe00640 1 -- 172.21.15.28:0/965339026 learned_addr learned my addr 172.21.15.28:0/965339026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.683+0000 7f177ea00640 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 0x5602c5f50ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.683+0000 7f177fe00640 1 -- 172.21.15.28:0/965339026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 msgr2=0x5602c5f509b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.683+0000 7f177fe00640 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f509b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.683+0000 7f177fe00640 1 -- 172.21.15.28:0/965339026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 msgr2=0x5602c5f50ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.683+0000 7f177fe00640 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 0x5602c5f50ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.683+0000 7f177fe00640 1 -- 172.21.15.28:0/965339026 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602c5c8d620 con 0x5602c5f53a00 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.684+0000 7f177fe00640 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f53a00 0x5602c5f51430 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f177000cea0 tx=0x7f17700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a4339a3be215c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.684+0000 7f1768c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:45:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.685+0000 7f1768c00640 1 -- 172.21.15.28:0/965339026 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770045070 con 0x5602c5f53a00 2024-04-26T01:45:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.685+0000 7f1768c00640 1 -- 172.21.15.28:0/965339026 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f1770011ae0 con 0x5602c5f53a00 2024-04-26T01:45:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.686+0000 7f1788f8f740 1 -- 172.21.15.28:0/965339026 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17780a5a60 con 0x5602c5f53a00 2024-04-26T01:45:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.687+0000 7f1768c00640 1 -- 172.21.15.28:0/965339026 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1770011ce0 con 0x5602c5f53a00 2024-04-26T01:45:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.688+0000 7f1788f8f740 10 client.25747 fetch_fsmap learned FSMap version 32 2024-04-26T01:45:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.689+0000 7f1788f8f740 1 -- 172.21.15.28:0/965339026 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5602c5f73970 con 0x5602c5f53a00 2024-04-26T01:45:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.690+0000 7f1768c00640 1 -- 172.21.15.28:0/965339026 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17780a5a60 con 0x5602c5f53a00 2024-04-26T01:45:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.690+0000 7f1768c00640 1 -- 172.21.15.28:0/965339026 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f177001d050 con 0x5602c5f53a00 2024-04-26T01:45:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.691+0000 7f1788f8f740 10 client.25747 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T01:45:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.691+0000 7f1788f8f740 10 client.25747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:45:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.691+0000 7f1788f8f740 20 client.25747 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.691+0000 7f1788f8f740 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602c5f74e30 0x5602c5f77220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.691+0000 7f1788f8f740 4 client.25747 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.691+0000 7f1788f8f740 1 -- 172.21.15.28:0/965339026 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602c5ec28b0 con 0x5602c5f74e30 2024-04-26T01:45:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.693+0000 7f177f400640 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602c5f74e30 0x5602c5f77220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.696+0000 7f177f400640 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602c5f74e30 0x5602c5f77220 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0ff371d22fd7e4f server_cookie=7dc18cbae4f8284b in_seq=0 out_seq=0 2024-04-26T01:45:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.696+0000 7f1768c00640 10 client.25747 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.698+0000 7f1768c00640 1 -- 172.21.15.28:0/965339026 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602c5ec28b0 con 0x5602c5f74e30 2024-04-26T01:45:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.698+0000 7f1768c00640 10 client.25747 handle_command_reply: tid=0 2024-04-26T01:45:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.698+0000 7f1788f8f740 1 client.25747 shutdown 2024-04-26T01:45:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.723+0000 7f1788f8f740 1 -- 172.21.15.28:0/965339026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f53a00 msgr2=0x5602c5f51430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.723+0000 7f1788f8f740 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f53a00 0x5602c5f51430 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f177000cea0 tx=0x7f17700046c0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.726+0000 7f1788f8f740 1 -- 172.21.15.28:0/965339026 shutdown_connections 2024-04-26T01:45:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.726+0000 7f1788f8f740 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602c5f74e30 0x5602c5f77220 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.726+0000 7f1768c00640 0 client.25747 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.726+0000 7f1788f8f740 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f53a00 0x5602c5f51430 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.726+0000 7f1788f8f740 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f49140 0x5602c5f50ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.726+0000 7f1788f8f740 1 --2- 172.21.15.28:0/965339026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.726+0000 7f1788f8f740 1 -- 172.21.15.28:0/965339026 >> 172.21.15.28:0/965339026 conn(0x5602c5f44d30 msgr2=0x5602c5f4aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.727+0000 7f1788f8f740 1 -- 172.21.15.28:0/965339026 shutdown_connections 2024-04-26T01:45:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.727+0000 7f1788f8f740 1 -- 172.21.15.28:0/965339026 wait complete. 2024-04-26T01:45:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.727+0000 7f1788f8f740 20 client.25747 trim_cache size 0 max 16384 2024-04-26T01:45:52.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.796+0000 7f1788f8f740 1 Processor -- start 2024-04-26T01:45:52.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.796+0000 7f1788f8f740 1 -- start start 2024-04-26T01:45:52.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.796+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f70500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.796+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f4b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.796+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f43d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.796+0000 7f1788f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f178019fc70 con 0x5602c5f53a00 2024-04-26T01:45:52.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.796+0000 7f1788f8f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1780073700 con 0x5602c5f48820 2024-04-26T01:45:52.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.796+0000 7f1788f8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1780071550 con 0x5602c5f49140 2024-04-26T01:45:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.797+0000 7f177ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f4b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.797+0000 7f177ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f4b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55292/0 (socket says 172.21.15.28:55292) 2024-04-26T01:45:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.797+0000 7f177ea00640 1 -- 172.21.15.28:0/2554850669 learned_addr learned my addr 172.21.15.28:0/2554850669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.797+0000 7f177f400640 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f70500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.799+0000 7f177fe00640 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f43d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.800+0000 7f177f400640 1 -- 172.21.15.28:0/2554850669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 msgr2=0x5602c5f4b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:52.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.800+0000 7f177f400640 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f4b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.800+0000 7f177f400640 1 -- 172.21.15.28:0/2554850669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 msgr2=0x5602c5f43d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:52.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.800+0000 7f177f400640 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f43d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.801+0000 7f177f400640 1 -- 172.21.15.28:0/2554850669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602c5f56160 con 0x5602c5f48820 2024-04-26T01:45:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.802+0000 7f177fe00640 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f43d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.802+0000 7f177ea00640 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f4b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:52.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.803+0000 7f177f400640 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f70500 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f176c00dc90 tx=0x7f176c005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77c37deca7f4709a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:52.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.804+0000 7f176b400640 1 -- 172.21.15.28:0/2554850669 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c00d330 con 0x5602c5f48820 2024-04-26T01:45:52.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.804+0000 7f176b400640 1 -- 172.21.15.28:0/2554850669 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f176c005040 con 0x5602c5f48820 2024-04-26T01:45:52.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.804+0000 7f176b400640 1 -- 172.21.15.28:0/2554850669 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c004920 con 0x5602c5f48820 2024-04-26T01:45:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.805+0000 7f1788f8f740 1 -- 172.21.15.28:0/2554850669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 msgr2=0x5602c5f70500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.805+0000 7f1788f8f740 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f70500 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f176c00dc90 tx=0x7f176c005b50 comp rx=0 tx=0).stop 2024-04-26T01:45:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.806+0000 7f1788f8f740 1 -- 172.21.15.28:0/2554850669 shutdown_connections 2024-04-26T01:45:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.806+0000 7f1788f8f740 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f43d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.806+0000 7f1788f8f740 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f4b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.806+0000 7f1788f8f740 1 --2- 172.21.15.28:0/2554850669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f70500 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.806+0000 7f1788f8f740 1 -- 172.21.15.28:0/2554850669 >> 172.21.15.28:0/2554850669 conn(0x5602c5f44d30 msgr2=0x5602c5f43600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:52.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.807+0000 7f1788f8f740 1 -- 172.21.15.28:0/2554850669 shutdown_connections 2024-04-26T01:45:52.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.807+0000 7f1788f8f740 1 -- 172.21.15.28:0/2554850669 wait complete. 2024-04-26T01:45:52.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.808+0000 7f1788f8f740 1 -- start start 2024-04-26T01:45:52.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.808+0000 7f1788f8f740 1 Processor -- start 2024-04-26T01:45:52.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.811+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f6ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.811+0000 7f177f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f6ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.812+0000 7f177f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f6ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37158/0 (socket says 172.21.15.28:37158) 2024-04-26T01:45:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f6a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f1788f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f6a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f1788f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1780071bc0 con 0x5602c5f53a00 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f1788f8f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1780073030 con 0x5602c5f48820 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f1788f8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1780072290 con 0x5602c5f49140 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f1788f8f740 10 client.0 _ll_register_callbacks cb 0x7f177000f750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f177ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f6a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f177ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f6a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55308/0 (socket says 172.21.15.28:55308) 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f177ea00640 1 -- 172.21.15.28:0/3432285854 learned_addr learned my addr 172.21.15.28:0/3432285854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.813+0000 7f177fe00640 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f6a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.814+0000 7f177ea00640 1 -- 172.21.15.28:0/3432285854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 msgr2=0x5602c5f6ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.814+0000 7f177ea00640 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f6ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.814+0000 7f177ea00640 1 -- 172.21.15.28:0/3432285854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 msgr2=0x5602c5f6a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:52.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.814+0000 7f177ea00640 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f6a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.814+0000 7f177ea00640 1 -- 172.21.15.28:0/3432285854 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602c5f6ad00 con 0x5602c5f49140 2024-04-26T01:45:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.815+0000 7f177ea00640 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f6a080 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1778011f00 tx=0x7f177809ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=946746643ee22a52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.818+0000 7f1768c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:45:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.819+0000 7f1768c00640 1 -- 172.21.15.28:0/3432285854 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778080060 con 0x5602c5f49140 2024-04-26T01:45:52.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.819+0000 7f1768c00640 1 -- 172.21.15.28:0/3432285854 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f17780b6240 con 0x5602c5f49140 2024-04-26T01:45:52.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.819+0000 7f1788f8f740 1 -- 172.21.15.28:0/3432285854 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f176c005040 con 0x5602c5f49140 2024-04-26T01:45:52.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.820+0000 7f1768c00640 1 -- 172.21.15.28:0/3432285854 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f17780a11a0 con 0x5602c5f49140 2024-04-26T01:45:52.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.820+0000 7f1788f8f740 10 client.25753 fetch_fsmap learned FSMap version 32 2024-04-26T01:45:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.820+0000 7f1788f8f740 1 -- 172.21.15.28:0/3432285854 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5602c5fdb240 con 0x5602c5f49140 2024-04-26T01:45:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.821+0000 7f1768c00640 1 -- 172.21.15.28:0/3432285854 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f176c005040 con 0x5602c5f49140 2024-04-26T01:45:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.821+0000 7f1768c00640 1 -- 172.21.15.28:0/3432285854 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f177800f040 con 0x5602c5f49140 2024-04-26T01:45:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.821+0000 7f1788f8f740 10 client.25753 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T01:45:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.821+0000 7f1788f8f740 10 client.25753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:45:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.821+0000 7f1788f8f740 20 client.25753 populate_metadata read hostname 'smithi028' 2024-04-26T01:45:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.821+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602c5f74e30 0x5602c5fdd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.821+0000 7f1788f8f740 4 client.25753 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:45:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.821+0000 7f1788f8f740 1 -- 172.21.15.28:0/3432285854 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602c5ebc320 con 0x5602c5f74e30 2024-04-26T01:45:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.822+0000 7f177f400640 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602c5f74e30 0x5602c5fdd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.823+0000 7f177f400640 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602c5f74e30 0x5602c5fdd590 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e05a3acf0642da3 server_cookie=2e1bbcb785972623 in_seq=0 out_seq=0 2024-04-26T01:45:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.823+0000 7f1768c00640 10 client.25753 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.967+0000 7f1768c00640 1 -- 172.21.15.28:0/3432285854 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602c5ebc320 con 0x5602c5f74e30 2024-04-26T01:45:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.967+0000 7f1768c00640 10 client.25753 handle_command_reply: tid=0 2024-04-26T01:45:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.968+0000 7f1788f8f740 1 client.25753 shutdown 2024-04-26T01:45:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.968+0000 7f1788f8f740 1 -- 172.21.15.28:0/3432285854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 msgr2=0x5602c5f6a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.968+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f6a080 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1778011f00 tx=0x7f177809ceb0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.968+0000 7f1788f8f740 1 -- 172.21.15.28:0/3432285854 shutdown_connections 2024-04-26T01:45:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.969+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5602c5f74e30 0x5602c5fdd590 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.969+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602c5f53a00 0x5602c5f6a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.969+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602c5f49140 0x5602c5f6a080 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.969+0000 7f1788f8f740 1 --2- 172.21.15.28:0/3432285854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5602c5f48820 0x5602c5f6ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.969+0000 7f1788f8f740 1 -- 172.21.15.28:0/3432285854 >> 172.21.15.28:0/3432285854 conn(0x5602c5f44d30 msgr2=0x5602c5f51200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.969+0000 7f1768c00640 0 client.25753 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:45:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.969+0000 7f1788f8f740 1 -- 172.21.15.28:0/3432285854 shutdown_connections 2024-04-26T01:45:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.969+0000 7f1788f8f740 1 -- 172.21.15.28:0/3432285854 wait complete. 2024-04-26T01:45:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.970+0000 7f1788f8f740 20 client.25753 trim_cache size 0 max 16384 2024-04-26T01:45:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:45:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:45:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:45:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:45:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 -- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f17640a3e40 msgr2=0x7f17640a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f17640a3e40 0x7f17640a6300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17700146b0 tx=0x7f1770004730 comp rx=0 tx=0).stop 2024-04-26T01:45:52.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 -- 172.21.15.28:0/524983459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1780100e10 msgr2=0x7f178019d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1780100e10 0x7f178019d7f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f177800eab0 tx=0x7f177800ef70 comp rx=0 tx=0).stop 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 -- 172.21.15.28:0/524983459 shutdown_connections 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f17640a3e40 0x7f17640a6300 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1780106f30 0x7f1780196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1780100e10 0x7f178019d7f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 --2- 172.21.15.28:0/524983459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1780100450 0x7f178019d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 -- 172.21.15.28:0/524983459 >> 172.21.15.28:0/524983459 conn(0x7f17800773e0 msgr2=0x7f17800fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 -- 172.21.15.28:0/524983459 shutdown_connections 2024-04-26T01:45:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:45:52.971+0000 7f1786000640 1 -- 172.21.15.28:0/524983459 wait complete. 2024-04-26T01:45:53.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:53 smithi169 ceph-mon[32851]: pgmap v150: 97 pgs: 97 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.1 MiB/s wr, 605 op/s 2024-04-26T01:45:53.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:53 smithi053 ceph-mon[32593]: pgmap v150: 97 pgs: 97 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.1 MiB/s wr, 605 op/s 2024-04-26T01:45:53.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:53 smithi028 ceph-mon[28247]: pgmap v150: 97 pgs: 97 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.1 MiB/s wr, 605 op/s 2024-04-26T01:45:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:55 smithi169 ceph-mon[32851]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 37 MiB/s wr, 1.89k op/s 2024-04-26T01:45:55.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:55 smithi053 ceph-mon[32593]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 37 MiB/s wr, 1.89k op/s 2024-04-26T01:45:55.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:55 smithi028 ceph-mon[28247]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 37 MiB/s wr, 1.89k op/s 2024-04-26T01:45:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 240.0 s, 2314.9 tps, lat 43.237 ms stddev 120.885 2024-04-26T01:45:57.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:57 smithi169 ceph-mon[32851]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 MiB/s wr, 1.58k op/s 2024-04-26T01:45:57.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:57 smithi053 ceph-mon[32593]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 MiB/s wr, 1.58k op/s 2024-04-26T01:45:57.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:57 smithi028 ceph-mon[28247]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 MiB/s wr, 1.58k op/s 2024-04-26T01:45:59.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:45:59 smithi169 ceph-mon[32851]: pgmap v153: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 MiB/s wr, 1.58k op/s 2024-04-26T01:45:59.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:45:59 smithi053 ceph-mon[32593]: pgmap v153: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 MiB/s wr, 1.58k op/s 2024-04-26T01:45:59.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:45:59 smithi028 ceph-mon[28247]: pgmap v153: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 MiB/s wr, 1.58k op/s 2024-04-26T01:46:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 245.0 s, 2404.9 tps, lat 39.554 ms stddev 60.231 2024-04-26T01:46:01.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:01 smithi169 ceph-mon[32851]: pgmap v154: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 1.87k op/s 2024-04-26T01:46:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:01 smithi053 ceph-mon[32593]: pgmap v154: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 1.87k op/s 2024-04-26T01:46:01.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:01 smithi028 ceph-mon[28247]: pgmap v154: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 1.87k op/s 2024-04-26T01:46:03.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:03 smithi169 ceph-mon[32851]: pgmap v155: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 1.57k op/s 2024-04-26T01:46:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:03 smithi053 ceph-mon[32593]: pgmap v155: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 1.57k op/s 2024-04-26T01:46:03.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:03 smithi028 ceph-mon[28247]: pgmap v155: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 1.57k op/s 2024-04-26T01:46:05.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:05 smithi028 ceph-mon[28247]: pgmap v156: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 45 MiB/s wr, 2.71k op/s 2024-04-26T01:46:05.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:05 smithi169 ceph-mon[32851]: pgmap v156: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 45 MiB/s wr, 2.71k op/s 2024-04-26T01:46:05.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:05 smithi053 ceph-mon[32593]: pgmap v156: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 45 MiB/s wr, 2.71k op/s 2024-04-26T01:46:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.167+0000 7ff907e00640 1 -- 172.21.15.28:0/2335714193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9081014a0 msgr2=0x7ff90810e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.167+0000 7ff907e00640 1 --2- 172.21.15.28:0/2335714193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9081014a0 0x7ff90810e160 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009900 tx=0x7ff8f402a400 comp rx=0 tx=0).stop 2024-04-26T01:46:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.167+0000 7ff907e00640 1 -- 172.21.15.28:0/2335714193 shutdown_connections 2024-04-26T01:46:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.167+0000 7ff907e00640 1 --2- 172.21.15.28:0/2335714193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90810e6a0 0x7ff908110a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.167+0000 7ff907e00640 1 --2- 172.21.15.28:0/2335714193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9081014a0 0x7ff90810e160 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.167+0000 7ff907e00640 1 --2- 172.21.15.28:0/2335714193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff908100af0 0x7ff908100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.167+0000 7ff907e00640 1 -- 172.21.15.28:0/2335714193 >> 172.21.15.28:0/2335714193 conn(0x7ff9080fc770 msgr2=0x7ff9080feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.167+0000 7ff907e00640 1 -- 172.21.15.28:0/2335714193 shutdown_connections 2024-04-26T01:46:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.168+0000 7ff907e00640 1 -- 172.21.15.28:0/2335714193 wait complete. 2024-04-26T01:46:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.169+0000 7ff907e00640 1 Processor -- start 2024-04-26T01:46:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.169+0000 7ff907e00640 1 -- start start 2024-04-26T01:46:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff907e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff908100af0 0x7ff9081a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff907e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081014a0 0x7ff9081a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff907e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff90810e6a0 0x7ff90819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff907e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff908114290 con 0x7ff9081014a0 2024-04-26T01:46:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff907e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff908114110 con 0x7ff908100af0 2024-04-26T01:46:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff907e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff908114410 con 0x7ff90810e6a0 2024-04-26T01:46:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff906a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff908100af0 0x7ff9081a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff906a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff908100af0 0x7ff9081a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46862/0 (socket says 172.21.15.28:46862) 2024-04-26T01:46:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff906a00640 1 -- 172.21.15.28:0/1720615012 learned_addr learned my addr 172.21.15.28:0/1720615012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.170+0000 7ff906000640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081014a0 0x7ff9081a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.171+0000 7ff907400640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff90810e6a0 0x7ff90819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.171+0000 7ff906a00640 1 -- 172.21.15.28:0/1720615012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff90810e6a0 msgr2=0x7ff90819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.171+0000 7ff906a00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff90810e6a0 0x7ff90819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.171+0000 7ff906a00640 1 -- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081014a0 msgr2=0x7ff9081a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.171+0000 7ff906a00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081014a0 0x7ff9081a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.171+0000 7ff906a00640 1 -- 172.21.15.28:0/1720615012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff90819ad00 con 0x7ff908100af0 2024-04-26T01:46:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.171+0000 7ff906a00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff908100af0 0x7ff9081a0400 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800c950 tx=0x7ff8f800ce10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.171+0000 7ff8ff400640 1 -- 172.21.15.28:0/1720615012 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8007c70 con 0x7ff908100af0 2024-04-26T01:46:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.172+0000 7ff8ff400640 1 -- 172.21.15.28:0/1720615012 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f8007e10 con 0x7ff908100af0 2024-04-26T01:46:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.172+0000 7ff8ff400640 1 -- 172.21.15.28:0/1720615012 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80187f0 con 0x7ff908100af0 2024-04-26T01:46:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.172+0000 7ff907e00640 1 -- 172.21.15.28:0/1720615012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff90819aff0 con 0x7ff908100af0 2024-04-26T01:46:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.172+0000 7ff907e00640 1 -- 172.21.15.28:0/1720615012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff90806b970 con 0x7ff908100af0 2024-04-26T01:46:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.178+0000 7ff91063e740 1 Processor -- start 2024-04-26T01:46:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.179+0000 7ff91063e740 1 -- start start 2024-04-26T01:46:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.180+0000 7ff91063e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb82d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.181+0000 7ff91063e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.181+0000 7ff91063e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadb8ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.181+0000 7ff91063e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fdadae8900 con 0x55fdadb829b0 2024-04-26T01:46:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.181+0000 7ff91063e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fdada8eef0 con 0x55fdadb832d0 2024-04-26T01:46:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.181+0000 7ff91063e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55fdadae48c0 con 0x55fdadb8db90 2024-04-26T01:46:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.181+0000 7ff907400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb82d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff906000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadb8ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff906000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadb8ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51038/0 (socket says 172.21.15.28:51038) 2024-04-26T01:46:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff906000640 1 -- 172.21.15.28:0/1936774186 learned_addr learned my addr 172.21.15.28:0/1936774186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff906000640 1 -- 172.21.15.28:0/1936774186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 msgr2=0x55fdadb8d650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:46:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff906000640 1 --2- 172.21.15.28:0/1936774186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff906000640 1 -- 172.21.15.28:0/1936774186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 msgr2=0x55fdadb82d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff906000640 1 --2- 172.21.15.28:0/1936774186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb82d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff906000640 1 -- 172.21.15.28:0/1936774186 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdad7fb210 con 0x55fdadb8db90 2024-04-26T01:46:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.182+0000 7ff907400640 1 --2- 172.21.15.28:0/1936774186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb82d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.183+0000 7ff906000640 1 --2- 172.21.15.28:0/1936774186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadb8ff80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4031440 tx=0x7ff8f4004380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b298b3dbc446e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.184+0000 7ff8ff400640 1 -- 172.21.15.28:0/1720615012 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff8f8018990 con 0x7ff908100af0 2024-04-26T01:46:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.185+0000 7ff8f3e00640 1 -- 172.21.15.28:0/1936774186 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f403f070 con 0x55fdadb8db90 2024-04-26T01:46:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.185+0000 7ff8f3e00640 1 -- 172.21.15.28:0/1936774186 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f40047a0 con 0x55fdadb8db90 2024-04-26T01:46:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.185+0000 7ff8f3e00640 1 -- 172.21.15.28:0/1936774186 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4035070 con 0x55fdadb8db90 2024-04-26T01:46:06.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.186+0000 7ff8ff400640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff8ec0a3de0 0x7ff8ec0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.186+0000 7ff8ff400640 1 -- 172.21.15.28:0/1720615012 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7ff8f80d0950 con 0x7ff908100af0 2024-04-26T01:46:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.187+0000 7ff91063e740 1 -- 172.21.15.28:0/1936774186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 msgr2=0x55fdadb8ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.187+0000 7ff91063e740 1 --2- 172.21.15.28:0/1936774186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadb8ff80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4031440 tx=0x7ff8f4004380 comp rx=0 tx=0).stop 2024-04-26T01:46:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.187+0000 7ff907400640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff8ec0a3de0 0x7ff8ec0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.187+0000 7ff91063e740 1 -- 172.21.15.28:0/1936774186 shutdown_connections 2024-04-26T01:46:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.188+0000 7ff91063e740 1 --2- 172.21.15.28:0/1936774186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadb8ff80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.188+0000 7ff91063e740 1 --2- 172.21.15.28:0/1936774186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.188+0000 7ff91063e740 1 --2- 172.21.15.28:0/1936774186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb82d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.189+0000 7ff91063e740 1 -- 172.21.15.28:0/1936774186 >> 172.21.15.28:0/1936774186 conn(0x55fdadb7eec0 msgr2=0x55fdadb7f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.190+0000 7ff91063e740 1 -- 172.21.15.28:0/1936774186 shutdown_connections 2024-04-26T01:46:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.190+0000 7ff907400640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff8ec0a3de0 0x7ff8ec0a62a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff908102cd0 tx=0x7ff90000e040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.190+0000 7ff91063e740 1 -- 172.21.15.28:0/1936774186 wait complete. 2024-04-26T01:46:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.191+0000 7ff91063e740 1 -- start start 2024-04-26T01:46:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.191+0000 7ff91063e740 1 Processor -- start 2024-04-26T01:46:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff91063e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb829b0 0x55fdadb8ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff91063e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb832d0 0x55fdadb8b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff91063e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadb8b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff91063e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fdadb90760 con 0x55fdadb8db90 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff91063e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fdadaa2310 con 0x55fdadb829b0 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff91063e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55fdadb908e0 con 0x55fdadb832d0 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff91063e740 10 client.0 _ll_register_callbacks cb 0x55fdada70cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff906a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb829b0 0x55fdadb8ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.194+0000 7ff906a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb829b0 0x55fdadb8ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46866/0 (socket says 172.21.15.28:46866) 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.195+0000 7ff906a00640 1 -- 172.21.15.28:0/1068315959 learned_addr learned my addr 172.21.15.28:0/1068315959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.195+0000 7ff906000640 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb832d0 0x55fdadb8b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.195+0000 7ff907400640 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadb8b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.196+0000 7ff906000640 1 -- 172.21.15.28:0/1068315959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb829b0 msgr2=0x55fdadb8ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.196+0000 7ff906000640 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb829b0 0x55fdadb8ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.196+0000 7ff906000640 1 -- 172.21.15.28:0/1068315959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 msgr2=0x55fdadb8b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.196+0000 7ff906000640 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadb8b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.196+0000 7ff906000640 1 -- 172.21.15.28:0/1068315959 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdad8c7620 con 0x55fdadb832d0 2024-04-26T01:46:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.196+0000 7ff907400640 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadb8b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.197+0000 7ff906000640 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb832d0 0x55fdadb8b050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4005ab0 tx=0x7ff8f4005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bad5d284ec8bb648 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.197+0000 7ff8f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:46:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.198+0000 7ff8f0c00640 1 -- 172.21.15.28:0/1068315959 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4035030 con 0x55fdadb832d0 2024-04-26T01:46:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.198+0000 7ff8f0c00640 1 -- 172.21.15.28:0/1068315959 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7ff8f40316a0 con 0x55fdadb832d0 2024-04-26T01:46:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.198+0000 7ff91063e740 1 -- 172.21.15.28:0/1068315959 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8f40047a0 con 0x55fdadb832d0 2024-04-26T01:46:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.199+0000 7ff8f0c00640 1 -- 172.21.15.28:0/1068315959 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8f403f040 con 0x55fdadb832d0 2024-04-26T01:46:06.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.199+0000 7ff91063e740 10 client.25762 fetch_fsmap learned FSMap version 32 2024-04-26T01:46:06.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.199+0000 7ff91063e740 1 -- 172.21.15.28:0/1068315959 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fdadb90760 con 0x55fdadb832d0 2024-04-26T01:46:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.200+0000 7ff8f0c00640 1 -- 172.21.15.28:0/1068315959 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f40047a0 con 0x55fdadb832d0 2024-04-26T01:46:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.200+0000 7ff8f0c00640 1 -- 172.21.15.28:0/1068315959 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7ff8f404c6a0 con 0x55fdadb832d0 2024-04-26T01:46:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.200+0000 7ff91063e740 10 client.25762 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T01:46:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.200+0000 7ff91063e740 10 client.25762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.201+0000 7ff91063e740 20 client.25762 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.201+0000 7ff91063e740 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fdadba63c0 0x55fdadba87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.201+0000 7ff91063e740 4 client.25762 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.201+0000 7ff91063e740 1 -- 172.21.15.28:0/1068315959 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdadafc8b0 con 0x55fdadba63c0 2024-04-26T01:46:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.201+0000 7ff906a00640 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fdadba63c0 0x55fdadba87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.203+0000 7ff906a00640 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fdadba63c0 0x55fdadba87b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=695a0ced08a6b7d1 server_cookie=330965d90fd63360 in_seq=0 out_seq=0 2024-04-26T01:46:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.203+0000 7ff8f0c00640 10 client.25762 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.203+0000 7ff8f0c00640 1 -- 172.21.15.28:0/1068315959 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdadafc8b0 con 0x55fdadba63c0 2024-04-26T01:46:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.203+0000 7ff8f0c00640 10 client.25762 handle_command_reply: tid=0 2024-04-26T01:46:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 client.25762 shutdown 2024-04-26T01:46:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 -- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb832d0 msgr2=0x55fdadb8b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb832d0 0x55fdadb8b050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4005ab0 tx=0x7ff8f4005230 comp rx=0 tx=0).stop 2024-04-26T01:46:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 -- 172.21.15.28:0/1068315959 shutdown_connections 2024-04-26T01:46:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fdadba63c0 0x55fdadba87b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadb8b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb832d0 0x55fdadb8b050 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 --2- 172.21.15.28:0/1068315959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb829b0 0x55fdadb8ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff91063e740 1 -- 172.21.15.28:0/1068315959 >> 172.21.15.28:0/1068315959 conn(0x55fdadb7eec0 msgr2=0x55fdadb84c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.204+0000 7ff8f0c00640 0 client.25762 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.205+0000 7ff91063e740 1 -- 172.21.15.28:0/1068315959 shutdown_connections 2024-04-26T01:46:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.205+0000 7ff91063e740 1 -- 172.21.15.28:0/1068315959 wait complete. 2024-04-26T01:46:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.206+0000 7ff91063e740 20 client.25762 trim_cache size 0 max 16384 2024-04-26T01:46:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 250.0 s, 2583.0 tps, lat 40.434 ms stddev 61.585 2024-04-26T01:46:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.235+0000 7ff91063e740 1 Processor -- start 2024-04-26T01:46:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.235+0000 7ff91063e740 1 -- start start 2024-04-26T01:46:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff91063e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb829b0 0x55fdadb85370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff91063e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff91063e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadbaddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff91063e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9080727c0 con 0x55fdadb8db90 2024-04-26T01:46:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff91063e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff908070030 con 0x55fdadb832d0 2024-04-26T01:46:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff91063e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff90806de80 con 0x55fdadb829b0 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff906a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb829b0 0x55fdadb85370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff906000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff907400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadbaddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff906000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46878/0 (socket says 172.21.15.28:46878) 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff906000640 1 -- 172.21.15.28:0/1252888605 learned_addr learned my addr 172.21.15.28:0/1252888605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.236+0000 7ff906a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb829b0 0x55fdadb85370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51054/0 (socket says 172.21.15.28:51054) 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.237+0000 7ff906000640 1 -- 172.21.15.28:0/1252888605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb829b0 msgr2=0x55fdadb85370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.237+0000 7ff906000640 1 --2- 172.21.15.28:0/1252888605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb829b0 0x55fdadb85370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.237+0000 7ff906000640 1 -- 172.21.15.28:0/1252888605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 msgr2=0x55fdadbaddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.237+0000 7ff906000640 1 --2- 172.21.15.28:0/1252888605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadbaddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.237+0000 7ff906000640 1 -- 172.21.15.28:0/1252888605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdadb89060 con 0x55fdadb832d0 2024-04-26T01:46:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.237+0000 7ff906000640 1 --2- 172.21.15.28:0/1252888605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8b3b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff8f404cbe0 tx=0x7ff8f4004380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdf0af7d91630472 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.238+0000 7ff8f3400640 1 -- 172.21.15.28:0/1252888605 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4059030 con 0x55fdadb832d0 2024-04-26T01:46:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.238+0000 7ff8f3400640 1 -- 172.21.15.28:0/1252888605 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f40047a0 con 0x55fdadb832d0 2024-04-26T01:46:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.238+0000 7ff8f3400640 1 -- 172.21.15.28:0/1252888605 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4035070 con 0x55fdadb832d0 2024-04-26T01:46:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.239+0000 7ff91063e740 1 -- 172.21.15.28:0/1252888605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 msgr2=0x55fdadb8b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.239+0000 7ff91063e740 1 --2- 172.21.15.28:0/1252888605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8b3b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff8f404cbe0 tx=0x7ff8f4004380 comp rx=0 tx=0).stop 2024-04-26T01:46:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.239+0000 7ff91063e740 1 -- 172.21.15.28:0/1252888605 shutdown_connections 2024-04-26T01:46:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.239+0000 7ff91063e740 1 --2- 172.21.15.28:0/1252888605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb8db90 0x55fdadbaddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.239+0000 7ff91063e740 1 --2- 172.21.15.28:0/1252888605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb8b3b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.239+0000 7ff91063e740 1 --2- 172.21.15.28:0/1252888605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb829b0 0x55fdadb85370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.239+0000 7ff91063e740 1 -- 172.21.15.28:0/1252888605 >> 172.21.15.28:0/1252888605 conn(0x55fdadb7eec0 msgr2=0x55fdadb7e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.239+0000 7ff91063e740 1 -- 172.21.15.28:0/1252888605 shutdown_connections 2024-04-26T01:46:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.240+0000 7ff91063e740 1 -- 172.21.15.28:0/1252888605 wait complete. 2024-04-26T01:46:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.240+0000 7ff91063e740 1 -- start start 2024-04-26T01:46:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.240+0000 7ff91063e740 1 Processor -- start 2024-04-26T01:46:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.241+0000 7ff91063e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb95170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.241+0000 7ff91063e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb956b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.241+0000 7ff91063e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadc04fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.241+0000 7ff906000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.241+0000 7ff906000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46892/0 (socket says 172.21.15.28:46892) 2024-04-26T01:46:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff906000640 1 -- 172.21.15.28:0/2743376236 learned_addr learned my addr 172.21.15.28:0/2743376236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff90806e4f0 con 0x55fdadb829b0 2024-04-26T01:46:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff90806f960 con 0x55fdadb832d0 2024-04-26T01:46:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff90806ebc0 con 0x55fdadb8db90 2024-04-26T01:46:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff91063e740 10 client.0 _ll_register_callbacks cb 0x7ff8f4004dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff907400640 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadc04fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff906a00640 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb95170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff906000640 1 -- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 msgr2=0x55fdadc04fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff906000640 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadc04fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff906000640 1 -- 172.21.15.28:0/2743376236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 msgr2=0x55fdadb95170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff906000640 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb95170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.242+0000 7ff906000640 1 -- 172.21.15.28:0/2743376236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdadc05690 con 0x55fdadb832d0 2024-04-26T01:46:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.243+0000 7ff907400640 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadc04fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.243+0000 7ff906000640 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb956b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40092a0 tx=0x7ff8f4005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1820a3e2ad4ce305 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.243+0000 7ff8f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:46:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.244+0000 7ff8f0c00640 1 -- 172.21.15.28:0/2743376236 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4035030 con 0x55fdadb832d0 2024-04-26T01:46:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.244+0000 7ff8f0c00640 1 -- 172.21.15.28:0/2743376236 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7ff8f404c630 con 0x55fdadb832d0 2024-04-26T01:46:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.244+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8f40047a0 con 0x55fdadb832d0 2024-04-26T01:46:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.245+0000 7ff8f0c00640 1 -- 172.21.15.28:0/2743376236 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8f4059040 con 0x55fdadb832d0 2024-04-26T01:46:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.245+0000 7ff91063e740 10 client.25751 fetch_fsmap learned FSMap version 32 2024-04-26T01:46:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.245+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fdadb9c940 con 0x55fdadb832d0 2024-04-26T01:46:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.247+0000 7ff8f0c00640 1 -- 172.21.15.28:0/2743376236 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f40047a0 con 0x55fdadb832d0 2024-04-26T01:46:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.247+0000 7ff8f0c00640 1 -- 172.21.15.28:0/2743376236 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7ff8f40314a0 con 0x55fdadb832d0 2024-04-26T01:46:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.247+0000 7ff91063e740 10 client.25751 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T01:46:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.247+0000 7ff91063e740 10 client.25751 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.247+0000 7ff91063e740 20 client.25751 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.248+0000 7ff91063e740 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fdadb9ce30 0x55fdadb9d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.248+0000 7ff91063e740 4 client.25751 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.248+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fdadaf6320 con 0x55fdadb9ce30 2024-04-26T01:46:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.248+0000 7ff906a00640 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fdadb9ce30 0x55fdadb9d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.249+0000 7ff906a00640 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fdadb9ce30 0x55fdadb9d240 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d40c6534a36288ac server_cookie=647f5f913aba823e in_seq=0 out_seq=0 2024-04-26T01:46:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.249+0000 7ff8f0c00640 10 client.25751 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.290+0000 7ff8f0c00640 1 -- 172.21.15.28:0/2743376236 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fdadaf6320 con 0x55fdadb9ce30 2024-04-26T01:46:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.291+0000 7ff8f0c00640 10 client.25751 handle_command_reply: tid=0 2024-04-26T01:46:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.291+0000 7ff91063e740 1 client.25751 shutdown 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.292+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 msgr2=0x55fdadb956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.292+0000 7ff91063e740 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb956b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40092a0 tx=0x7ff8f4005be0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 shutdown_connections 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fdadb9ce30 0x55fdadb9d240 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff8f0c00640 0 client.25751 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fdadb8db90 0x55fdadc04fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fdadb832d0 0x55fdadb956b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 1 --2- 172.21.15.28:0/2743376236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdadb829b0 0x55fdadb95170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 >> 172.21.15.28:0/2743376236 conn(0x55fdadb7eec0 msgr2=0x55fdadb8c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 shutdown_connections 2024-04-26T01:46:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 1 -- 172.21.15.28:0/2743376236 wait complete. 2024-04-26T01:46:06.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.293+0000 7ff91063e740 20 client.25751 trim_cache size 0 max 16384 2024-04-26T01:46:06.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:46:06.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:46:06.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:46:06.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 -- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff8ec0a3de0 msgr2=0x7ff8ec0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff8ec0a3de0 0x7ff8ec0a62a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff908102cd0 tx=0x7ff90000e040 comp rx=0 tx=0).stop 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 -- 172.21.15.28:0/1720615012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff908100af0 msgr2=0x7ff9081a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff908100af0 0x7ff9081a0400 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800c950 tx=0x7ff8f800ce10 comp rx=0 tx=0).stop 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 -- 172.21.15.28:0/1720615012 shutdown_connections 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff8ec0a3de0 0x7ff8ec0a62a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff90810e6a0 0x7ff90819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081014a0 0x7ff9081a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 --2- 172.21.15.28:0/1720615012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff908100af0 0x7ff9081a0400 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.294+0000 7ff907e00640 1 -- 172.21.15.28:0/1720615012 >> 172.21.15.28:0/1720615012 conn(0x7ff9080fc770 msgr2=0x7ff90810f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.295+0000 7ff907e00640 1 -- 172.21.15.28:0/1720615012 shutdown_connections 2024-04-26T01:46:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:06.295+0000 7ff907e00640 1 -- 172.21.15.28:0/1720615012 wait complete. 2024-04-26T01:46:07.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:07 smithi169 ceph-mon[32851]: pgmap v157: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 1.43k op/s 2024-04-26T01:46:07.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:07 smithi053 ceph-mon[32593]: pgmap v157: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 1.43k op/s 2024-04-26T01:46:07.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:07 smithi028 ceph-mon[28247]: pgmap v157: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 1.43k op/s 2024-04-26T01:46:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:09 smithi169 ceph-mon[32851]: pgmap v158: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 1.43k op/s 2024-04-26T01:46:09.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:09 smithi053 ceph-mon[32593]: pgmap v158: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 1.43k op/s 2024-04-26T01:46:09.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:09 smithi028 ceph-mon[28247]: pgmap v158: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 1.43k op/s 2024-04-26T01:46:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 255.0 s, 2786.2 tps, lat 35.875 ms stddev 40.306 2024-04-26T01:46:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:11 smithi169 ceph-mon[32851]: pgmap v159: 97 pgs: 97 active+clean; 2.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 19 MiB/s wr, 1.77k op/s 2024-04-26T01:46:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:11 smithi053 ceph-mon[32593]: pgmap v159: 97 pgs: 97 active+clean; 2.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 19 MiB/s wr, 1.77k op/s 2024-04-26T01:46:11.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:11 smithi028 ceph-mon[28247]: pgmap v159: 97 pgs: 97 active+clean; 2.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 19 MiB/s wr, 1.77k op/s 2024-04-26T01:46:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:13 smithi169 ceph-mon[32851]: pgmap v160: 97 pgs: 97 active+clean; 2.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 16 MiB/s wr, 1.48k op/s 2024-04-26T01:46:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:13 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:13 smithi053 ceph-mon[32593]: pgmap v160: 97 pgs: 97 active+clean; 2.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 16 MiB/s wr, 1.48k op/s 2024-04-26T01:46:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:13 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:13 smithi028 ceph-mon[28247]: pgmap v160: 97 pgs: 97 active+clean; 2.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 16 MiB/s wr, 1.48k op/s 2024-04-26T01:46:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:13 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:15 smithi169 ceph-mon[32851]: pgmap v161: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 18 MiB/s wr, 1.74k op/s 2024-04-26T01:46:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:15 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:15 smithi053 ceph-mon[32593]: pgmap v161: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 18 MiB/s wr, 1.74k op/s 2024-04-26T01:46:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:15 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:15.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:15 smithi028 ceph-mon[28247]: pgmap v161: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 18 MiB/s wr, 1.74k op/s 2024-04-26T01:46:15.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:15 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 260.0 s, 3033.2 tps, lat 33.025 ms stddev 39.287 2024-04-26T01:46:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:17 smithi169 ceph-mon[32851]: pgmap v162: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 713 op/s 2024-04-26T01:46:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:17 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:17 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:17 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:46:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:17 smithi053 ceph-mon[32593]: pgmap v162: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 713 op/s 2024-04-26T01:46:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:17 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:17 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:17 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:46:17.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:17 smithi028 ceph-mon[28247]: pgmap v162: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 713 op/s 2024-04-26T01:46:17.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:17 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:17.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:17 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:17.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:17 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:46:18.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:18 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:18.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:18 smithi053 ceph-mon[32593]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 713 op/s 2024-04-26T01:46:19.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:18 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:19.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:18 smithi028 ceph-mon[28247]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 713 op/s 2024-04-26T01:46:19.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:18 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:19.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:18 smithi169 ceph-mon[32851]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 713 op/s 2024-04-26T01:46:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.494+0000 7f31c8600640 1 -- 172.21.15.28:0/4251738914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0069ca0 msgr2=0x7f31c0108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.494+0000 7f31c8600640 1 --2- 172.21.15.28:0/4251738914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0069ca0 0x7f31c0108f50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009930 tx=0x7f31ac02a460 comp rx=0 tx=0).stop 2024-04-26T01:46:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.494+0000 7f31c8600640 1 -- 172.21.15.28:0/4251738914 shutdown_connections 2024-04-26T01:46:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.494+0000 7f31c8600640 1 --2- 172.21.15.28:0/4251738914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0109490 0x7f31c0109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.494+0000 7f31c8600640 1 --2- 172.21.15.28:0/4251738914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0069ca0 0x7f31c0108f50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.494+0000 7f31c8600640 1 --2- 172.21.15.28:0/4251738914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31c010e980 0x7f31c010ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.494+0000 7f31c8600640 1 -- 172.21.15.28:0/4251738914 >> 172.21.15.28:0/4251738914 conn(0x7f31c006d4b0 msgr2=0x7f31c006d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.495+0000 7f31c8600640 1 -- 172.21.15.28:0/4251738914 shutdown_connections 2024-04-26T01:46:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.495+0000 7f31c8600640 1 -- 172.21.15.28:0/4251738914 wait complete. 2024-04-26T01:46:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.496+0000 7f31c8600640 1 Processor -- start 2024-04-26T01:46:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.496+0000 7f31c8600640 1 -- start start 2024-04-26T01:46:19.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c8600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0069ca0 0x7f31c011a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c8600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0109490 0x7f31c011aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c8600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31c010e980 0x7f31c01b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c8600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f31c0118910 con 0x7f31c010e980 2024-04-26T01:46:19.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c8600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f31c0118790 con 0x7f31c0109490 2024-04-26T01:46:19.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c8600640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f31c0118a90 con 0x7f31c0069ca0 2024-04-26T01:46:19.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0109490 0x7f31c011aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0109490 0x7f31c011aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38108/0 (socket says 172.21.15.28:38108) 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c5200640 1 -- 172.21.15.28:0/2103909876 learned_addr learned my addr 172.21.15.28:0/2103909876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.497+0000 7f31c6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31c010e980 0x7f31c01b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.498+0000 7f31c5c00640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0069ca0 0x7f31c011a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.498+0000 7f31c5200640 1 -- 172.21.15.28:0/2103909876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0069ca0 msgr2=0x7f31c011a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.498+0000 7f31c5200640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0069ca0 0x7f31c011a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.498+0000 7f31c5200640 1 -- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31c010e980 msgr2=0x7f31c01b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.498+0000 7f31c5200640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31c010e980 0x7f31c01b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.498+0000 7f31c5200640 1 -- 172.21.15.28:0/2103909876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c01b3850 con 0x7f31c0109490 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.498+0000 7f31c5c00640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0069ca0 0x7f31c011a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.498+0000 7f31c5200640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0109490 0x7f31c011aee0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f31ac02a980 tx=0x7f31ac003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.499+0000 7f31be000640 1 -- 172.21.15.28:0/2103909876 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac03f070 con 0x7f31c0109490 2024-04-26T01:46:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.499+0000 7f31be000640 1 -- 172.21.15.28:0/2103909876 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31ac03bb10 con 0x7f31c0109490 2024-04-26T01:46:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.499+0000 7f31be000640 1 -- 172.21.15.28:0/2103909876 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac035030 con 0x7f31c0109490 2024-04-26T01:46:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.499+0000 7f31c8600640 1 -- 172.21.15.28:0/2103909876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c01a3ac0 con 0x7f31c0109490 2024-04-26T01:46:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.499+0000 7f31c8600640 1 -- 172.21.15.28:0/2103909876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c01a3f90 con 0x7f31c0109490 2024-04-26T01:46:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.505+0000 7f31cb5c0740 1 Processor -- start 2024-04-26T01:46:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.505+0000 7f31be000640 1 -- 172.21.15.28:0/2103909876 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f31ac042070 con 0x7f31c0109490 2024-04-26T01:46:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.505+0000 7f31cb5c0740 1 -- start start 2024-04-26T01:46:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.506+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b884820 0x55979b884c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.506+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 0x55979b88f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.506+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b891df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.506+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55979b7ea900 con 0x55979b884820 2024-04-26T01:46:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.506+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55979b790ef0 con 0x55979b88fa00 2024-04-26T01:46:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.506+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55979b7e68c0 con 0x55979b885140 2024-04-26T01:46:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.507+0000 7f31be000640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f31a80b4e50 0x7f31a80b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.507+0000 7f31be000640 1 -- 172.21.15.28:0/2103909876 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f31ac0f0f30 con 0x7f31c0109490 2024-04-26T01:46:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.507+0000 7f31c6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b884820 0x55979b884c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.507+0000 7f31c5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b891df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.507+0000 7f31c5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b891df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38116/0 (socket says 172.21.15.28:38116) 2024-04-26T01:46:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.507+0000 7f31c5200640 1 -- 172.21.15.28:0/1157996301 learned_addr learned my addr 172.21.15.28:0/1157996301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.508+0000 7f31c5c00640 1 --2- 172.21.15.28:0/1157996301 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 0x55979b88f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.508+0000 7f31c5200640 1 -- 172.21.15.28:0/1157996301 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 msgr2=0x55979b88f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.508+0000 7f31c5200640 1 --2- 172.21.15.28:0/1157996301 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 0x55979b88f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.508+0000 7f31c6600640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f31a80b4e50 0x7f31a80b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.508+0000 7f31c5200640 1 -- 172.21.15.28:0/1157996301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b884820 msgr2=0x55979b884c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.508+0000 7f31c5200640 1 --2- 172.21.15.28:0/1157996301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b884820 0x55979b884c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.508+0000 7f31c5200640 1 -- 172.21.15.28:0/1157996301 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55979b4fd210 con 0x55979b88fa00 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.509+0000 7f31c5200640 1 --2- 172.21.15.28:0/1157996301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b891df0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0c5640 tx=0x7f31ac0317d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5bb06d09c8f60f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.509+0000 7f31c6600640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f31a80b4e50 0x7f31a80b7310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009fd0 tx=0x7f31b8004ab0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.509+0000 7f31b2a00640 1 -- 172.21.15.28:0/1157996301 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac0bcf00 con 0x55979b88fa00 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.509+0000 7f31b2a00640 1 -- 172.21.15.28:0/1157996301 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31ac0c12b0 con 0x55979b88fa00 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.510+0000 7f31b2a00640 1 -- 172.21.15.28:0/1157996301 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac0ee930 con 0x55979b88fa00 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.511+0000 7f31cb5c0740 1 -- 172.21.15.28:0/1157996301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 msgr2=0x55979b891df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.511+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/1157996301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b891df0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0c5640 tx=0x7f31ac0317d0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.511+0000 7f31cb5c0740 1 -- 172.21.15.28:0/1157996301 shutdown_connections 2024-04-26T01:46:19.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.511+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/1157996301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b891df0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.511+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/1157996301 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 0x55979b88f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.512+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/1157996301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b884820 0x55979b884c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.512+0000 7f31cb5c0740 1 -- 172.21.15.28:0/1157996301 >> 172.21.15.28:0/1157996301 conn(0x55979b880d30 msgr2=0x55979b881110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.512+0000 7f31cb5c0740 1 -- 172.21.15.28:0/1157996301 shutdown_connections 2024-04-26T01:46:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.513+0000 7f31cb5c0740 1 -- 172.21.15.28:0/1157996301 wait complete. 2024-04-26T01:46:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.513+0000 7f31cb5c0740 1 -- start start 2024-04-26T01:46:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.514+0000 7f31cb5c0740 1 Processor -- start 2024-04-26T01:46:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.514+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 0x55979b88ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.514+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b885140 0x55979b88cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.514+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b88d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.514+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55979b8923f0 con 0x55979b885140 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.514+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55979b7a4310 con 0x55979b88fa00 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.514+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55979b892570 con 0x55979b884820 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.514+0000 7f31cb5c0740 10 client.0 _ll_register_callbacks cb 0x55979b772cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.515+0000 7f31c6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b88d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.515+0000 7f31c6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b88d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38128/0 (socket says 172.21.15.28:38128) 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.515+0000 7f31c6600640 1 -- 172.21.15.28:0/2191887911 learned_addr learned my addr 172.21.15.28:0/2191887911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.515+0000 7f31c5c00640 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 0x55979b88ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.515+0000 7f31c6600640 1 -- 172.21.15.28:0/2191887911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 msgr2=0x55979b88ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.515+0000 7f31c6600640 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 0x55979b88ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.515+0000 7f31c6600640 1 -- 172.21.15.28:0/2191887911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b885140 msgr2=0x55979b88cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.516+0000 7f31c5200640 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b885140 0x55979b88cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.516+0000 7f31c6600640 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b885140 0x55979b88cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.516+0000 7f31c6600640 1 -- 172.21.15.28:0/2191887911 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55979b5c9620 con 0x55979b88fa00 2024-04-26T01:46:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.516+0000 7f31c5200640 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b885140 0x55979b88cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.516+0000 7f31a7400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:46:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.516+0000 7f31c6600640 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b88d480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f31b80132a0 tx=0x7f31b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a5b50c738bc8164 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.518+0000 7f31a7400640 1 -- 172.21.15.28:0/2191887911 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8024070 con 0x55979b88fa00 2024-04-26T01:46:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.518+0000 7f31a7400640 1 -- 172.21.15.28:0/2191887911 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f31b8012b70 con 0x55979b88fa00 2024-04-26T01:46:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.518+0000 7f31cb5c0740 1 -- 172.21.15.28:0/2191887911 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31ac0c12b0 con 0x55979b88fa00 2024-04-26T01:46:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.518+0000 7f31a7400640 1 -- 172.21.15.28:0/2191887911 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f31b8030050 con 0x55979b88fa00 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.518+0000 7f31cb5c0740 10 client.25763 fetch_fsmap learned FSMap version 34 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.518+0000 7f31cb5c0740 1 -- 172.21.15.28:0/2191887911 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55979b89e680 con 0x55979b88fa00 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.518+0000 7f31a7400640 1 -- 172.21.15.28:0/2191887911 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8012d70 con 0x55979b88fa00 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.519+0000 7f31a7400640 1 -- 172.21.15.28:0/2191887911 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f31b800f3f0 con 0x55979b88fa00 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.519+0000 7f31cb5c0740 10 client.25763 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.519+0000 7f31cb5c0740 10 client.25763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.519+0000 7f31cb5c0740 20 client.25763 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.519+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55979b89fbb0 0x55979b8a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.519+0000 7f31cb5c0740 4 client.25763 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.519+0000 7f31cb5c0740 1 -- 172.21.15.28:0/2191887911 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f31ac0c12b0 con 0x55979b89fbb0 2024-04-26T01:46:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.520+0000 7f31c5c00640 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55979b89fbb0 0x55979b8a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.523+0000 7f31c5c00640 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55979b89fbb0 0x55979b8a1fe0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=227dcc4fd7ea6428 server_cookie=b28f5b8179467c48 in_seq=0 out_seq=0 2024-04-26T01:46:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.523+0000 7f31a7400640 10 client.25763 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.524+0000 7f31a7400640 1 -- 172.21.15.28:0/2191887911 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f31ac0c12b0 con 0x55979b89fbb0 2024-04-26T01:46:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.524+0000 7f31a7400640 10 client.25763 handle_command_reply: tid=0 2024-04-26T01:46:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.524+0000 7f31cb5c0740 1 client.25763 shutdown 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31cb5c0740 1 -- 172.21.15.28:0/2191887911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 msgr2=0x55979b88d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b88d480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f31b80132a0 tx=0x7f31b8004a40 comp rx=0 tx=0).stop 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31cb5c0740 1 -- 172.21.15.28:0/2191887911 shutdown_connections 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55979b89fbb0 0x55979b8a1fe0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b88fa00 0x55979b88d480 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b885140 0x55979b88cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/2191887911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 0x55979b88ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31a7400640 0 client.25763 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.525+0000 7f31cb5c0740 1 -- 172.21.15.28:0/2191887911 >> 172.21.15.28:0/2191887911 conn(0x55979b880d30 msgr2=0x55979b886a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.526+0000 7f31cb5c0740 1 -- 172.21.15.28:0/2191887911 shutdown_connections 2024-04-26T01:46:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.526+0000 7f31cb5c0740 1 -- 172.21.15.28:0/2191887911 wait complete. 2024-04-26T01:46:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.526+0000 7f31cb5c0740 20 client.25763 trim_cache size 0 max 16384 2024-04-26T01:46:19.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.557+0000 7f31cb5c0740 1 Processor -- start 2024-04-26T01:46:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.557+0000 7f31cb5c0740 1 -- start start 2024-04-26T01:46:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.557+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 0x55979b89b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.557+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b885140 0x55979b8871e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.557+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b8a78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.557+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01b6a10 con 0x55979b88fa00 2024-04-26T01:46:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.557+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01b4280 con 0x55979b885140 2024-04-26T01:46:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.557+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01aa970 con 0x55979b884820 2024-04-26T01:46:19.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.558+0000 7f31c6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b8a78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.558+0000 7f31c6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b8a78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50400/0 (socket says 172.21.15.28:50400) 2024-04-26T01:46:19.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.558+0000 7f31c6600640 1 -- 172.21.15.28:0/3140263047 learned_addr learned my addr 172.21.15.28:0/3140263047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:19.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.558+0000 7f31c5200640 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b885140 0x55979b8871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.558+0000 7f31c6600640 1 -- 172.21.15.28:0/3140263047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 msgr2=0x55979b89b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:19.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.558+0000 7f31c5c00640 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 0x55979b89b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.559+0000 7f31c6600640 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 0x55979b89b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.559+0000 7f31c6600640 1 -- 172.21.15.28:0/3140263047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b885140 msgr2=0x55979b8871e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:19.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.559+0000 7f31c6600640 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b885140 0x55979b8871e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.559+0000 7f31c6600640 1 -- 172.21.15.28:0/3140263047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55979b8a2660 con 0x55979b88fa00 2024-04-26T01:46:19.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.559+0000 7f31c6600640 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b8a78f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f31b8006010 tx=0x7f31b800ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b203c13360f638e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.560+0000 7f31b2000640 1 -- 172.21.15.28:0/3140263047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8038020 con 0x55979b88fa00 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.560+0000 7f31b2000640 1 -- 172.21.15.28:0/3140263047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b800fa30 con 0x55979b88fa00 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.560+0000 7f31b2000640 1 -- 172.21.15.28:0/3140263047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8024070 con 0x55979b88fa00 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.561+0000 7f31cb5c0740 1 -- 172.21.15.28:0/3140263047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 msgr2=0x55979b8a78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.561+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b8a78f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f31b8006010 tx=0x7f31b800ff90 comp rx=0 tx=0).stop 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.561+0000 7f31cb5c0740 1 -- 172.21.15.28:0/3140263047 shutdown_connections 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.561+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b8a78f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.561+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b885140 0x55979b8871e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.561+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/3140263047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b884820 0x55979b89b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.561+0000 7f31cb5c0740 1 -- 172.21.15.28:0/3140263047 >> 172.21.15.28:0/3140263047 conn(0x55979b880d30 msgr2=0x55979b87fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.562+0000 7f31cb5c0740 1 -- 172.21.15.28:0/3140263047 shutdown_connections 2024-04-26T01:46:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.562+0000 7f31cb5c0740 1 -- 172.21.15.28:0/3140263047 wait complete. 2024-04-26T01:46:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.562+0000 7f31cb5c0740 1 -- start start 2024-04-26T01:46:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.563+0000 7f31cb5c0740 1 Processor -- start 2024-04-26T01:46:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.563+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b884820 0x55979b895140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 0x55979b895680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31cb5c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b895bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01aafe0 con 0x55979b88fa00 2024-04-26T01:46:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01b3bb0 con 0x55979b884820 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31cb5c0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01ab6b0 con 0x55979b885140 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b884820 0x55979b895140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31cb5c0740 10 client.0 _ll_register_callbacks cb 0x7f31b8013eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b884820 0x55979b895140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38154/0 (socket says 172.21.15.28:38154) 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5c00640 1 -- 172.21.15.28:0/153570730 learned_addr learned my addr 172.21.15.28:0/153570730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5200640 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 0x55979b895680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5c00640 1 -- 172.21.15.28:0/153570730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 msgr2=0x55979b895680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5c00640 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 0x55979b895680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5c00640 1 -- 172.21.15.28:0/153570730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 msgr2=0x55979b895bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5c00640 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b895bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.564+0000 7f31c5c00640 1 -- 172.21.15.28:0/153570730 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55979b905170 con 0x55979b884820 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.565+0000 7f31cb5c0740 1 -- 172.21.15.28:0/153570730 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b800fa30 con 0x55979b884820 2024-04-26T01:46:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.565+0000 7f31c5c00640 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b884820 0x55979b895140 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55979b8a2890 tx=0x7f31b4003de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31b4896dc083ba15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.565+0000 7f31a7400640 10 client.25769 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:46:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.565+0000 7f31a7400640 1 -- 172.21.15.28:0/153570730 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4013b30 con 0x55979b884820 2024-04-26T01:46:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.565+0000 7f31a7400640 1 -- 172.21.15.28:0/153570730 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f31b4013cd0 con 0x55979b884820 2024-04-26T01:46:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.565+0000 7f31a7400640 1 -- 172.21.15.28:0/153570730 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f31b401a2c0 con 0x55979b884820 2024-04-26T01:46:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.565+0000 7f31cb5c0740 10 client.25769 fetch_fsmap learned FSMap version 34 2024-04-26T01:46:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.565+0000 7f31cb5c0740 1 -- 172.21.15.28:0/153570730 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({fsmap=34}) -- 0x55979b906400 con 0x55979b884820 2024-04-26T01:46:19.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.566+0000 7f31a7400640 1 -- 172.21.15.28:0/153570730 <== mon.2 v2:172.21.15.53:3300/0 4 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f31b401a460 con 0x55979b884820 2024-04-26T01:46:19.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.567+0000 7f31cb5c0740 10 client.25769 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:46:19.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.567+0000 7f31cb5c0740 10 client.25769 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:19.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.567+0000 7f31cb5c0740 20 client.25769 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:19.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.567+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55979b906b20 0x55979b908f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.567+0000 7f31cb5c0740 4 client.25769 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:19.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.567+0000 7f31cb5c0740 1 -- 172.21.15.28:0/153570730 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55979b7fe8b0 con 0x55979b906b20 2024-04-26T01:46:19.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.568+0000 7f31c5200640 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55979b906b20 0x55979b908f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.570+0000 7f31c5200640 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55979b906b20 0x55979b908f60 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e69d774b3db327 server_cookie=305c84e152be8297 in_seq=0 out_seq=0 2024-04-26T01:46:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.570+0000 7f31a7400640 10 client.25769 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.641+0000 7f31a7400640 1 -- 172.21.15.28:0/153570730 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55979b7fe8b0 con 0x55979b906b20 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.641+0000 7f31a7400640 10 client.25769 handle_command_reply: tid=0 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.641+0000 7f31cb5c0740 1 client.25769 shutdown 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.641+0000 7f31cb5c0740 1 -- 172.21.15.28:0/153570730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b884820 msgr2=0x55979b895140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.641+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b884820 0x55979b895140 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55979b8a2890 tx=0x7f31b4003de0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.641+0000 7f31cb5c0740 1 -- 172.21.15.28:0/153570730 shutdown_connections 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.641+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55979b906b20 0x55979b908f60 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.642+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55979b88fa00 0x55979b895bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.642+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55979b885140 0x55979b895680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.642+0000 7f31cb5c0740 1 --2- 172.21.15.28:0/153570730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55979b884820 0x55979b895140 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.642+0000 7f31cb5c0740 1 -- 172.21.15.28:0/153570730 >> 172.21.15.28:0/153570730 conn(0x55979b880d30 msgr2=0x55979b87fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.642+0000 7f31a7400640 0 client.25769 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.642+0000 7f31cb5c0740 1 -- 172.21.15.28:0/153570730 shutdown_connections 2024-04-26T01:46:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.642+0000 7f31cb5c0740 1 -- 172.21.15.28:0/153570730 wait complete. 2024-04-26T01:46:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.643+0000 7f31cb5c0740 20 client.25769 trim_cache size 0 max 16384 2024-04-26T01:46:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:46:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:46:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:46:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:46:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.644+0000 7f31c8600640 1 -- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f31a80b4e50 msgr2=0x7f31a80b7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f31a80b4e50 0x7f31a80b7310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009fd0 tx=0x7f31b8004ab0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 -- 172.21.15.28:0/2103909876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0109490 msgr2=0x7f31c011aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0109490 0x7f31c011aee0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f31ac02a980 tx=0x7f31ac003c00 comp rx=0 tx=0).stop 2024-04-26T01:46:19.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 -- 172.21.15.28:0/2103909876 shutdown_connections 2024-04-26T01:46:19.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f31a80b4e50 0x7f31a80b7310 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31c010e980 0x7f31c01b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c0109490 0x7f31c011aee0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 --2- 172.21.15.28:0/2103909876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0069ca0 0x7f31c011a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 -- 172.21.15.28:0/2103909876 >> 172.21.15.28:0/2103909876 conn(0x7f31c006d4b0 msgr2=0x7f31c0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.645+0000 7f31c8600640 1 -- 172.21.15.28:0/2103909876 shutdown_connections 2024-04-26T01:46:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:19.646+0000 7f31c8600640 1 -- 172.21.15.28:0/2103909876 wait complete. 2024-04-26T01:46:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 265.0 s, 3235.0 tps, lat 30.927 ms stddev 33.053 2024-04-26T01:46:21.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:21 smithi053 ceph-mon[32593]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-26T01:46:21.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:21 smithi028 ceph-mon[28247]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-26T01:46:21.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:21 smithi169 ceph-mon[32851]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-26T01:46:23.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:23 smithi053 ceph-mon[32593]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 MiB/s wr, 685 op/s 2024-04-26T01:46:23.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:23 smithi028 ceph-mon[28247]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 MiB/s wr, 685 op/s 2024-04-26T01:46:23.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:23 smithi169 ceph-mon[32851]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 MiB/s wr, 685 op/s 2024-04-26T01:46:25.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:25 smithi053 ceph-mon[32593]: pgmap v166: 97 pgs: 97 active+clean; 2.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.4 MiB/s wr, 845 op/s 2024-04-26T01:46:25.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:25 smithi028 ceph-mon[28247]: pgmap v166: 97 pgs: 97 active+clean; 2.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.4 MiB/s wr, 845 op/s 2024-04-26T01:46:25.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:25 smithi169 ceph-mon[32851]: pgmap v166: 97 pgs: 97 active+clean; 2.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.4 MiB/s wr, 845 op/s 2024-04-26T01:46:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 270.0 s, 2681.0 tps, lat 36.440 ms stddev 41.496 2024-04-26T01:46:27.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:27 smithi053 ceph-mon[32593]: pgmap v167: 97 pgs: 97 active+clean; 2.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.6 MiB/s wr, 735 op/s 2024-04-26T01:46:27.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:27 smithi028 ceph-mon[28247]: pgmap v167: 97 pgs: 97 active+clean; 2.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.6 MiB/s wr, 735 op/s 2024-04-26T01:46:27.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:27 smithi169 ceph-mon[32851]: pgmap v167: 97 pgs: 97 active+clean; 2.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.6 MiB/s wr, 735 op/s 2024-04-26T01:46:29.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:29 smithi053 ceph-mon[32593]: pgmap v168: 97 pgs: 97 active+clean; 2.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.3 MiB/s wr, 624 op/s 2024-04-26T01:46:29.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:29 smithi028 ceph-mon[28247]: pgmap v168: 97 pgs: 97 active+clean; 2.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.3 MiB/s wr, 624 op/s 2024-04-26T01:46:29.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:29 smithi169 ceph-mon[32851]: pgmap v168: 97 pgs: 97 active+clean; 2.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.3 MiB/s wr, 624 op/s 2024-04-26T01:46:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 275.0 s, 2518.4 tps, lat 40.516 ms stddev 45.929 2024-04-26T01:46:31.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:31 smithi053 ceph-mon[32593]: pgmap v169: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 911 op/s 2024-04-26T01:46:31.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:31 smithi028 ceph-mon[28247]: pgmap v169: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 911 op/s 2024-04-26T01:46:31.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:31 smithi169 ceph-mon[32851]: pgmap v169: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 911 op/s 2024-04-26T01:46:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.841+0000 7f0a0f800640 1 -- 172.21.15.28:0/3351946524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a08107270 msgr2=0x7f0a0810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.841+0000 7f0a0f800640 1 --2- 172.21.15.28:0/3351946524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a08107270 0x7f0a0810ee40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009900 tx=0x7f0a0002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.842+0000 7f0a0f800640 1 -- 172.21.15.28:0/3351946524 shutdown_connections 2024-04-26T01:46:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.842+0000 7f0a0f800640 1 --2- 172.21.15.28:0/3351946524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a08107270 0x7f0a0810ee40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.842+0000 7f0a0f800640 1 --2- 172.21.15.28:0/3351946524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a08102b50 0x7f0a08106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.842+0000 7f0a0f800640 1 --2- 172.21.15.28:0/3351946524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a081021a0 0x7f0a08102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.842+0000 7f0a0f800640 1 -- 172.21.15.28:0/3351946524 >> 172.21.15.28:0/3351946524 conn(0x7f0a080fda00 msgr2=0x7f0a080ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.842+0000 7f0a0f800640 1 -- 172.21.15.28:0/3351946524 shutdown_connections 2024-04-26T01:46:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.842+0000 7f0a0f800640 1 -- 172.21.15.28:0/3351946524 wait complete. 2024-04-26T01:46:32.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.843+0000 7f0a0f800640 1 Processor -- start 2024-04-26T01:46:32.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.844+0000 7f0a0f800640 1 -- start start 2024-04-26T01:46:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a081021a0 0x7f0a0810e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0f800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a08107270 0x7f0a0810eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0f800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0810c850 0x7f0a0810cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0810f710 con 0x7f0a081021a0 2024-04-26T01:46:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0f800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0810f590 con 0x7f0a0810c850 2024-04-26T01:46:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0f800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0810f890 con 0x7f0a08107270 2024-04-26T01:46:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0810c850 0x7f0a0810cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0810c850 0x7f0a0810cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43918/0 (socket says 172.21.15.28:43918) 2024-04-26T01:46:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a07e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a08107270 0x7f0a0810eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.845+0000 7f0a0d800640 1 -- 172.21.15.28:0/4109128031 learned_addr learned my addr 172.21.15.28:0/4109128031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.846+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a081021a0 0x7f0a0810e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.846+0000 7f0a0d800640 1 -- 172.21.15.28:0/4109128031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a08107270 msgr2=0x7f0a0810eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.846+0000 7f0a0d800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a08107270 0x7f0a0810eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.846+0000 7f0a0d800640 1 -- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a081021a0 msgr2=0x7f0a0810e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.846+0000 7f0a0d800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a081021a0 0x7f0a0810e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.846+0000 7f0a0d800640 1 -- 172.21.15.28:0/4109128031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0810d370 con 0x7f0a0810c850 2024-04-26T01:46:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.847+0000 7f0a0d800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0810c850 0x7f0a0810cd00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0a0002a960 tx=0x7f0a00003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.847+0000 7f0a05600640 1 -- 172.21.15.28:0/4109128031 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0003f070 con 0x7f0a0810c850 2024-04-26T01:46:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.847+0000 7f0a05600640 1 -- 172.21.15.28:0/4109128031 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a00031b50 con 0x7f0a0810c850 2024-04-26T01:46:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.847+0000 7f0a05600640 1 -- 172.21.15.28:0/4109128031 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00035030 con 0x7f0a0810c850 2024-04-26T01:46:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.848+0000 7f0a0f800640 1 -- 172.21.15.28:0/4109128031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0810d600 con 0x7f0a0810c850 2024-04-26T01:46:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.848+0000 7f0a0f800640 1 -- 172.21.15.28:0/4109128031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a081ab990 con 0x7f0a0810c850 2024-04-26T01:46:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.853+0000 7f0a05600640 1 -- 172.21.15.28:0/4109128031 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0a00042070 con 0x7f0a0810c850 2024-04-26T01:46:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.855+0000 7f0a1278d740 1 Processor -- start 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.856+0000 7f0a1278d740 1 -- start start 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.857+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 0x55a59314bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.857+0000 7f0a05600640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f09ec0a3ff0 0x7f09ec0a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.858+0000 7f0a05600640 1 -- 172.21.15.28:0/4109128031 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f0a000f0f50 con 0x7f0a0810c850 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.858+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a5931564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.859+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593158df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.859+0000 7f0a1278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5930b1900 con 0x55a59314c140 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.859+0000 7f0a1278d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a593057ef0 con 0x55a593156a00 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.859+0000 7f0a1278d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55a5930ad8c0 con 0x55a59314b820 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.859+0000 7f0a07e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 0x55a59314bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.860+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.860+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43922/0 (socket says 172.21.15.28:43922) 2024-04-26T01:46:32.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.860+0000 7f0a0d800640 1 -- 172.21.15.28:0/19284511 learned_addr learned my addr 172.21.15.28:0/19284511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:32.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.861+0000 7f0a07e00640 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a5931564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.861+0000 7f0a0d800640 1 -- 172.21.15.28:0/19284511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 msgr2=0x55a59314bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:32.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.861+0000 7f0a0d800640 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 0x55a59314bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.861+0000 7f0a0d800640 1 -- 172.21.15.28:0/19284511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 msgr2=0x55a5931564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.861+0000 7f0a0d800640 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a5931564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.861+0000 7f0a0d800640 1 -- 172.21.15.28:0/19284511 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a592dc4210 con 0x55a593156a00 2024-04-26T01:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.862+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f09ec0a3ff0 0x7f09ec0a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.862+0000 7f0a07e00640 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a5931564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.865+0000 7f0a0d800640 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593158df0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0a000c5600 tx=0x7f0a000bc820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c8d0dea6ee66ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.865+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f09ec0a3ff0 0x7f09ec0a64b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f09f0005e10 tx=0x7f09f0005da0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.865+0000 7f09fe000640 1 -- 172.21.15.28:0/19284511 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000d5020 con 0x55a593156a00 2024-04-26T01:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.865+0000 7f09fe000640 1 -- 172.21.15.28:0/19284511 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a000c1270 con 0x55a593156a00 2024-04-26T01:46:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.865+0000 7f09fe000640 1 -- 172.21.15.28:0/19284511 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000ee8f0 con 0x55a593156a00 2024-04-26T01:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.868+0000 7f0a1278d740 1 -- 172.21.15.28:0/19284511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 msgr2=0x55a593158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.868+0000 7f0a1278d740 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593158df0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0a000c5600 tx=0x7f0a000bc820 comp rx=0 tx=0).stop 2024-04-26T01:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.868+0000 7f0a1278d740 1 -- 172.21.15.28:0/19284511 shutdown_connections 2024-04-26T01:46:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.868+0000 7f0a1278d740 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593158df0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.868+0000 7f0a1278d740 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a5931564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.868+0000 7f0a1278d740 1 --2- 172.21.15.28:0/19284511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 0x55a59314bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.868+0000 7f0a1278d740 1 -- 172.21.15.28:0/19284511 >> 172.21.15.28:0/19284511 conn(0x55a593147d30 msgr2=0x55a593148110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.870+0000 7f0a1278d740 1 -- 172.21.15.28:0/19284511 shutdown_connections 2024-04-26T01:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.870+0000 7f0a1278d740 1 -- 172.21.15.28:0/19284511 wait complete. 2024-04-26T01:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.871+0000 7f0a1278d740 1 -- start start 2024-04-26T01:46:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.871+0000 7f0a1278d740 1 Processor -- start 2024-04-26T01:46:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314b820 0x55a5931539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314c140 0x55a593153f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593154460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a1278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a59316a400 con 0x55a59314b820 2024-04-26T01:46:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a1278d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a59306b310 con 0x55a593156a00 2024-04-26T01:46:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a1278d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55a59316a580 con 0x55a59314c140 2024-04-26T01:46:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a1278d740 10 client.0 _ll_register_callbacks cb 0x55a593039cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a07e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314c140 0x55a593153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43938/0 (socket says 172.21.15.28:43938) 2024-04-26T01:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.872+0000 7f0a0d800640 1 -- 172.21.15.28:0/1270183839 learned_addr learned my addr 172.21.15.28:0/1270183839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.873+0000 7f0a07e00640 1 -- 172.21.15.28:0/1270183839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 msgr2=0x55a593154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.873+0000 7f0a07e00640 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.873+0000 7f0a07e00640 1 -- 172.21.15.28:0/1270183839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314b820 msgr2=0x55a5931539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.873+0000 7f0a07e00640 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314b820 0x55a5931539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.873+0000 7f0a07e00640 1 -- 172.21.15.28:0/1270183839 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a592e90620 con 0x55a59314c140 2024-04-26T01:46:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.875+0000 7f0a07e00640 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314c140 0x55a593153f20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0a08069280 tx=0x7f09f8012d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b1ecae1d8ba11dd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.875+0000 7f09f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.875+0000 7f09f6a00640 1 -- 172.21.15.28:0/1270183839 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f8004d20 con 0x55a59314c140 2024-04-26T01:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.875+0000 7f09f6a00640 1 -- 172.21.15.28:0/1270183839 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f09f8011ac0 con 0x55a59314c140 2024-04-26T01:46:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.876+0000 7f0a1278d740 1 -- 172.21.15.28:0/1270183839 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a000c1270 con 0x55a59314c140 2024-04-26T01:46:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.876+0000 7f09f6a00640 1 -- 172.21.15.28:0/1270183839 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f09f8011d00 con 0x55a59314c140 2024-04-26T01:46:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.876+0000 7f0a1278d740 10 client.25792 fetch_fsmap learned FSMap version 34 2024-04-26T01:46:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.876+0000 7f0a1278d740 1 -- 172.21.15.28:0/1270183839 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a5931765f0 con 0x55a59314c140 2024-04-26T01:46:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.876+0000 7f09f6a00640 1 -- 172.21.15.28:0/1270183839 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a000c1270 con 0x55a59314c140 2024-04-26T01:46:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.877+0000 7f09f6a00640 1 -- 172.21.15.28:0/1270183839 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f09f801b030 con 0x55a59314c140 2024-04-26T01:46:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.877+0000 7f0a1278d740 10 client.25792 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:46:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.877+0000 7f0a1278d740 10 client.25792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.877+0000 7f0a1278d740 20 client.25792 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.877+0000 7f0a1278d740 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a593177ad0 0x55a593179ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.877+0000 7f0a1278d740 4 client.25792 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.877+0000 7f0a1278d740 1 -- 172.21.15.28:0/1270183839 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5930c58b0 con 0x55a593177ad0 2024-04-26T01:46:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.878+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a593177ad0 0x55a593179ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.878+0000 7f09f6a00640 10 client.25792 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.879+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a593177ad0 0x55a593179ec0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4d22fbcb331f83e server_cookie=3a291b3b58d210a8 in_seq=0 out_seq=0 2024-04-26T01:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.880+0000 7f09f6a00640 1 -- 172.21.15.28:0/1270183839 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5930c58b0 con 0x55a593177ad0 2024-04-26T01:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.880+0000 7f09f6a00640 10 client.25792 handle_command_reply: tid=0 2024-04-26T01:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.880+0000 7f0a1278d740 1 client.25792 shutdown 2024-04-26T01:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.881+0000 7f0a1278d740 1 -- 172.21.15.28:0/1270183839 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314c140 msgr2=0x55a593153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.881+0000 7f0a1278d740 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314c140 0x55a593153f20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0a08069280 tx=0x7f09f8012d50 comp rx=0 tx=0).stop 2024-04-26T01:46:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.882+0000 7f0a1278d740 1 -- 172.21.15.28:0/1270183839 shutdown_connections 2024-04-26T01:46:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.882+0000 7f0a1278d740 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a593177ad0 0x55a593179ec0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.882+0000 7f0a1278d740 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a593154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.882+0000 7f0a1278d740 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314c140 0x55a593153f20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.882+0000 7f0a1278d740 1 --2- 172.21.15.28:0/1270183839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314b820 0x55a5931539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.882+0000 7f0a1278d740 1 -- 172.21.15.28:0/1270183839 >> 172.21.15.28:0/1270183839 conn(0x55a593147d30 msgr2=0x55a59314dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.882+0000 7f09f6a00640 0 client.25792 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.883+0000 7f0a1278d740 1 -- 172.21.15.28:0/1270183839 shutdown_connections 2024-04-26T01:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.883+0000 7f0a1278d740 1 -- 172.21.15.28:0/1270183839 wait complete. 2024-04-26T01:46:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.883+0000 7f0a1278d740 20 client.25792 trim_cache size 0 max 16384 2024-04-26T01:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.908+0000 7f0a1278d740 1 Processor -- start 2024-04-26T01:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.908+0000 7f0a1278d740 1 -- start start 2024-04-26T01:46:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.909+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 0x55a593173230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.909+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59314e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.909+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a5931540b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.909+0000 7f0a1278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081ae1f0 con 0x55a59314c140 2024-04-26T01:46:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.909+0000 7f0a1278d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0811c600 con 0x55a593156a00 2024-04-26T01:46:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.909+0000 7f0a1278d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0811a450 con 0x55a59314b820 2024-04-26T01:46:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.910+0000 7f0a0ce00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 0x55a593173230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.910+0000 7f0a07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59314e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.910+0000 7f0a07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59314e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52910/0 (socket says 172.21.15.28:52910) 2024-04-26T01:46:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.910+0000 7f0a07e00640 1 -- 172.21.15.28:0/3538415784 learned_addr learned my addr 172.21.15.28:0/3538415784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.909+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a5931540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.911+0000 7f0a0d800640 1 -- 172.21.15.28:0/3538415784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 msgr2=0x55a593173230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.911+0000 7f0a0d800640 1 --2- 172.21.15.28:0/3538415784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 0x55a593173230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.911+0000 7f0a0d800640 1 -- 172.21.15.28:0/3538415784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 msgr2=0x55a59314e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.911+0000 7f0a0d800640 1 --2- 172.21.15.28:0/3538415784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59314e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.911+0000 7f0a0d800640 1 -- 172.21.15.28:0/3538415784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59317a540 con 0x55a593156a00 2024-04-26T01:46:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.912+0000 7f0a0d800640 1 --2- 172.21.15.28:0/3538415784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a5931540b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0a000b8ba0 tx=0x7f0a00005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d52baf58fc8b0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.912+0000 7f09fd600640 1 -- 172.21.15.28:0/3538415784 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000d5040 con 0x55a593156a00 2024-04-26T01:46:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.912+0000 7f09fd600640 1 -- 172.21.15.28:0/3538415784 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a0003b460 con 0x55a593156a00 2024-04-26T01:46:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.912+0000 7f09fd600640 1 -- 172.21.15.28:0/3538415784 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000d6220 con 0x55a593156a00 2024-04-26T01:46:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.913+0000 7f0a1278d740 1 -- 172.21.15.28:0/3538415784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 msgr2=0x55a5931540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.913+0000 7f0a1278d740 1 --2- 172.21.15.28:0/3538415784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a5931540b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0a000b8ba0 tx=0x7f0a00005c90 comp rx=0 tx=0).stop 2024-04-26T01:46:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.913+0000 7f0a1278d740 1 -- 172.21.15.28:0/3538415784 shutdown_connections 2024-04-26T01:46:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.913+0000 7f0a1278d740 1 --2- 172.21.15.28:0/3538415784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a593156a00 0x55a5931540b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.913+0000 7f0a1278d740 1 --2- 172.21.15.28:0/3538415784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59314e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.913+0000 7f0a1278d740 1 --2- 172.21.15.28:0/3538415784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a59314b820 0x55a593173230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.913+0000 7f0a1278d740 1 -- 172.21.15.28:0/3538415784 >> 172.21.15.28:0/3538415784 conn(0x55a593147d30 msgr2=0x55a5931539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:32.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.914+0000 7f0a1278d740 1 -- 172.21.15.28:0/3538415784 shutdown_connections 2024-04-26T01:46:32.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.914+0000 7f0a1278d740 1 -- 172.21.15.28:0/3538415784 wait complete. 2024-04-26T01:46:32.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.914+0000 7f0a1278d740 1 -- start start 2024-04-26T01:46:32.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.914+0000 7f0a1278d740 1 Processor -- start 2024-04-26T01:46:32.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a59314b820 0x55a59316f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59316f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a1278d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a593156a00 0x55a59316ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a1278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0811aac0 con 0x55a59314c140 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a1278d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0811bf30 con 0x55a59314b820 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a1278d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0811b190 con 0x55a593156a00 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a1278d740 10 client.0 _ll_register_callbacks cb 0x7f09f8012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a593156a00 0x55a59316ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a0d800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a593156a00 0x55a59316ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37476/0 (socket says 172.21.15.28:37476) 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.916+0000 7f0a0d800640 1 -- 172.21.15.28:0/2614888949 learned_addr learned my addr 172.21.15.28:0/2614888949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.917+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a59314b820 0x55a59316f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.917+0000 7f0a0d800640 1 -- 172.21.15.28:0/2614888949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a59314b820 msgr2=0x55a59316f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.917+0000 7f0a0d800640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a59314b820 0x55a59316f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.917+0000 7f0a0d800640 1 -- 172.21.15.28:0/2614888949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 msgr2=0x55a59316f9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:46:32.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.917+0000 7f0a07e00640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59316f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.918+0000 7f0a0d800640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59316f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.918+0000 7f0a0d800640 1 -- 172.21.15.28:0/2614888949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a593158df0 con 0x55a593156a00 2024-04-26T01:46:32.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.918+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a59314b820 0x55a59316f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:32.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.918+0000 7f0a07e00640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59316f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.918+0000 7f0a0d800640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a593156a00 0x55a59316ca60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a000bce50 tx=0x7f0a00009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5e533971daa9a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.918+0000 7f09f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.918+0000 7f09f6a00640 1 -- 172.21.15.28:0/2614888949 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000c9030 con 0x55a593156a00 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.918+0000 7f0a1278d740 1 -- 172.21.15.28:0/2614888949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a0003b460 con 0x55a593156a00 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.919+0000 7f09f6a00640 1 -- 172.21.15.28:0/2614888949 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f0a000b9530 con 0x55a593156a00 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.919+0000 7f09f6a00640 1 -- 172.21.15.28:0/2614888949 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0a000d5070 con 0x55a593156a00 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.919+0000 7f0a1278d740 10 client.25798 fetch_fsmap learned FSMap version 34 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.919+0000 7f0a1278d740 1 -- 172.21.15.28:0/2614888949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a5931dda10 con 0x55a593156a00 2024-04-26T01:46:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.919+0000 7f09f6a00640 1 -- 172.21.15.28:0/2614888949 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a0003b460 con 0x55a593156a00 2024-04-26T01:46:32.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.919+0000 7f09f6a00640 1 -- 172.21.15.28:0/2614888949 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f0a000bc310 con 0x55a593156a00 2024-04-26T01:46:32.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.920+0000 7f0a1278d740 10 client.25798 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:46:32.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.920+0000 7f0a1278d740 10 client.25798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.920+0000 7f0a1278d740 20 client.25798 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.920+0000 7f0a1278d740 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a5931ddda0 0x55a5931e0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.920+0000 7f0a1278d740 4 client.25798 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.920+0000 7f0a1278d740 1 -- 172.21.15.28:0/2614888949 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a5930bf320 con 0x55a5931ddda0 2024-04-26T01:46:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.920+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a5931ddda0 0x55a5931e0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.922+0000 7f0a0ce00640 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a5931ddda0 0x55a5931e0190 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=517909657ccd0202 server_cookie=bf5685ff15974bd0 in_seq=0 out_seq=0 2024-04-26T01:46:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.922+0000 7f09f6a00640 10 client.25798 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.934+0000 7f09f6a00640 1 -- 172.21.15.28:0/2614888949 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a5930bf320 con 0x55a5931ddda0 2024-04-26T01:46:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.934+0000 7f09f6a00640 10 client.25798 handle_command_reply: tid=0 2024-04-26T01:46:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.934+0000 7f0a1278d740 1 client.25798 shutdown 2024-04-26T01:46:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.934+0000 7f0a1278d740 1 -- 172.21.15.28:0/2614888949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a593156a00 msgr2=0x55a59316ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.934+0000 7f0a1278d740 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a593156a00 0x55a59316ca60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a000bce50 tx=0x7f0a00009260 comp rx=0 tx=0).stop 2024-04-26T01:46:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.934+0000 7f0a1278d740 1 -- 172.21.15.28:0/2614888949 shutdown_connections 2024-04-26T01:46:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.934+0000 7f0a1278d740 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55a5931ddda0 0x55a5931e0190 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.935+0000 7f0a1278d740 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a593156a00 0x55a59316ca60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.935+0000 7f0a1278d740 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59314c140 0x55a59316f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.935+0000 7f0a1278d740 1 --2- 172.21.15.28:0/2614888949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a59314b820 0x55a59316f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.935+0000 7f0a1278d740 1 -- 172.21.15.28:0/2614888949 >> 172.21.15.28:0/2614888949 conn(0x55a593147d30 msgr2=0x55a593154210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.935+0000 7f09f6a00640 0 client.25798 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.935+0000 7f0a1278d740 1 -- 172.21.15.28:0/2614888949 shutdown_connections 2024-04-26T01:46:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.935+0000 7f0a1278d740 1 -- 172.21.15.28:0/2614888949 wait complete. 2024-04-26T01:46:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.935+0000 7f0a1278d740 20 client.25798 trim_cache size 0 max 16384 2024-04-26T01:46:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:46:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:46:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:46:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:46:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 -- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f09ec0a3ff0 msgr2=0x7f09ec0a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f09ec0a3ff0 0x7f09ec0a64b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f09f0005e10 tx=0x7f09f0005da0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 -- 172.21.15.28:0/4109128031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0810c850 msgr2=0x7f0a0810cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0810c850 0x7f0a0810cd00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0a0002a960 tx=0x7f0a00003cd0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 -- 172.21.15.28:0/4109128031 shutdown_connections 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f09ec0a3ff0 0x7f09ec0a64b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0810c850 0x7f0a0810cd00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a08107270 0x7f0a0810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 --2- 172.21.15.28:0/4109128031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a081021a0 0x7f0a0810e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 -- 172.21.15.28:0/4109128031 >> 172.21.15.28:0/4109128031 conn(0x7f0a080fda00 msgr2=0x7f0a080fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 -- 172.21.15.28:0/4109128031 shutdown_connections 2024-04-26T01:46:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:32.938+0000 7f0a0f800640 1 -- 172.21.15.28:0/4109128031 wait complete. 2024-04-26T01:46:33.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:33 smithi053 ceph-mon[32593]: pgmap v170: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.9 MiB/s wr, 592 op/s 2024-04-26T01:46:33.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:33 smithi028 ceph-mon[28247]: pgmap v170: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.9 MiB/s wr, 592 op/s 2024-04-26T01:46:33.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:33 smithi169 ceph-mon[32851]: pgmap v170: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.9 MiB/s wr, 592 op/s 2024-04-26T01:46:35.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:35 smithi053 ceph-mon[32593]: pgmap v171: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.0 MiB/s wr, 805 op/s 2024-04-26T01:46:35.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:35 smithi028 ceph-mon[28247]: pgmap v171: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.0 MiB/s wr, 805 op/s 2024-04-26T01:46:35.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:35 smithi169 ceph-mon[32851]: pgmap v171: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.0 MiB/s wr, 805 op/s 2024-04-26T01:46:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 280.0 s, 2598.8 tps, lat 38.451 ms stddev 42.892 2024-04-26T01:46:37.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:37 smithi053 ceph-mon[32593]: pgmap v172: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 717 op/s 2024-04-26T01:46:37.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:37 smithi028 ceph-mon[28247]: pgmap v172: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 717 op/s 2024-04-26T01:46:37.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:37 smithi169 ceph-mon[32851]: pgmap v172: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 717 op/s 2024-04-26T01:46:39.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:39 smithi053 ceph-mon[32593]: pgmap v173: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.7 MiB/s wr, 571 op/s 2024-04-26T01:46:39.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:39 smithi028 ceph-mon[28247]: pgmap v173: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.7 MiB/s wr, 571 op/s 2024-04-26T01:46:39.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:39 smithi169 ceph-mon[32851]: pgmap v173: 97 pgs: 97 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.7 MiB/s wr, 571 op/s 2024-04-26T01:46:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 285.0 s, 2532.8 tps, lat 39.580 ms stddev 46.353 2024-04-26T01:46:41.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:41 smithi053 ceph-mon[32593]: pgmap v174: 97 pgs: 97 active+clean; 2.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.4 MiB/s wr, 861 op/s 2024-04-26T01:46:41.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:41 smithi028 ceph-mon[28247]: pgmap v174: 97 pgs: 97 active+clean; 2.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.4 MiB/s wr, 861 op/s 2024-04-26T01:46:41.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:41 smithi169 ceph-mon[32851]: pgmap v174: 97 pgs: 97 active+clean; 2.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.4 MiB/s wr, 861 op/s 2024-04-26T01:46:43.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:43 smithi028 ceph-mon[28247]: pgmap v175: 97 pgs: 97 active+clean; 2.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.4 MiB/s wr, 574 op/s 2024-04-26T01:46:43.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:43 smithi169 ceph-mon[32851]: pgmap v175: 97 pgs: 97 active+clean; 2.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.4 MiB/s wr, 574 op/s 2024-04-26T01:46:43.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:43 smithi053 ceph-mon[32593]: pgmap v175: 97 pgs: 97 active+clean; 2.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.4 MiB/s wr, 574 op/s 2024-04-26T01:46:45.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:45 smithi028 ceph-mon[28247]: pgmap v176: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-04-26T01:46:45.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:45 smithi169 ceph-mon[32851]: pgmap v176: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-04-26T01:46:45.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:45 smithi053 ceph-mon[32593]: pgmap v176: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-04-26T01:46:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.123+0000 7fcf0a800640 1 -- 172.21.15.28:0/3926898286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04112dd0 msgr2=0x7fcf041151c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.123+0000 7fcf0a800640 1 --2- 172.21.15.28:0/3926898286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04112dd0 0x7fcf041151c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009cd0 tx=0x7fcefc02a8e0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.124+0000 7fcf0a800640 1 -- 172.21.15.28:0/3926898286 shutdown_connections 2024-04-26T01:46:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.124+0000 7fcf0a800640 1 --2- 172.21.15.28:0/3926898286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04112dd0 0x7fcf041151c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.124+0000 7fcf0a800640 1 --2- 172.21.15.28:0/3926898286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf04076da0 0x7fcf04112890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.124+0000 7fcf0a800640 1 --2- 172.21.15.28:0/3926898286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf04076480 0x7fcf04076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.124+0000 7fcf0a800640 1 -- 172.21.15.28:0/3926898286 >> 172.21.15.28:0/3926898286 conn(0x7fcf040ffc40 msgr2=0x7fcf04102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.124+0000 7fcf0a800640 1 -- 172.21.15.28:0/3926898286 shutdown_connections 2024-04-26T01:46:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.124+0000 7fcf0a800640 1 -- 172.21.15.28:0/3926898286 wait complete. 2024-04-26T01:46:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.125+0000 7fcf0a800640 1 Processor -- start 2024-04-26T01:46:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.125+0000 7fcf0a800640 1 -- start start 2024-04-26T01:46:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf0a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04076480 0x7fcf041a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf03400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04076480 0x7fcf041a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf03400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04076480 0x7fcf041a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43470/0 (socket says 172.21.15.28:43470) 2024-04-26T01:46:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf0a800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf04076da0 0x7fcf041a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf03400640 1 -- 172.21.15.28:0/185804166 learned_addr learned my addr 172.21.15.28:0/185804166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf0a800640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf04112dd0 0x7fcf0419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf04118810 con 0x7fcf04112dd0 2024-04-26T01:46:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcf04118690 con 0x7fcf04076480 2024-04-26T01:46:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.126+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcf04118990 con 0x7fcf04076da0 2024-04-26T01:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.127+0000 7fcf03400640 1 -- 172.21.15.28:0/185804166 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf04076da0 msgr2=0x7fcf041a4fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T01:46:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.127+0000 7fcf03e00640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf04112dd0 0x7fcf0419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.127+0000 7fcf03400640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf04076da0 0x7fcf041a4fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.127+0000 7fcf03400640 1 -- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf04112dd0 msgr2=0x7fcf0419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.127+0000 7fcf03400640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf04112dd0 0x7fcf0419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.127+0000 7fcf03400640 1 -- 172.21.15.28:0/185804166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0419f3f0 con 0x7fcf04076480 2024-04-26T01:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.127+0000 7fcf03e00640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf04112dd0 0x7fcf0419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.128+0000 7fcf03400640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04076480 0x7fcf041a4a70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e9f0 tx=0x7fcef400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.128+0000 7fcefbe00640 1 -- 172.21.15.28:0/185804166 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef400cd80 con 0x7fcf04076480 2024-04-26T01:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.128+0000 7fcefbe00640 1 -- 172.21.15.28:0/185804166 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcef40054d0 con 0x7fcf04076480 2024-04-26T01:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.128+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0419f6e0 con 0x7fcf04076480 2024-04-26T01:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.128+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf041ab9f0 con 0x7fcf04076480 2024-04-26T01:46:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.129+0000 7fcefbe00640 1 -- 172.21.15.28:0/185804166 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4010690 con 0x7fcf04076480 2024-04-26T01:46:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.135+0000 7fcefbe00640 1 -- 172.21.15.28:0/185804166 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcef4004e90 con 0x7fcf04076480 2024-04-26T01:46:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.136+0000 7fcf0d593740 1 Processor -- start 2024-04-26T01:46:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.137+0000 7fcf0d593740 1 -- start start 2024-04-26T01:46:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.137+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2010c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.137+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc201b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.137+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc201ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.137+0000 7fcf0d593740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560fc1f76900 con 0x560fc201ba00 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.137+0000 7fcf0d593740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560fc1f1cef0 con 0x560fc2011140 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.137+0000 7fcf0d593740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x560fc1f728c0 con 0x560fc2010820 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc201b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc201b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43472/0 (socket says 172.21.15.28:43472) 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03400640 1 -- 172.21.15.28:0/28220029 learned_addr learned my addr 172.21.15.28:0/28220029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf02a00640 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc201ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03e00640 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2010c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03400640 1 -- 172.21.15.28:0/28220029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 msgr2=0x560fc2010c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03400640 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2010c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03400640 1 -- 172.21.15.28:0/28220029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 msgr2=0x560fc201ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03400640 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc201ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf03400640 1 -- 172.21.15.28:0/28220029 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fc1c89210 con 0x560fc2011140 2024-04-26T01:46:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcefbe00640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcee80a3dc0 0x7fcee80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.138+0000 7fcf02a00640 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc201ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.139+0000 7fcefbe00640 1 -- 172.21.15.28:0/185804166 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcef40d0d10 con 0x7fcf04076480 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.139+0000 7fcf03400640 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc201b4c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcef40ce400 tx=0x7fcef40d4dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfd1d184d0e07165 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.139+0000 7fcf03e00640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcee80a3dc0 0x7fcee80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.139+0000 7fcef8c00640 1 -- 172.21.15.28:0/28220029 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef409cab0 con 0x560fc2011140 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.139+0000 7fcef8c00640 1 -- 172.21.15.28:0/28220029 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcef4005680 con 0x560fc2011140 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.139+0000 7fcef8c00640 1 -- 172.21.15.28:0/28220029 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4005180 con 0x560fc2011140 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 -- 172.21.15.28:0/28220029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 msgr2=0x560fc201b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc201b4c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcef40ce400 tx=0x7fcef40d4dd0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf03e00640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcee80a3dc0 0x7fcee80a6280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009430 tx=0x7fcefc004a40 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 -- 172.21.15.28:0/28220029 shutdown_connections 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc201ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc201b4c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 --2- 172.21.15.28:0/28220029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2010c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 -- 172.21.15.28:0/28220029 >> 172.21.15.28:0/28220029 conn(0x560fc200cd30 msgr2=0x560fc200d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 -- 172.21.15.28:0/28220029 shutdown_connections 2024-04-26T01:46:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 -- 172.21.15.28:0/28220029 wait complete. 2024-04-26T01:46:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.140+0000 7fcf0d593740 1 -- start start 2024-04-26T01:46:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.142+0000 7fcf0d593740 1 Processor -- start 2024-04-26T01:46:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.143+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc20189c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.143+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2011140 0x560fc2018f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.143+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc201ba00 0x560fc2019440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.143+0000 7fcf0d593740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560fc201e630 con 0x560fc2011140 2024-04-26T01:46:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.143+0000 7fcf0d593740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560fc1f30310 con 0x560fc201ba00 2024-04-26T01:46:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.143+0000 7fcf0d593740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x560fc201e7b0 con 0x560fc2010820 2024-04-26T01:46:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.143+0000 7fcf0d593740 10 client.0 _ll_register_callbacks cb 0x560fc1efecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.143+0000 7fcf02a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2011140 0x560fc2018f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc20189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc20189c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54158/0 (socket says 172.21.15.28:54158) 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03400640 1 -- 172.21.15.28:0/3326119237 learned_addr learned my addr 172.21.15.28:0/3326119237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03e00640 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc201ba00 0x560fc2019440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03400640 1 -- 172.21.15.28:0/3326119237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc201ba00 msgr2=0x560fc2019440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03400640 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc201ba00 0x560fc2019440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03400640 1 -- 172.21.15.28:0/3326119237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2011140 msgr2=0x560fc2018f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03400640 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2011140 0x560fc2018f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.144+0000 7fcf03400640 1 -- 172.21.15.28:0/3326119237 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fc1d55620 con 0x560fc2010820 2024-04-26T01:46:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.145+0000 7fceed600640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:46:46.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.145+0000 7fcf03400640 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc20189c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcef40996b0 tx=0x7fcef40ce430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1790889045a9c518 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:46.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.145+0000 7fceed600640 1 -- 172.21.15.28:0/3326119237 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef409c640 con 0x560fc2010820 2024-04-26T01:46:46.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.145+0000 7fcf0d593740 1 -- 172.21.15.28:0/3326119237 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcef4005680 con 0x560fc2010820 2024-04-26T01:46:46.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.145+0000 7fceed600640 1 -- 172.21.15.28:0/3326119237 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcef409afe0 con 0x560fc2010820 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.146+0000 7fceed600640 1 -- 172.21.15.28:0/3326119237 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcef409c640 con 0x560fc2010820 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.146+0000 7fcf0d593740 10 client.25807 fetch_fsmap learned FSMap version 34 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.146+0000 7fcf0d593740 1 -- 172.21.15.28:0/3326119237 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560fc201e630 con 0x560fc2010820 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fceed600640 1 -- 172.21.15.28:0/3326119237 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcef4005680 con 0x560fc2010820 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fceed600640 1 -- 172.21.15.28:0/3326119237 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fcef4099350 con 0x560fc2010820 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fcf0d593740 10 client.25807 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fcf0d593740 10 client.25807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fcf0d593740 20 client.25807 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560fc202ba40 0x560fc202de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fcf0d593740 4 client.25807 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fcf0d593740 1 -- 172.21.15.28:0/3326119237 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fc1f8a8b0 con 0x560fc202ba40 2024-04-26T01:46:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.147+0000 7fcf02a00640 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560fc202ba40 0x560fc202de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.148+0000 7fcf02a00640 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560fc202ba40 0x560fc202de30 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ec8ca0de5715c57 server_cookie=7f75e78bad935d75 in_seq=0 out_seq=0 2024-04-26T01:46:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.148+0000 7fceed600640 10 client.25807 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fceed600640 1 -- 172.21.15.28:0/3326119237 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560fc1f8a8b0 con 0x560fc202ba40 2024-04-26T01:46:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fceed600640 10 client.25807 handle_command_reply: tid=0 2024-04-26T01:46:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 client.25807 shutdown 2024-04-26T01:46:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 -- 172.21.15.28:0/3326119237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 msgr2=0x560fc20189c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc20189c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcef40996b0 tx=0x7fcef40ce430 comp rx=0 tx=0).stop 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 -- 172.21.15.28:0/3326119237 shutdown_connections 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560fc202ba40 0x560fc202de30 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fceed600640 0 client.25807 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc201ba00 0x560fc2019440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2011140 0x560fc2018f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3326119237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc20189c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.149+0000 7fcf0d593740 1 -- 172.21.15.28:0/3326119237 >> 172.21.15.28:0/3326119237 conn(0x560fc200cd30 msgr2=0x560fc2012a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.150+0000 7fcf0d593740 1 -- 172.21.15.28:0/3326119237 shutdown_connections 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.150+0000 7fcf0d593740 1 -- 172.21.15.28:0/3326119237 wait complete. 2024-04-26T01:46:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.150+0000 7fcf0d593740 20 client.25807 trim_cache size 0 max 16384 2024-04-26T01:46:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.177+0000 7fcf0d593740 1 Processor -- start 2024-04-26T01:46:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.177+0000 7fcf0d593740 1 -- start start 2024-04-26T01:46:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.177+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2010820 0x560fc20274f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.177+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc200bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.177+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc201ba00 0x560fc2033550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.177+0000 7fcf0d593740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041b6a60 con 0x560fc2010820 2024-04-26T01:46:46.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.177+0000 7fcf0d593740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041b42d0 con 0x560fc2011140 2024-04-26T01:46:46.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.177+0000 7fcf0d593740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041b2120 con 0x560fc201ba00 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.178+0000 7fcf03e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc201ba00 0x560fc2033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.178+0000 7fcf02a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc200bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.178+0000 7fcf03e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc201ba00 0x560fc2033550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54174/0 (socket says 172.21.15.28:54174) 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.178+0000 7fcf03e00640 1 -- 172.21.15.28:0/3003048127 learned_addr learned my addr 172.21.15.28:0/3003048127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.178+0000 7fcf03400640 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2010820 0x560fc20274f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fcf03e00640 1 -- 172.21.15.28:0/3003048127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 msgr2=0x560fc200bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fcf03e00640 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc200bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fcf03e00640 1 -- 172.21.15.28:0/3003048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2010820 msgr2=0x560fc20274f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fcf03e00640 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2010820 0x560fc20274f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fcf03e00640 1 -- 172.21.15.28:0/3003048127 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fc2008440 con 0x560fc201ba00 2024-04-26T01:46:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fcf03400640 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2010820 0x560fc20274f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fcf03e00640 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc201ba00 0x560fc2033550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcefc035000 tx=0x7fcefc03f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f58494d3d09414d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fceefe00640 1 -- 172.21.15.28:0/3003048127 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc003b70 con 0x560fc201ba00 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fceefe00640 1 -- 172.21.15.28:0/3003048127 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcefc036040 con 0x560fc201ba00 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.179+0000 7fceefe00640 1 -- 172.21.15.28:0/3003048127 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0307e0 con 0x560fc201ba00 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.180+0000 7fcf0d593740 1 -- 172.21.15.28:0/3003048127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc201ba00 msgr2=0x560fc2033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.180+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc201ba00 0x560fc2033550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcefc035000 tx=0x7fcefc03f000 comp rx=0 tx=0).stop 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.181+0000 7fcf0d593740 1 -- 172.21.15.28:0/3003048127 shutdown_connections 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.181+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc201ba00 0x560fc2033550 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.181+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc200bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.181+0000 7fcf0d593740 1 --2- 172.21.15.28:0/3003048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc2010820 0x560fc20274f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.181+0000 7fcf0d593740 1 -- 172.21.15.28:0/3003048127 >> 172.21.15.28:0/3003048127 conn(0x560fc200cd30 msgr2=0x560fc200b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.182+0000 7fcf0d593740 1 -- 172.21.15.28:0/3003048127 shutdown_connections 2024-04-26T01:46:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.182+0000 7fcf0d593740 1 -- 172.21.15.28:0/3003048127 wait complete. 2024-04-26T01:46:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.183+0000 7fcf0d593740 1 -- start start 2024-04-26T01:46:46.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.183+0000 7fcf0d593740 1 Processor -- start 2024-04-26T01:46:46.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.184+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2021c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.184+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc2022160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.184+0000 7fcf0d593740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc2090dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.184+0000 7fcf0d593740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041b2790 con 0x560fc201ba00 2024-04-26T01:46:46.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.184+0000 7fcf0d593740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041b3c00 con 0x560fc2011140 2024-04-26T01:46:46.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.184+0000 7fcf0d593740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041b2e60 con 0x560fc2010820 2024-04-26T01:46:46.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.184+0000 7fcf0d593740 10 client.0 _ll_register_callbacks cb 0x7fcef40d4c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc2090dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf02a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc2022160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf02a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc2022160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:43510/0 (socket says 172.21.15.28:43510) 2024-04-26T01:46:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc2090dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51318/0 (socket says 172.21.15.28:51318) 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf02a00640 1 -- 172.21.15.28:0/1029376772 learned_addr learned my addr 172.21.15.28:0/1029376772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf03400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2021c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf02a00640 1 -- 172.21.15.28:0/1029376772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 msgr2=0x560fc2021c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf02a00640 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2021c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf02a00640 1 -- 172.21.15.28:0/1029376772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 msgr2=0x560fc2090dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf02a00640 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc2090dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf02a00640 1 -- 172.21.15.28:0/1029376772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fc20913c0 con 0x560fc2011140 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.185+0000 7fcf03400640 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2021c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.186+0000 7fcf03e00640 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc2090dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.186+0000 7fcf02a00640 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc2022160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x560fc200e150 tx=0x7fcef0010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7932e9f016a342a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.186+0000 7fceed600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:46:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.187+0000 7fceed600640 1 -- 172.21.15.28:0/1029376772 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0004250 con 0x560fc2011140 2024-04-26T01:46:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.187+0000 7fceed600640 1 -- 172.21.15.28:0/1029376772 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7fcef00043f0 con 0x560fc2011140 2024-04-26T01:46:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.187+0000 7fcf0d593740 1 -- 172.21.15.28:0/1029376772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcefc036040 con 0x560fc2011140 2024-04-26T01:46:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.188+0000 7fceed600640 1 -- 172.21.15.28:0/1029376772 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcef002c050 con 0x560fc2011140 2024-04-26T01:46:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.188+0000 7fcf0d593740 10 client.25802 fetch_fsmap learned FSMap version 34 2024-04-26T01:46:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.188+0000 7fcf0d593740 1 -- 172.21.15.28:0/1029376772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560fc20926e0 con 0x560fc2011140 2024-04-26T01:46:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.188+0000 7fceed600640 1 -- 172.21.15.28:0/1029376772 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcefc036040 con 0x560fc2011140 2024-04-26T01:46:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.189+0000 7fceed600640 1 -- 172.21.15.28:0/1029376772 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fcef0003c60 con 0x560fc2011140 2024-04-26T01:46:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.189+0000 7fcf0d593740 10 client.25802 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:46:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.189+0000 7fcf0d593740 10 client.25802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.189+0000 7fcf0d593740 20 client.25802 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.189+0000 7fcf0d593740 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560fc2092ea0 0x560fc20952c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.189+0000 7fcf0d593740 4 client.25802 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.189+0000 7fcf0d593740 1 -- 172.21.15.28:0/1029376772 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560fc1f84320 con 0x560fc2092ea0 2024-04-26T01:46:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.189+0000 7fcf03400640 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560fc2092ea0 0x560fc20952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:46.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.191+0000 7fcf03400640 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560fc2092ea0 0x560fc20952c0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e8df059f72ff76 server_cookie=f29518b86ac359cc in_seq=0 out_seq=0 2024-04-26T01:46:46.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.191+0000 7fceed600640 10 client.25802 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:46.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fceed600640 1 -- 172.21.15.28:0/1029376772 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560fc1f84320 con 0x560fc2092ea0 2024-04-26T01:46:46.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fceed600640 10 client.25802 handle_command_reply: tid=0 2024-04-26T01:46:46.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 client.25802 shutdown 2024-04-26T01:46:46.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 -- 172.21.15.28:0/1029376772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 msgr2=0x560fc2022160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc2022160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x560fc200e150 tx=0x7fcef0010800 comp rx=0 tx=0).stop 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 -- 172.21.15.28:0/1029376772 shutdown_connections 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x560fc2092ea0 0x560fc20952c0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc201ba00 0x560fc2090dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560fc2011140 0x560fc2022160 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 --2- 172.21.15.28:0/1029376772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560fc2010820 0x560fc2021c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fcf0d593740 1 -- 172.21.15.28:0/1029376772 >> 172.21.15.28:0/1029376772 conn(0x560fc200cd30 msgr2=0x560fc200be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.206+0000 7fceed600640 0 client.25802 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.207+0000 7fcf0d593740 1 -- 172.21.15.28:0/1029376772 shutdown_connections 2024-04-26T01:46:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.207+0000 7fcf0d593740 1 -- 172.21.15.28:0/1029376772 wait complete. 2024-04-26T01:46:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.207+0000 7fcf0d593740 20 client.25802 trim_cache size 0 max 16384 2024-04-26T01:46:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:46:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:46:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:46:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:46:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcee80a3dc0 msgr2=0x7fcee80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcee80a3dc0 0x7fcee80a6280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009430 tx=0x7fcefc004a40 comp rx=0 tx=0).stop 2024-04-26T01:46:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04076480 msgr2=0x7fcf041a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04076480 0x7fcf041a4a70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e9f0 tx=0x7fcef400edf0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 shutdown_connections 2024-04-26T01:46:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcee80a3dc0 0x7fcee80a6280 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf04112dd0 0x7fcf0419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf04076da0 0x7fcf041a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 --2- 172.21.15.28:0/185804166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf04076480 0x7fcf041a4a70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 >> 172.21.15.28:0/185804166 conn(0x7fcf040ffc40 msgr2=0x7fcf04113a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 shutdown_connections 2024-04-26T01:46:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:46.208+0000 7fcf0a800640 1 -- 172.21.15.28:0/185804166 wait complete. 2024-04-26T01:46:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 290.0 s, 2213.1 tps, lat 44.717 ms stddev 56.132 2024-04-26T01:46:47.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:47 smithi028 ceph-mon[28247]: pgmap v177: 97 pgs: 97 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 752 op/s 2024-04-26T01:46:47.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:47 smithi169 ceph-mon[32851]: pgmap v177: 97 pgs: 97 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 752 op/s 2024-04-26T01:46:47.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:47 smithi053 ceph-mon[32593]: pgmap v177: 97 pgs: 97 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 752 op/s 2024-04-26T01:46:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:49 smithi028 ceph-mon[28247]: pgmap v178: 97 pgs: 97 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 680 op/s 2024-04-26T01:46:49.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:49 smithi169 ceph-mon[32851]: pgmap v178: 97 pgs: 97 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 680 op/s 2024-04-26T01:46:49.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:49 smithi053 ceph-mon[32593]: pgmap v178: 97 pgs: 97 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 680 op/s 2024-04-26T01:46:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 295.0 s, 2117.8 tps, lat 47.621 ms stddev 73.829 2024-04-26T01:46:51.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:51 smithi028 ceph-mon[28247]: pgmap v179: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 28 MiB/s wr, 1.02k op/s 2024-04-26T01:46:51.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:51 smithi169 ceph-mon[32851]: pgmap v179: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 28 MiB/s wr, 1.02k op/s 2024-04-26T01:46:51.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:51 smithi053 ceph-mon[32593]: pgmap v179: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 28 MiB/s wr, 1.02k op/s 2024-04-26T01:46:53.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:53 smithi028 ceph-mon[28247]: pgmap v180: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 727 op/s 2024-04-26T01:46:53.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:53 smithi169 ceph-mon[32851]: pgmap v180: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 727 op/s 2024-04-26T01:46:53.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:53 smithi053 ceph-mon[32593]: pgmap v180: 97 pgs: 97 active+clean; 2.7 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 727 op/s 2024-04-26T01:46:55.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:55 smithi028 ceph-mon[28247]: pgmap v181: 97 pgs: 97 active+clean; 2.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 36 MiB/s wr, 938 op/s 2024-04-26T01:46:55.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:55 smithi169 ceph-mon[32851]: pgmap v181: 97 pgs: 97 active+clean; 2.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 36 MiB/s wr, 938 op/s 2024-04-26T01:46:55.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:55 smithi053 ceph-mon[32593]: pgmap v181: 97 pgs: 97 active+clean; 2.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 36 MiB/s wr, 938 op/s 2024-04-26T01:46:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 300.0 s, 2268.4 tps, lat 44.169 ms stddev 57.628 2024-04-26T01:46:57.520 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:57 smithi053 ceph-mon[32593]: pgmap v182: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 35 MiB/s wr, 749 op/s 2024-04-26T01:46:57.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:57 smithi028 ceph-mon[28247]: pgmap v182: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 35 MiB/s wr, 749 op/s 2024-04-26T01:46:57.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:57 smithi169 ceph-mon[32851]: pgmap v182: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 35 MiB/s wr, 749 op/s 2024-04-26T01:46:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 -- 172.21.15.28:0/838388775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 msgr2=0x7f19c410ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 --2- 172.21.15.28:0/838388775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 0x7f19c410ee70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009ca0 tx=0x7f19bc02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 -- 172.21.15.28:0/838388775 shutdown_connections 2024-04-26T01:46:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 --2- 172.21.15.28:0/838388775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 0x7f19c410ee70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 --2- 172.21.15.28:0/838388775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c4102b80 0x7f19c4106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 --2- 172.21.15.28:0/838388775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c41021d0 0x7f19c41025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 -- 172.21.15.28:0/838388775 >> 172.21.15.28:0/838388775 conn(0x7f19c40793e0 msgr2=0x7f19c40797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 -- 172.21.15.28:0/838388775 shutdown_connections 2024-04-26T01:46:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.394+0000 7f19c8a00640 1 -- 172.21.15.28:0/838388775 wait complete. 2024-04-26T01:46:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.398+0000 7f19c8a00640 1 Processor -- start 2024-04-26T01:46:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.398+0000 7f19c8a00640 1 -- start start 2024-04-26T01:46:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c41021d0 0x7f19c41a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c8a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 0x7f19c41a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c419a6a0 0x7f19c419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19c4112240 con 0x7f19c41021d0 2024-04-26T01:46:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f19c41120c0 con 0x7f19c419a6a0 2024-04-26T01:46:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c8a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f19c41123c0 con 0x7f19c41072a0 2024-04-26T01:46:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c1400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 0x7f19c41a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c419a6a0 0x7f19c419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c1400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 0x7f19c41a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51764/0 (socket says 172.21.15.28:51764) 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c1400640 1 -- 172.21.15.28:0/2511132021 learned_addr learned my addr 172.21.15.28:0/2511132021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.399+0000 7f19c1e00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c41021d0 0x7f19c41a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.400+0000 7f19c2800640 1 -- 172.21.15.28:0/2511132021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 msgr2=0x7f19c41a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.400+0000 7f19c2800640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 0x7f19c41a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.400+0000 7f19c2800640 1 -- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c41021d0 msgr2=0x7f19c41a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.400+0000 7f19c2800640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c41021d0 0x7f19c41a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.400+0000 7f19c2800640 1 -- 172.21.15.28:0/2511132021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c419b2a0 con 0x7f19c419a6a0 2024-04-26T01:46:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.400+0000 7f19c1400640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 0x7f19c41a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.400+0000 7f19c2800640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c419a6a0 0x7f19c419ab20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f19bc005e30 tx=0x7f19bc0317c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.401+0000 7f19baa00640 1 -- 172.21.15.28:0/2511132021 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc003b70 con 0x7f19c419a6a0 2024-04-26T01:46:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.401+0000 7f19baa00640 1 -- 172.21.15.28:0/2511132021 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19bc003d10 con 0x7f19c419a6a0 2024-04-26T01:46:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.401+0000 7f19baa00640 1 -- 172.21.15.28:0/2511132021 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc02f970 con 0x7f19c419a6a0 2024-04-26T01:46:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.401+0000 7f19c8a00640 1 -- 172.21.15.28:0/2511132021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19c419b530 con 0x7f19c419a6a0 2024-04-26T01:46:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.401+0000 7f19c8a00640 1 -- 172.21.15.28:0/2511132021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19c41a7630 con 0x7f19c419a6a0 2024-04-26T01:46:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.404+0000 7f19baa00640 1 -- 172.21.15.28:0/2511132021 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f19bc02fb10 con 0x7f19c419a6a0 2024-04-26T01:46:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.406+0000 7f19baa00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f19a80a3f20 0x7f19a80a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.407+0000 7f19baa00640 1 -- 172.21.15.28:0/2511132021 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f19bc0f1a50 con 0x7f19c419a6a0 2024-04-26T01:46:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.410+0000 7f19c1e00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f19a80a3f20 0x7f19a80a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.410+0000 7f19cb9b0740 1 Processor -- start 2024-04-26T01:46:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.411+0000 7f19c1e00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f19a80a3f20 0x7f19a80a63e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009bd0 tx=0x7f19b4005f90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.412+0000 7f19cb9b0740 1 -- start start 2024-04-26T01:46:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc96e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc978cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc979210 0x55ddcc97b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddcc8cb900 con 0x55ddcc979210 2024-04-26T01:46:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ddcc871ef0 con 0x55ddcc96e950 2024-04-26T01:46:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55ddcc8c78c0 con 0x55ddcc96e030 2024-04-26T01:46:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19c1e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc96e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19c1e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc96e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51772/0 (socket says 172.21.15.28:51772) 2024-04-26T01:46:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19c1e00640 1 -- 172.21.15.28:0/3037949351 learned_addr learned my addr 172.21.15.28:0/3037949351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.413+0000 7f19c1400640 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc978cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.414+0000 7f19c1e00640 1 -- 172.21.15.28:0/3037949351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 msgr2=0x55ddcc978cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.414+0000 7f19c1e00640 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc978cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.414+0000 7f19c1e00640 1 -- 172.21.15.28:0/3037949351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc979210 msgr2=0x55ddcc97b600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:46:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.414+0000 7f19c2800640 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc979210 0x55ddcc97b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.415+0000 7f19c1e00640 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc979210 0x55ddcc97b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.415+0000 7f19c1e00640 1 -- 172.21.15.28:0/3037949351 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddcc5de210 con 0x55ddcc96e030 2024-04-26T01:46:59.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.415+0000 7f19c1400640 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc978cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.415+0000 7f19c2800640 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc979210 0x55ddcc97b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.415+0000 7f19c1e00640 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc96e410 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f19b400e630 tx=0x7f19b4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57e8cfdc8bab4c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.416+0000 7f19af000640 1 -- 172.21.15.28:0/3037949351 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4004020 con 0x55ddcc96e030 2024-04-26T01:46:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.416+0000 7f19af000640 1 -- 172.21.15.28:0/3037949351 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b40041c0 con 0x55ddcc96e030 2024-04-26T01:46:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.416+0000 7f19af000640 1 -- 172.21.15.28:0/3037949351 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b401c6b0 con 0x55ddcc96e030 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.418+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3037949351 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 msgr2=0x55ddcc96e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.418+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc96e410 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f19b400e630 tx=0x7f19b4014000 comp rx=0 tx=0).stop 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.419+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3037949351 shutdown_connections 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.419+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc979210 0x55ddcc97b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.419+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc978cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.419+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/3037949351 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc96e410 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.419+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3037949351 >> 172.21.15.28:0/3037949351 conn(0x55ddcc96a540 msgr2=0x55ddcc96a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.419+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3037949351 shutdown_connections 2024-04-26T01:46:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.419+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3037949351 wait complete. 2024-04-26T01:46:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.420+0000 7f19cb9b0740 1 -- start start 2024-04-26T01:46:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.420+0000 7f19cb9b0740 1 Processor -- start 2024-04-26T01:46:59.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc976150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc976690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc979210 0x55ddcc976bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddcc97be40 con 0x55ddcc96e950 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ddcc885310 con 0x55ddcc979210 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55ddcc97bfc0 con 0x55ddcc96e030 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19cb9b0740 10 client.0 _ll_register_callbacks cb 0x55ddcc853cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19c2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc979210 0x55ddcc976bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19c1e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc976150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19c2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc979210 0x55ddcc976bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46956/0 (socket says 172.21.15.28:46956) 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.423+0000 7f19c2800640 1 -- 172.21.15.28:0/1804889701 learned_addr learned my addr 172.21.15.28:0/1804889701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.424+0000 7f19c1400640 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc976690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.424+0000 7f19c2800640 1 -- 172.21.15.28:0/1804889701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 msgr2=0x55ddcc976150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.424+0000 7f19c2800640 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc976150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.424+0000 7f19c2800640 1 -- 172.21.15.28:0/1804889701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 msgr2=0x55ddcc976690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.424+0000 7f19c2800640 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc976690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.424+0000 7f19c2800640 1 -- 172.21.15.28:0/1804889701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddcc6aa620 con 0x55ddcc979210 2024-04-26T01:46:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.424+0000 7f19c2800640 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc979210 0x55ddcc976bd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0b9f30 tx=0x7f19bc0bd4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c9694c3b1e0479 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.424+0000 7f19a3400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:46:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.425+0000 7f19a3400640 1 -- 172.21.15.28:0/1804889701 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0c1e80 con 0x55ddcc979210 2024-04-26T01:46:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.426+0000 7f19cb9b0740 1 -- 172.21.15.28:0/1804889701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19b40041c0 con 0x55ddcc979210 2024-04-26T01:46:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.425+0000 7f19a3400640 1 -- 172.21.15.28:0/1804889701 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f19bc0de070 con 0x55ddcc979210 2024-04-26T01:46:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.426+0000 7f19a3400640 1 -- 172.21.15.28:0/1804889701 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f19bc0c6740 con 0x55ddcc979210 2024-04-26T01:46:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.426+0000 7f19cb9b0740 10 client.25811 fetch_fsmap learned FSMap version 34 2024-04-26T01:46:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.426+0000 7f19cb9b0740 1 -- 172.21.15.28:0/1804889701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ddcc97be40 con 0x55ddcc979210 2024-04-26T01:46:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.427+0000 7f19a3400640 1 -- 172.21.15.28:0/1804889701 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b40041c0 con 0x55ddcc979210 2024-04-26T01:46:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.427+0000 7f19a3400640 1 -- 172.21.15.28:0/1804889701 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f19bc0d9060 con 0x55ddcc979210 2024-04-26T01:46:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.427+0000 7f19cb9b0740 10 client.25811 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:46:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.427+0000 7f19cb9b0740 10 client.25811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.427+0000 7f19cb9b0740 20 client.25811 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.427+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ddcc989230 0x55ddcc98b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.428+0000 7f19cb9b0740 4 client.25811 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.428+0000 7f19cb9b0740 1 -- 172.21.15.28:0/1804889701 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddcc8df8b0 con 0x55ddcc989230 2024-04-26T01:46:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.429+0000 7f19c1e00640 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ddcc989230 0x55ddcc98b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.430+0000 7f19c1e00640 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ddcc989230 0x55ddcc98b650 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95034a48c1f9387f server_cookie=741f8c5735f11727 in_seq=0 out_seq=0 2024-04-26T01:46:59.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.430+0000 7f19a3400640 10 client.25811 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:59.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.431+0000 7f19a3400640 1 -- 172.21.15.28:0/1804889701 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ddcc8df8b0 con 0x55ddcc989230 2024-04-26T01:46:59.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.431+0000 7f19a3400640 10 client.25811 handle_command_reply: tid=0 2024-04-26T01:46:59.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.431+0000 7f19cb9b0740 1 client.25811 shutdown 2024-04-26T01:46:59.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 -- 172.21.15.28:0/1804889701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc979210 msgr2=0x55ddcc976bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc979210 0x55ddcc976bd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0b9f30 tx=0x7f19bc0bd4b0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 -- 172.21.15.28:0/1804889701 shutdown_connections 2024-04-26T01:46:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ddcc989230 0x55ddcc98b650 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc979210 0x55ddcc976bd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc976690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19a3400640 0 client.25811 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/1804889701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc96e030 0x55ddcc976150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 -- 172.21.15.28:0/1804889701 >> 172.21.15.28:0/1804889701 conn(0x55ddcc96a540 msgr2=0x55ddcc960e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.432+0000 7f19cb9b0740 1 -- 172.21.15.28:0/1804889701 shutdown_connections 2024-04-26T01:46:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.433+0000 7f19cb9b0740 1 -- 172.21.15.28:0/1804889701 wait complete. 2024-04-26T01:46:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.433+0000 7f19cb9b0740 20 client.25811 trim_cache size 0 max 16384 2024-04-26T01:46:59.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.469+0000 7f19cb9b0740 1 Processor -- start 2024-04-26T01:46:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.469+0000 7f19cb9b0740 1 -- start start 2024-04-26T01:46:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e030 0x55ddcc984d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc960d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc990f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19c41ae220 con 0x55ddcc96e030 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f19c41aba90 con 0x55ddcc96e950 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19cb9b0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f19c41a9920 con 0x55ddcc979210 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19c1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc960d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e030 0x55ddcc984d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19c1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc960d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46960/0 (socket says 172.21.15.28:46960) 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e030 0x55ddcc984d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46526/0 (socket says 172.21.15.28:46526) 2024-04-26T01:46:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.470+0000 7f19c1e00640 1 -- 172.21.15.28:0/3686203684 learned_addr learned my addr 172.21.15.28:0/3686203684 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.471+0000 7f19c2800640 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc990f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.471+0000 7f19c1e00640 1 -- 172.21.15.28:0/3686203684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 msgr2=0x55ddcc990f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.471+0000 7f19c1e00640 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc990f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.471+0000 7f19c1e00640 1 -- 172.21.15.28:0/3686203684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 msgr2=0x55ddcc960d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.471+0000 7f19c1e00640 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc960d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.471+0000 7f19c1e00640 1 -- 172.21.15.28:0/3686203684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddcc98b9c0 con 0x55ddcc96e030 2024-04-26T01:46:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.471+0000 7f19c2800640 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc990f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.471+0000 7f19c1e00640 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e030 0x55ddcc984d00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009460 tx=0x7f19b4003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f5ed1cb36f7835 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.473+0000 7f19ae600640 1 -- 172.21.15.28:0/3686203684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b400ee20 con 0x55ddcc96e030 2024-04-26T01:46:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.473+0000 7f19ae600640 1 -- 172.21.15.28:0/3686203684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b40035f0 con 0x55ddcc96e030 2024-04-26T01:46:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.473+0000 7f19ae600640 1 -- 172.21.15.28:0/3686203684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4004680 con 0x55ddcc96e030 2024-04-26T01:46:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.474+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3686203684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e030 msgr2=0x55ddcc984d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.474+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e030 0x55ddcc984d00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009460 tx=0x7f19b4003e20 comp rx=0 tx=0).stop 2024-04-26T01:46:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.474+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3686203684 shutdown_connections 2024-04-26T01:46:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.474+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc990f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.474+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e950 0x55ddcc960d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.474+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/3686203684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e030 0x55ddcc984d00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.474+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3686203684 >> 172.21.15.28:0/3686203684 conn(0x55ddcc96a540 msgr2=0x55ddcc960600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.474+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3686203684 shutdown_connections 2024-04-26T01:46:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.475+0000 7f19cb9b0740 1 -- 172.21.15.28:0/3686203684 wait complete. 2024-04-26T01:46:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.475+0000 7f19cb9b0740 1 -- start start 2024-04-26T01:46:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.475+0000 7f19cb9b0740 1 Processor -- start 2024-04-26T01:46:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.477+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e030 0x55ddcc9815a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.477+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc97e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.477+0000 7f19cb9b0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc97ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.477+0000 7f19c1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e030 0x55ddcc9815a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19c1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e030 0x55ddcc9815a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46970/0 (socket says 172.21.15.28:46970) 2024-04-26T01:46:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19c2800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc97ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19c2800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc97ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51800/0 (socket says 172.21.15.28:51800) 2024-04-26T01:46:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19c2800640 1 -- 172.21.15.28:0/2273159726 learned_addr learned my addr 172.21.15.28:0/2273159726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:46:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19c41a9f70 con 0x55ddcc96e950 2024-04-26T01:46:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19c1400640 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc97e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f19c41ab3c0 con 0x55ddcc96e030 2024-04-26T01:46:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f19c41aa620 con 0x55ddcc979210 2024-04-26T01:46:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.478+0000 7f19cb9b0740 10 client.0 _ll_register_callbacks cb 0x7f19bc0bd970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.479+0000 7f19c1400640 1 -- 172.21.15.28:0/2273159726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 msgr2=0x55ddcc97ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.480+0000 7f19c1400640 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc97ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.480+0000 7f19c1400640 1 -- 172.21.15.28:0/2273159726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e030 msgr2=0x55ddcc9815a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.480+0000 7f19c1400640 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e030 0x55ddcc9815a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.480+0000 7f19c1400640 1 -- 172.21.15.28:0/2273159726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddcc97f680 con 0x55ddcc96e950 2024-04-26T01:46:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.480+0000 7f19c2800640 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc97ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.480+0000 7f19c1e00640 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e030 0x55ddcc9815a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.481+0000 7f19c1400640 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc97e930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55ddcc9759b0 tx=0x7f19b00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47f3e8ae9cc4d94 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.481+0000 7f19a3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:46:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.482+0000 7f19a3400640 1 -- 172.21.15.28:0/2273159726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b0005020 con 0x55ddcc96e950 2024-04-26T01:46:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.483+0000 7f19a3400640 1 -- 172.21.15.28:0/2273159726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8469+0+0 (secure 0 0 0) 0x7f19b00051c0 con 0x55ddcc96e950 2024-04-26T01:46:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.483+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19b40035f0 con 0x55ddcc96e950 2024-04-26T01:46:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.484+0000 7f19a3400640 1 -- 172.21.15.28:0/2273159726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f19b00053c0 con 0x55ddcc96e950 2024-04-26T01:46:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.484+0000 7f19cb9b0740 10 client.15624 fetch_fsmap learned FSMap version 34 2024-04-26T01:46:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.484+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ddcc9ef7a0 con 0x55ddcc96e950 2024-04-26T01:46:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.485+0000 7f19a3400640 1 -- 172.21.15.28:0/2273159726 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b40035f0 con 0x55ddcc96e950 2024-04-26T01:46:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.485+0000 7f19a3400640 1 -- 172.21.15.28:0/2273159726 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f19b00139e0 con 0x55ddcc96e950 2024-04-26T01:46:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.485+0000 7f19cb9b0740 10 client.15624 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:46:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.485+0000 7f19cb9b0740 10 client.15624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.485+0000 7f19cb9b0740 20 client.15624 populate_metadata read hostname 'smithi028' 2024-04-26T01:46:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.485+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ddcc9eff00 0x55ddcc9f2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.485+0000 7f19cb9b0740 4 client.15624 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.485+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ddcc8d9320 con 0x55ddcc9eff00 2024-04-26T01:46:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.486+0000 7f19c1e00640 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ddcc9eff00 0x55ddcc9f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.487+0000 7f19c1e00640 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ddcc9eff00 0x55ddcc9f2320 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9a90b3d2ba083e9 server_cookie=887178ce81a0eeb1 in_seq=0 out_seq=0 2024-04-26T01:46:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.487+0000 7f19a3400640 10 client.15624 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.505+0000 7f19a3400640 1 -- 172.21.15.28:0/2273159726 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ddcc8d9320 con 0x55ddcc9eff00 2024-04-26T01:46:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.505+0000 7f19a3400640 10 client.15624 handle_command_reply: tid=0 2024-04-26T01:46:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.505+0000 7f19cb9b0740 1 client.15624 shutdown 2024-04-26T01:46:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 msgr2=0x55ddcc97e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc97e930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55ddcc9759b0 tx=0x7f19b00039c0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 shutdown_connections 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ddcc9eff00 0x55ddcc9f2320 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ddcc979210 0x55ddcc97ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcc96e950 0x55ddcc97e930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 --2- 172.21.15.28:0/2273159726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ddcc96e030 0x55ddcc9815a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 >> 172.21.15.28:0/2273159726 conn(0x55ddcc96a540 msgr2=0x55ddcc9769a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19a3400640 0 client.15624 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.506+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 shutdown_connections 2024-04-26T01:46:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.507+0000 7f19cb9b0740 1 -- 172.21.15.28:0/2273159726 wait complete. 2024-04-26T01:46:59.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:46:59.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:46:59.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:46:59.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.507+0000 7f19cb9b0740 20 client.15624 trim_cache size 0 max 16384 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 -- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f19a80a3f20 msgr2=0x7f19a80a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f19a80a3f20 0x7f19a80a63e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009bd0 tx=0x7f19b4005f90 comp rx=0 tx=0).stop 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 -- 172.21.15.28:0/2511132021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c419a6a0 msgr2=0x7f19c419ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c419a6a0 0x7f19c419ab20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f19bc005e30 tx=0x7f19bc0317c0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 -- 172.21.15.28:0/2511132021 shutdown_connections 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f19a80a3f20 0x7f19a80a63e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c419a6a0 0x7f19c419ab20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c41072a0 0x7f19c41a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 --2- 172.21.15.28:0/2511132021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c41021d0 0x7f19c41a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.508+0000 7f19c8a00640 1 -- 172.21.15.28:0/2511132021 >> 172.21.15.28:0/2511132021 conn(0x7f19c40793e0 msgr2=0x7f19c40ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.509+0000 7f19c8a00640 1 -- 172.21.15.28:0/2511132021 shutdown_connections 2024-04-26T01:46:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:46:59.509+0000 7f19c8a00640 1 -- 172.21.15.28:0/2511132021 wait complete. 2024-04-26T01:46:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:46:59 smithi028 ceph-mon[28247]: pgmap v183: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 31 MiB/s wr, 630 op/s 2024-04-26T01:46:59.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:46:59 smithi169 ceph-mon[32851]: pgmap v183: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 31 MiB/s wr, 630 op/s 2024-04-26T01:46:59.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:46:59 smithi053 ceph-mon[32593]: pgmap v183: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 31 MiB/s wr, 630 op/s 2024-04-26T01:46:59.850 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-26T01:46:59.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:47:00.469 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 305.0 s, 2176.4 tps, lat 44.028 ms stddev 69.696 2024-04-26T01:47:01.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:01 smithi028 ceph-mon[28247]: pgmap v184: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 44 MiB/s wr, 925 op/s 2024-04-26T01:47:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:01 smithi169 ceph-mon[32851]: pgmap v184: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 44 MiB/s wr, 925 op/s 2024-04-26T01:47:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:01 smithi053 ceph-mon[32593]: pgmap v184: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 44 MiB/s wr, 925 op/s 2024-04-26T01:47:02.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.855+0000 7f4ffe800700 1 -- 172.21.15.28:0/1893769294 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ff810f4a0 msgr2=0x7f4ff8111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.855+0000 7f4ffe800700 1 --2- 172.21.15.28:0/1893769294 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ff810f4a0 0x7f4ff8111890 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000b0a0 tx=0x7f4ff000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.856+0000 7f4ffe800700 1 -- 172.21.15.28:0/1893769294 shutdown_connections 2024-04-26T01:47:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.856+0000 7f4ffe800700 1 --2- 172.21.15.28:0/1893769294 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ff810f4a0 0x7f4ff8111890 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.856+0000 7f4ffe800700 1 --2- 172.21.15.28:0/1893769294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff8074c70 0x7f4ff8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.856+0000 7f4ffe800700 1 --2- 172.21.15.28:0/1893769294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ff8075590 0x7f4ff810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.856+0000 7f4ffe800700 1 -- 172.21.15.28:0/1893769294 >> 172.21.15.28:0/1893769294 conn(0x7f4ff8079ba0 msgr2=0x7f4ff8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.857+0000 7f4ffe800700 1 -- 172.21.15.28:0/1893769294 shutdown_connections 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.857+0000 7f4ffe800700 1 -- 172.21.15.28:0/1893769294 wait complete. 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.859+0000 7f4ffe800700 1 Processor -- start 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.860+0000 7f4ffe800700 1 -- start start 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.860+0000 7f4ffe800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ff8074c70 0x7f4ff810e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.860+0000 7f4ffe800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff8075590 0x7f4ff810eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.860+0000 7f4ffe800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ff810f4a0 0x7f4ff8105b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.860+0000 7f4ffe800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff8104670 con 0x7f4ff8075590 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.860+0000 7f4ffe800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff81044f0 con 0x7f4ff810f4a0 2024-04-26T01:47:02.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.860+0000 7f4ffe800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff81047f0 con 0x7f4ff8074c70 2024-04-26T01:47:02.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.860+0000 7f4fefe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff8075590 0x7f4ff810eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.861+0000 7f4fefe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff8075590 0x7f4ff810eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46548/0 (socket says 172.21.15.28:46548) 2024-04-26T01:47:02.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.861+0000 7f4fefe00700 1 -- 172.21.15.28:0/476420912 learned_addr learned my addr 172.21.15.28:0/476420912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.861+0000 7f4ff7e00700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ff810f4a0 0x7f4ff8105b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.861+0000 7f4ff7400700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ff8074c70 0x7f4ff810e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.861+0000 7f4ff7e00700 1 -- 172.21.15.28:0/476420912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ff8074c70 msgr2=0x7f4ff810e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.861+0000 7f4ff7e00700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ff8074c70 0x7f4ff810e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.861+0000 7f4ff7e00700 1 -- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff8075590 msgr2=0x7f4ff810eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.861+0000 7f4ff7e00700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff8075590 0x7f4ff810eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.862+0000 7f4ff7e00700 1 -- 172.21.15.28:0/476420912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff81063f0 con 0x7f4ff810f4a0 2024-04-26T01:47:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.862+0000 7f4ff7e00700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ff810f4a0 0x7f4ff8105b80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000b070 tx=0x7f4ff0004d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.863+0000 7f4ff4c00700 1 -- 172.21.15.28:0/476420912 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0021070 con 0x7f4ff810f4a0 2024-04-26T01:47:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.863+0000 7f4ff4c00700 1 -- 172.21.15.28:0/476420912 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ff0007e30 con 0x7f4ff810f4a0 2024-04-26T01:47:02.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.863+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff8106680 con 0x7f4ff810f4a0 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.864+0000 7f4ff4c00700 1 -- 172.21.15.28:0/476420912 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff00045f0 con 0x7f4ff810f4a0 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.865+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff810a5b0 con 0x7f4ff810f4a0 2024-04-26T01:47:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.867+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff8076330 con 0x7f4ff810f4a0 2024-04-26T01:47:02.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.870+0000 7f4ff4c00700 1 -- 172.21.15.28:0/476420912 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4ff0007980 con 0x7f4ff810f4a0 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.872+0000 7f4ff4c00700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4fdc0a3fb0 0x7f4fdc0a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.872+0000 7f4ff4c00700 1 -- 172.21.15.28:0/476420912 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8557+0+0 (secure 0 0 0) 0x7f4ff00d09f0 con 0x7f4ff810f4a0 2024-04-26T01:47:02.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.874+0000 7f4ff7400700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4fdc0a3fb0 0x7f4fdc0a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.876+0000 7f4ff7400700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4fdc0a3fb0 0x7f4fdc0a6470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009bf0 tx=0x7f4fe8009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:02.876+0000 7f4ff4c00700 1 -- 172.21.15.28:0/476420912 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ff009d0a0 con 0x7f4ff810f4a0 2024-04-26T01:47:03.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.248+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ff810a810 con 0x7f4ff810f4a0 2024-04-26T01:47:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.249+0000 7f4ff4c00700 1 -- 172.21.15.28:0/476420912 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11867 (secure 0 0 0) 0x7f4ff009c9a0 con 0x7f4ff810f4a0 2024-04-26T01:47:03.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:47:03.248 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:46:17.099601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:47:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.253+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4fdc0a3fb0 msgr2=0x7f4fdc0a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.253+0000 7f4ffe800700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4fdc0a3fb0 0x7f4fdc0a6470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009bf0 tx=0x7f4fe8009260 comp rx=0 tx=0).stop 2024-04-26T01:47:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.253+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ff810f4a0 msgr2=0x7f4ff8105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.253+0000 7f4ffe800700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ff810f4a0 0x7f4ff8105b80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000b070 tx=0x7f4ff0004d30 comp rx=0 tx=0).stop 2024-04-26T01:47:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.254+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 shutdown_connections 2024-04-26T01:47:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.254+0000 7f4ffe800700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4fdc0a3fb0 0x7f4fdc0a6470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.254+0000 7f4ffe800700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ff810f4a0 0x7f4ff8105b80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.254+0000 7f4ffe800700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ff8074c70 0x7f4ff810e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.254+0000 7f4ffe800700 1 --2- 172.21.15.28:0/476420912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff8075590 0x7f4ff810eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.254+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 >> 172.21.15.28:0/476420912 conn(0x7f4ff8079ba0 msgr2=0x7f4ff80ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.254+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 shutdown_connections 2024-04-26T01:47:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:03.254+0000 7f4ffe800700 1 -- 172.21.15.28:0/476420912 wait complete. 2024-04-26T01:47:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-04-26T01:47:03.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:03 smithi028 ceph-mon[28247]: pgmap v185: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 28 MiB/s wr, 588 op/s 2024-04-26T01:47:03.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:03 smithi028 ceph-mon[28247]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:47:03.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:03 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/476420912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:03 smithi169 ceph-mon[32851]: pgmap v185: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 28 MiB/s wr, 588 op/s 2024-04-26T01:47:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:03 smithi169 ceph-mon[32851]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:47:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:03 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/476420912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:03 smithi053 ceph-mon[32593]: pgmap v185: 97 pgs: 97 active+clean; 2.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 28 MiB/s wr, 588 op/s 2024-04-26T01:47:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:03 smithi053 ceph-mon[32593]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:47:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:03 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/476420912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:04.582 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 70 --await-for 42.0' 2024-04-26T01:47:04.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 70 --await-for 42.0 2024-04-26T01:47:05.078 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:05.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:05 smithi028 ceph-mon[28247]: pgmap v187: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 23 MiB/s wr, 630 op/s 2024-04-26T01:47:05.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:05 smithi169 ceph-mon[32851]: pgmap v187: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 23 MiB/s wr, 630 op/s 2024-04-26T01:47:05.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:05 smithi053 ceph-mon[32593]: pgmap v187: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 23 MiB/s wr, 630 op/s 2024-04-26T01:47:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 310.0 s, 168.7 tps, lat 571.455 ms stddev 373.057 2024-04-26T01:47:07.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:07 smithi028 ceph-mon[28247]: pgmap v188: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 MiB/s wr, 601 op/s 2024-04-26T01:47:07.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:07 smithi169 ceph-mon[32851]: pgmap v188: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 MiB/s wr, 601 op/s 2024-04-26T01:47:07.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:07 smithi053 ceph-mon[32593]: pgmap v188: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 MiB/s wr, 601 op/s 2024-04-26T01:47:08.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 -- 172.21.15.28:0/1729920095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 msgr2=0x7f1c58107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/1729920095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 0x7f1c58107d10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009990 tx=0x7f1c4c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 -- 172.21.15.28:0/1729920095 shutdown_connections 2024-04-26T01:47:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/1729920095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c58102300 0x7f1c5810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/1729920095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 0x7f1c58107d10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/1729920095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c58101940 0x7f1c58101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 -- 172.21.15.28:0/1729920095 >> 172.21.15.28:0/1729920095 conn(0x7f1c580788e0 msgr2=0x7f1c58078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 -- 172.21.15.28:0/1729920095 shutdown_connections 2024-04-26T01:47:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.070+0000 7f1c5fe00700 1 -- 172.21.15.28:0/1729920095 wait complete. 2024-04-26T01:47:08.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.072+0000 7f1c5fe00700 1 Processor -- start 2024-04-26T01:47:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.072+0000 7f1c5fe00700 1 -- start start 2024-04-26T01:47:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.073+0000 7f1c5fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c58101940 0x7f1c5819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.073+0000 7f1c5fe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c58102300 0x7f1c5819b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.073+0000 7f1c5fe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 0x7f1c5819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.073+0000 7f1c5fe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c58110a40 con 0x7f1c58101940 2024-04-26T01:47:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.073+0000 7f1c5fe00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581108c0 con 0x7f1c58102300 2024-04-26T01:47:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.073+0000 7f1c5fe00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1c58110bc0 con 0x7f1c58107930 2024-04-26T01:47:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.074+0000 7f1c5de00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 0x7f1c5819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.074+0000 7f1c5de00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 0x7f1c5819fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51846/0 (socket says 172.21.15.28:51846) 2024-04-26T01:47:08.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.074+0000 7f1c5de00700 1 -- 172.21.15.28:0/377233386 learned_addr learned my addr 172.21.15.28:0/377233386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:08.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.074+0000 7f1c5ca00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c58102300 0x7f1c5819b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.074+0000 7f1c5d400700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c58101940 0x7f1c5819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.075+0000 7f1c5de00700 1 -- 172.21.15.28:0/377233386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c58102300 msgr2=0x7f1c5819b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.075+0000 7f1c5de00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c58102300 0x7f1c5819b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.075+0000 7f1c5de00700 1 -- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c58101940 msgr2=0x7f1c5819b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.075+0000 7f1c5de00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c58101940 0x7f1c5819b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.075+0000 7f1c5de00700 1 -- 172.21.15.28:0/377233386 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c581a0240 con 0x7f1c58107930 2024-04-26T01:47:08.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.076+0000 7f1c5de00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 0x7f1c5819fb00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1c50007d70 tx=0x7f1c5000a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:08.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.076+0000 7f1c56000700 1 -- 172.21.15.28:0/377233386 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50016020 con 0x7f1c58107930 2024-04-26T01:47:08.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.076+0000 7f1c56000700 1 -- 172.21.15.28:0/377233386 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c5000ad40 con 0x7f1c58107930 2024-04-26T01:47:08.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.076+0000 7f1c56000700 1 -- 172.21.15.28:0/377233386 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c500115e0 con 0x7f1c58107930 2024-04-26T01:47:08.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.077+0000 7f1c5fe00700 1 -- 172.21.15.28:0/377233386 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c581a0530 con 0x7f1c58107930 2024-04-26T01:47:08.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.077+0000 7f1c5fe00700 1 -- 172.21.15.28:0/377233386 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c5806c460 con 0x7f1c58107930 2024-04-26T01:47:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.082+0000 7f1c56000700 1 -- 172.21.15.28:0/377233386 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1c50019020 con 0x7f1c58107930 2024-04-26T01:47:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.083+0000 7f1c56000700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1c400a3f50 0x7f1c400a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.084+0000 7f1c56000700 1 -- 172.21.15.28:0/377233386 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8557+0+0 (secure 0 0 0) 0x7f1c500d0670 con 0x7f1c58107930 2024-04-26T01:47:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.084+0000 7f1c5d400700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1c400a3f50 0x7f1c400a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.085+0000 7f1c5d400700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1c400a3f50 0x7f1c400a6410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00b2e0 tx=0x7f1c4c01a040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.086+0000 7f1c6a869b80 1 Processor -- start 2024-04-26T01:47:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.086+0000 7f1c6a869b80 1 -- start start 2024-04-26T01:47:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f695ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695b2d0 0x5633f696e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c5d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f695ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 0x5633f6970d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c5d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f695ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46570/0 (socket says 172.21.15.28:46570) 2024-04-26T01:47:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c5d400700 1 -- 172.21.15.28:0/875739461 learned_addr learned my addr 172.21.15.28:0/875739461 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c6a869b80 1 -- 172.21.15.28:0/875739461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5633f68ae020 con 0x5633f695a9b0 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c6a869b80 1 -- 172.21.15.28:0/875739461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5633f68548c0 con 0x5633f695b2d0 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.087+0000 7f1c6a869b80 1 -- 172.21.15.28:0/875739461 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5633f68a9fe0 con 0x5633f696e990 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.088+0000 7f1c5ca00700 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695b2d0 0x5633f696e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.088+0000 7f1c5de00700 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 0x5633f6970d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.088+0000 7f1c5d400700 1 -- 172.21.15.28:0/875739461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 msgr2=0x5633f6970d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.088+0000 7f1c5d400700 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 0x5633f6970d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.088+0000 7f1c5d400700 1 -- 172.21.15.28:0/875739461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695b2d0 msgr2=0x5633f696e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.088+0000 7f1c5d400700 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695b2d0 0x5633f696e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.088+0000 7f1c5d400700 1 -- 172.21.15.28:0/875739461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633f684a810 con 0x5633f695a9b0 2024-04-26T01:47:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.089+0000 7f1c5d400700 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f695ad90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00bdc0 tx=0x7f1c4c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40aa20f8c00dfa42 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.089+0000 7f1c4aa00700 1 -- 172.21.15.28:0/875739461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c00bdf0 con 0x5633f695a9b0 2024-04-26T01:47:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.089+0000 7f1c4aa00700 1 -- 172.21.15.28:0/875739461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c4c00fc00 con 0x5633f695a9b0 2024-04-26T01:47:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.089+0000 7f1c4aa00700 1 -- 172.21.15.28:0/875739461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c02d3f0 con 0x5633f695a9b0 2024-04-26T01:47:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.091+0000 7f1c6a869b80 1 -- 172.21.15.28:0/875739461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 msgr2=0x5633f695ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.091+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f695ad90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00bdc0 tx=0x7f1c4c01e000 comp rx=0 tx=0).stop 2024-04-26T01:47:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.091+0000 7f1c6a869b80 1 -- 172.21.15.28:0/875739461 shutdown_connections 2024-04-26T01:47:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.091+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 0x5633f6970d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.091+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f695ad90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.091+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/875739461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695b2d0 0x5633f696e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.091+0000 7f1c6a869b80 1 -- 172.21.15.28:0/875739461 >> 172.21.15.28:0/875739461 conn(0x5633f6889730 msgr2=0x5633f6956e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.092+0000 7f1c6a869b80 1 -- 172.21.15.28:0/875739461 shutdown_connections 2024-04-26T01:47:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.092+0000 7f1c6a869b80 1 -- 172.21.15.28:0/875739461 wait complete. 2024-04-26T01:47:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.092+0000 7f1c6a869b80 1 -- start start 2024-04-26T01:47:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.093+0000 7f1c6a869b80 1 Processor -- start 2024-04-26T01:47:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.094+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f696e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.094+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f696b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.094+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f696e990 0x5633f696b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c5de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f696e990 0x5633f696b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c5ca00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f696b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c5ca00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f696b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51860/0 (socket says 172.21.15.28:51860) 2024-04-26T01:47:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c5de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f696e990 0x5633f696b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47038/0 (socket says 172.21.15.28:47038) 2024-04-26T01:47:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5633f6971450 con 0x5633f695a9b0 2024-04-26T01:47:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5633f68679f0 con 0x5633f696e990 2024-04-26T01:47:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5633f69715d0 con 0x5633f695b2d0 2024-04-26T01:47:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c5ca00700 1 -- 172.21.15.28:0/1738996024 learned_addr learned my addr 172.21.15.28:0/1738996024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.095+0000 7f1c6a869b80 10 client.0 _ll_register_callbacks cb 0x5633f68468b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.096+0000 7f1c5de00700 1 -- 172.21.15.28:0/1738996024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 msgr2=0x5633f696b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.096+0000 7f1c5de00700 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f696b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.096+0000 7f1c5de00700 1 -- 172.21.15.28:0/1738996024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 msgr2=0x5633f696e090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:47:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.097+0000 7f1c5d400700 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f696e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.097+0000 7f1c5de00700 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f696e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.097+0000 7f1c5de00700 1 -- 172.21.15.28:0/1738996024 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633f67fba60 con 0x5633f696e990 2024-04-26T01:47:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.098+0000 7f1c6a869b80 1 -- 172.21.15.28:0/1738996024 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c4c00fc00 con 0x5633f696e990 2024-04-26T01:47:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.098+0000 7f1c5d400700 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f696e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:47:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.099+0000 7f1c5de00700 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f696e990 0x5633f696b8d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1c500cde50 tx=0x7f1c5009c5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a4c0efd25a87ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.099+0000 7f1c3f400700 10 client.25832 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:47:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.099+0000 7f1c3f400700 1 -- 172.21.15.28:0/1738996024 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5009c6a0 con 0x5633f696e990 2024-04-26T01:47:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.099+0000 7f1c3f400700 1 -- 172.21.15.28:0/1738996024 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8557+0+0 (secure 0 0 0) 0x7f1c50085be0 con 0x5633f696e990 2024-04-26T01:47:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.099+0000 7f1c3f400700 1 -- 172.21.15.28:0/1738996024 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f1c50085e00 con 0x5633f696e990 2024-04-26T01:47:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.099+0000 7f1c6a869b80 10 client.25832 fetch_fsmap learned FSMap version 34 2024-04-26T01:47:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.099+0000 7f1c6a869b80 1 -- 172.21.15.28:0/1738996024 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({fsmap=34}) -- 0x5633f6844c80 con 0x5633f696e990 2024-04-26T01:47:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.101+0000 7f1c3f400700 1 -- 172.21.15.28:0/1738996024 <== mon.2 v2:172.21.15.53:3300/0 4 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f1c5009c900 con 0x5633f696e990 2024-04-26T01:47:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.101+0000 7f1c6a869b80 10 client.25832 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:47:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.101+0000 7f1c6a869b80 10 client.25832 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:47:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.101+0000 7f1c6a869b80 20 client.25832 populate_metadata read hostname 'smithi028' 2024-04-26T01:47:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.101+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5633f695ffa0 0x5633f6962440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.101+0000 7f1c6a869b80 4 client.25832 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.101+0000 7f1c6a869b80 1 -- 172.21.15.28:0/1738996024 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633f68bb810 con 0x5633f695ffa0 2024-04-26T01:47:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.102+0000 7f1c5d400700 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5633f695ffa0 0x5633f6962440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.102+0000 7f1c5d400700 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5633f695ffa0 0x5633f6962440 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d24c25994143d313 server_cookie=aa936715bc5e88df in_seq=0 out_seq=0 2024-04-26T01:47:08.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.102+0000 7f1c3f400700 10 client.25832 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c3f400700 1 -- 172.21.15.28:0/1738996024 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5633f68bb810 con 0x5633f695ffa0 2024-04-26T01:47:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c3f400700 10 client.25832 handle_command_reply: tid=0 2024-04-26T01:47:08.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c6a869b80 1 client.25832 shutdown 2024-04-26T01:47:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c6a869b80 1 -- 172.21.15.28:0/1738996024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f696e990 msgr2=0x5633f696b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f696e990 0x5633f696b8d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1c500cde50 tx=0x7f1c5009c5a0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c6a869b80 1 -- 172.21.15.28:0/1738996024 shutdown_connections 2024-04-26T01:47:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f696e990 0x5633f696b8d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695a9b0 0x5633f696e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.103+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5633f695ffa0 0x5633f6962440 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.104+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/1738996024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f696b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.104+0000 7f1c3f400700 0 client.25832 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.104+0000 7f1c6a869b80 1 -- 172.21.15.28:0/1738996024 >> 172.21.15.28:0/1738996024 conn(0x5633f6889730 msgr2=0x5633f6956a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.104+0000 7f1c6a869b80 1 -- 172.21.15.28:0/1738996024 shutdown_connections 2024-04-26T01:47:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.104+0000 7f1c6a869b80 1 -- 172.21.15.28:0/1738996024 wait complete. 2024-04-26T01:47:08.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.105+0000 7f1c6a869b80 20 client.25832 trim_cache size 0 max 16384 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.144+0000 7f1c6a869b80 1 Processor -- start 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.144+0000 7f1c6a869b80 1 -- start start 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.144+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6956fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.144+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695b2d0 0x5633f6980bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.144+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 0x5633f69810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.145+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c58072080 con 0x5633f695b2d0 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.145+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5806f8b0 con 0x5633f695a9b0 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.145+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581153f0 con 0x5633f696e990 2024-04-26T01:47:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.145+0000 7f1c5d400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6956fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.145+0000 7f1c5d400700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6956fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:47048/0 (socket says 172.21.15.28:47048) 2024-04-26T01:47:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.145+0000 7f1c5d400700 1 -- 172.21.15.28:0/3696172577 learned_addr learned my addr 172.21.15.28:0/3696172577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.145+0000 7f1c5ca00700 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695b2d0 0x5633f6980bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.145+0000 7f1c5de00700 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 0x5633f69810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.146+0000 7f1c5d400700 1 -- 172.21.15.28:0/3696172577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 msgr2=0x5633f69810f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.146+0000 7f1c5d400700 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 0x5633f69810f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.146+0000 7f1c5d400700 1 -- 172.21.15.28:0/3696172577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695b2d0 msgr2=0x5633f6980bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.146+0000 7f1c5d400700 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695b2d0 0x5633f6980bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.146+0000 7f1c5d400700 1 -- 172.21.15.28:0/3696172577 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633f6962ac0 con 0x5633f695a9b0 2024-04-26T01:47:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.146+0000 7f1c5ca00700 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695b2d0 0x5633f6980bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.147+0000 7f1c5d400700 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6956fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c023b50 tx=0x7f1c4c022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c96bdba9ea98dc3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.147+0000 7f1c4a000700 1 -- 172.21.15.28:0/3696172577 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c023410 con 0x5633f695a9b0 2024-04-26T01:47:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.147+0000 7f1c4a000700 1 -- 172.21.15.28:0/3696172577 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c4c036410 con 0x5633f695a9b0 2024-04-26T01:47:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.147+0000 7f1c4a000700 1 -- 172.21.15.28:0/3696172577 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c02d650 con 0x5633f695a9b0 2024-04-26T01:47:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.148+0000 7f1c6a869b80 1 -- 172.21.15.28:0/3696172577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 msgr2=0x5633f6956fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.148+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6956fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c023b50 tx=0x7f1c4c022000 comp rx=0 tx=0).stop 2024-04-26T01:47:08.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.149+0000 7f1c6a869b80 1 -- 172.21.15.28:0/3696172577 shutdown_connections 2024-04-26T01:47:08.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.149+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f696e990 0x5633f69810f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.149+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6956fa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.149+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/3696172577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f695b2d0 0x5633f6980bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.149+0000 7f1c6a869b80 1 -- 172.21.15.28:0/3696172577 >> 172.21.15.28:0/3696172577 conn(0x5633f6889730 msgr2=0x5633f695c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:08.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.149+0000 7f1c6a869b80 1 -- 172.21.15.28:0/3696172577 shutdown_connections 2024-04-26T01:47:08.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.149+0000 7f1c6a869b80 1 -- 172.21.15.28:0/3696172577 wait complete. 2024-04-26T01:47:08.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.150+0000 7f1c6a869b80 1 -- start start 2024-04-26T01:47:08.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.150+0000 7f1c6a869b80 1 Processor -- start 2024-04-26T01:47:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.151+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6976ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.151+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f6973e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.151+0000 7f1c6a869b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f696e990 0x5633f69743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.151+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5806dd90 con 0x5633f696e990 2024-04-26T01:47:08.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.151+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5806f1e0 con 0x5633f695a9b0 2024-04-26T01:47:08.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.151+0000 7f1c6a869b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5806e440 con 0x5633f695b2d0 2024-04-26T01:47:08.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.151+0000 7f1c6a869b80 10 client.0 _ll_register_callbacks cb 0x5633f68468b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:08.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.152+0000 7f1c5ca00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f6973e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.152+0000 7f1c5ca00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f6973e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51880/0 (socket says 172.21.15.28:51880) 2024-04-26T01:47:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.152+0000 7f1c5ca00700 1 -- 172.21.15.28:0/4101929939 learned_addr learned my addr 172.21.15.28:0/4101929939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.152+0000 7f1c5ca00700 1 -- 172.21.15.28:0/4101929939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 msgr2=0x5633f6976ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.152+0000 7f1c5d400700 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6976ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.152+0000 7f1c5ca00700 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6976ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.153+0000 7f1c5ca00700 1 -- 172.21.15.28:0/4101929939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f696e990 msgr2=0x5633f69743a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:47:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.153+0000 7f1c5de00700 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f696e990 0x5633f69743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.153+0000 7f1c5ca00700 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f696e990 0x5633f69743a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.153+0000 7f1c5ca00700 1 -- 172.21.15.28:0/4101929939 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633f6974a50 con 0x5633f695b2d0 2024-04-26T01:47:08.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.154+0000 7f1c5ca00700 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f6973e60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1c440106b0 tx=0x7f1c44010a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49928949409d5c2a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:08.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.154+0000 7f1c3f400700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:47:08.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.155+0000 7f1c3f400700 1 -- 172.21.15.28:0/4101929939 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4400f950 con 0x5633f695b2d0 2024-04-26T01:47:08.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.155+0000 7f1c3f400700 1 -- 172.21.15.28:0/4101929939 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8557+0+0 (secure 0 0 0) 0x7f1c4400f750 con 0x5633f695b2d0 2024-04-26T01:47:08.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.155+0000 7f1c6a869b80 1 -- 172.21.15.28:0/4101929939 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c4c036410 con 0x5633f695b2d0 2024-04-26T01:47:08.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.155+0000 7f1c3f400700 1 -- 172.21.15.28:0/4101929939 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f1c4402b050 con 0x5633f695b2d0 2024-04-26T01:47:08.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.155+0000 7f1c6a869b80 10 client.25855 fetch_fsmap learned FSMap version 34 2024-04-26T01:47:08.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.155+0000 7f1c6a869b80 1 -- 172.21.15.28:0/4101929939 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5633f69dd300 con 0x5633f695b2d0 2024-04-26T01:47:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.156+0000 7f1c3f400700 1 -- 172.21.15.28:0/4101929939 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c4c036410 con 0x5633f695b2d0 2024-04-26T01:47:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.156+0000 7f1c3f400700 1 -- 172.21.15.28:0/4101929939 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f1c4401dce0 con 0x5633f695b2d0 2024-04-26T01:47:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.158+0000 7f1c6a869b80 10 client.25855 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:47:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.158+0000 7f1c6a869b80 10 client.25855 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:47:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.158+0000 7f1c6a869b80 20 client.25855 populate_metadata read hostname 'smithi028' 2024-04-26T01:47:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.158+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5633f69ddb00 0x5633f69dff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.158+0000 7f1c6a869b80 4 client.25855 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}] 2024-04-26T01:47:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.158+0000 7f1c6a869b80 1 -- 172.21.15.28:0/4101929939 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}) -- 0x5633f68b58b0 con 0x5633f69ddb00 2024-04-26T01:47:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.158+0000 7f1c5d400700 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5633f69ddb00 0x5633f69dff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.161+0000 7f1c3f400700 10 client.25855 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.161+0000 7f1c5d400700 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5633f69ddb00 0x5633f69dff20 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d69e309bfc9d4dba server_cookie=645559032d39f307 in_seq=0 out_seq=0 2024-04-26T01:47:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c3f400700 1 -- 172.21.15.28:0/4101929939 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5633f68b58b0 con 0x5633f69ddb00 2024-04-26T01:47:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c3f400700 10 client.25855 handle_command_reply: tid=0 2024-04-26T01:47:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 client.25855 shutdown 2024-04-26T01:47:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 -- 172.21.15.28:0/4101929939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 msgr2=0x5633f6973e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f6973e60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1c440106b0 tx=0x7f1c44010a70 comp rx=0 tx=0).stop 2024-04-26T01:47:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 -- 172.21.15.28:0/4101929939 shutdown_connections 2024-04-26T01:47:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633f696e990 0x5633f69743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5633f695a9b0 0x5633f6976ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5633f69ddb00 0x5633f69dff20 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 --2- 172.21.15.28:0/4101929939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5633f695b2d0 0x5633f6973e60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 -- 172.21.15.28:0/4101929939 >> 172.21.15.28:0/4101929939 conn(0x5633f6889730 msgr2=0x5633f695d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c3f400700 0 client.25855 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 -- 172.21.15.28:0/4101929939 shutdown_connections 2024-04-26T01:47:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.387+0000 7f1c6a869b80 1 -- 172.21.15.28:0/4101929939 wait complete. 2024-04-26T01:47:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.388+0000 7f1c6a869b80 20 client.25855 trim_cache size 0 max 16384 2024-04-26T01:47:08.387 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:47:08.387 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 34, 2024-04-26T01:47:08.387 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:47:08.387 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 14, 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "6288c6a": { 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 14, 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 60.0, 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 70.0, 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:47:08.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:47:08.389 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:47:08.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:47:08.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:47:08.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:47:08.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:47:08.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:47:08.389 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:47:08.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 -- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1c400a3f50 msgr2=0x7f1c400a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1c400a3f50 0x7f1c400a6410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00b2e0 tx=0x7f1c4c01a040 comp rx=0 tx=0).stop 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 -- 172.21.15.28:0/377233386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 msgr2=0x7f1c5819fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 0x7f1c5819fb00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1c50007d70 tx=0x7f1c5000a340 comp rx=0 tx=0).stop 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 -- 172.21.15.28:0/377233386 shutdown_connections 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1c400a3f50 0x7f1c400a6410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c58107930 0x7f1c5819fb00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c58101940 0x7f1c5819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 --2- 172.21.15.28:0/377233386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c58102300 0x7f1c5819b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.389+0000 7f1c5fe00700 1 -- 172.21.15.28:0/377233386 >> 172.21.15.28:0/377233386 conn(0x7f1c580788e0 msgr2=0x7f1c580ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.390+0000 7f1c5fe00700 1 -- 172.21.15.28:0/377233386 shutdown_connections 2024-04-26T01:47:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:08.390+0000 7f1c5fe00700 1 -- 172.21.15.28:0/377233386 wait complete. 2024-04-26T01:47:09.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:09 smithi053 ceph-mon[32593]: pgmap v189: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 MiB/s wr, 601 op/s 2024-04-26T01:47:09.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:09 smithi053 ceph-mon[32593]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:47:09.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:09 smithi028 ceph-mon[28247]: pgmap v189: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 MiB/s wr, 601 op/s 2024-04-26T01:47:09.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:09 smithi028 ceph-mon[28247]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:47:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:09 smithi169 ceph-mon[32851]: pgmap v189: 97 pgs: 97 active+clean; 2.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 MiB/s wr, 601 op/s 2024-04-26T01:47:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:09 smithi169 ceph-mon[32851]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:47:09.890 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6288c6a', quiesce took 10.0 seconds. Will release after: 10 2024-04-26T01:47:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 315.0 s, 652.9 tps, lat 72.945 ms stddev 161.835 2024-04-26T01:47:11.462 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:11 smithi169 ceph-mon[32851]: pgmap v191: 97 pgs: 97 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 56 MiB/s wr, 1.77k op/s 2024-04-26T01:47:11.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:11 smithi053 ceph-mon[32593]: pgmap v191: 97 pgs: 97 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 56 MiB/s wr, 1.77k op/s 2024-04-26T01:47:11.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:11 smithi028 ceph-mon[28247]: pgmap v191: 97 pgs: 97 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 56 MiB/s wr, 1.77k op/s 2024-04-26T01:47:12.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 -- 172.21.15.28:0/3034408881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c10e6d0 msgr2=0x7ff25c110ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 --2- 172.21.15.28:0/3034408881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c10e6d0 0x7ff25c110ac0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff254009900 tx=0x7ff25402a400 comp rx=0 tx=0).stop 2024-04-26T01:47:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 -- 172.21.15.28:0/3034408881 shutdown_connections 2024-04-26T01:47:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 --2- 172.21.15.28:0/3034408881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c10e6d0 0x7ff25c110ac0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 --2- 172.21.15.28:0/3034408881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff25c1014d0 0x7ff25c10e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 --2- 172.21.15.28:0/3034408881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff25c100b20 0x7ff25c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 -- 172.21.15.28:0/3034408881 >> 172.21.15.28:0/3034408881 conn(0x7ff25c0fc770 msgr2=0x7ff25c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 -- 172.21.15.28:0/3034408881 shutdown_connections 2024-04-26T01:47:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.664+0000 7ff25b200640 1 -- 172.21.15.28:0/3034408881 wait complete. 2024-04-26T01:47:12.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.665+0000 7ff25b200640 1 Processor -- start 2024-04-26T01:47:12.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.665+0000 7ff25b200640 1 -- start start 2024-04-26T01:47:12.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff25b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff25c100b20 0x7ff25c1a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff25b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c1014d0 0x7ff25c1a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff25b200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff25c10e6d0 0x7ff25c19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff25b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c114200 con 0x7ff25c1014d0 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff25b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c114080 con 0x7ff25c100b20 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff25b200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c114380 con 0x7ff25c10e6d0 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff259400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c1014d0 0x7ff25c1a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff259e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff25c100b20 0x7ff25c1a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff259e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff25c100b20 0x7ff25c1a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59562/0 (socket says 172.21.15.28:59562) 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff259e00640 1 -- 172.21.15.28:0/1333588918 learned_addr learned my addr 172.21.15.28:0/1333588918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.666+0000 7ff259400640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c1014d0 0x7ff25c1a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33180/0 (socket says 172.21.15.28:33180) 2024-04-26T01:47:12.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.667+0000 7ff25a800640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff25c10e6d0 0x7ff25c19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.667+0000 7ff259e00640 1 -- 172.21.15.28:0/1333588918 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff25c10e6d0 msgr2=0x7ff25c19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.667+0000 7ff259e00640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff25c10e6d0 0x7ff25c19a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.667+0000 7ff259e00640 1 -- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c1014d0 msgr2=0x7ff25c1a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.667+0000 7ff259e00640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c1014d0 0x7ff25c1a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.667+0000 7ff259e00640 1 -- 172.21.15.28:0/1333588918 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25c19ad60 con 0x7ff25c100b20 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.667+0000 7ff259400640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c1014d0 0x7ff25c1a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.667+0000 7ff259e00640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff25c100b20 0x7ff25c1a03e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00eab0 tx=0x7ff24c00ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.668+0000 7ff252a00640 1 -- 172.21.15.28:0/1333588918 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c00ce40 con 0x7ff25c100b20 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.668+0000 7ff25b200640 1 -- 172.21.15.28:0/1333588918 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25c19b050 con 0x7ff25c100b20 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.668+0000 7ff252a00640 1 -- 172.21.15.28:0/1333588918 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24c011920 con 0x7ff25c100b20 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.668+0000 7ff252a00640 1 -- 172.21.15.28:0/1333588918 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c010a30 con 0x7ff25c100b20 2024-04-26T01:47:12.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.668+0000 7ff25b200640 1 -- 172.21.15.28:0/1333588918 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25c1a72a0 con 0x7ff25c100b20 2024-04-26T01:47:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.671+0000 7ff263522740 1 Processor -- start 2024-04-26T01:47:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.671+0000 7ff263522740 1 -- start start 2024-04-26T01:47:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff252a00640 1 -- 172.21.15.28:0/1333588918 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff24c010bd0 con 0x7ff25c100b20 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff263522740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb856f9b0 0x55adb856fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff263522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb857a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff263522740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 0x55adb857cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff263522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55adb84d5900 con 0x55adb85702d0 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff263522740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55adb847bef0 con 0x55adb857ab90 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff263522740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55adb84d18c0 con 0x55adb856f9b0 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff25a800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb856f9b0 0x55adb856fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff259e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb857a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff25a800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb856f9b0 0x55adb856fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:57646/0 (socket says 172.21.15.28:57646) 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff25a800640 1 -- 172.21.15.28:0/416521375 learned_addr learned my addr 172.21.15.28:0/416521375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.672+0000 7ff259400640 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 0x55adb857cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.673+0000 7ff25a800640 1 -- 172.21.15.28:0/416521375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 msgr2=0x55adb857cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.673+0000 7ff25a800640 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 0x55adb857cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.673+0000 7ff25a800640 1 -- 172.21.15.28:0/416521375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 msgr2=0x55adb857a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.673+0000 7ff25a800640 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb857a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.673+0000 7ff25a800640 1 -- 172.21.15.28:0/416521375 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adb81e8210 con 0x55adb856f9b0 2024-04-26T01:47:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.673+0000 7ff259e00640 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb857a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:12.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.673+0000 7ff25a800640 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb856f9b0 0x55adb856fd90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff25c101a70 tx=0x7ff2540041a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66bdb167c49ef00c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.674+0000 7ff24b000640 1 -- 172.21.15.28:0/416521375 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254040070 con 0x55adb856f9b0 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.674+0000 7ff24b000640 1 -- 172.21.15.28:0/416521375 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff254003aa0 con 0x55adb856f9b0 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.674+0000 7ff24b000640 1 -- 172.21.15.28:0/416521375 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254035040 con 0x55adb856f9b0 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.674+0000 7ff252a00640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff2400a3d10 0x7ff2400a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.674+0000 7ff252a00640 1 -- 172.21.15.28:0/1333588918 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7ff24c0d0d60 con 0x7ff25c100b20 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.674+0000 7ff259e00640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff2400a3d10 0x7ff2400a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 -- 172.21.15.28:0/416521375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb856f9b0 msgr2=0x55adb856fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb856f9b0 0x55adb856fd90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff25c101a70 tx=0x7ff2540041a0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 -- 172.21.15.28:0/416521375 shutdown_connections 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 0x55adb857cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb857a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 --2- 172.21.15.28:0/416521375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb856f9b0 0x55adb856fd90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 -- 172.21.15.28:0/416521375 >> 172.21.15.28:0/416521375 conn(0x55adb856bec0 msgr2=0x55adb856c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 -- 172.21.15.28:0/416521375 shutdown_connections 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff259e00640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff2400a3d10 0x7ff2400a61d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0ce560 tx=0x7ff24c0104d0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 -- 172.21.15.28:0/416521375 wait complete. 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.675+0000 7ff263522740 1 -- start start 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff263522740 1 Processor -- start 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff263522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb856f9b0 0x55adb8577b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff263522740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb85702d0 0x55adb8578070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff263522740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 0x55adb85785b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff263522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55adb857d760 con 0x55adb856f9b0 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff263522740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55adb848f310 con 0x55adb857ab90 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff263522740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55adb857d8e0 con 0x55adb85702d0 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff263522740 10 client.0 _ll_register_callbacks cb 0x55adb845dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff259400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb85702d0 0x55adb8578070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff25a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb856f9b0 0x55adb8577b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff259400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb85702d0 0x55adb8578070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:57660/0 (socket says 172.21.15.28:57660) 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.676+0000 7ff259400640 1 -- 172.21.15.28:0/905010933 learned_addr learned my addr 172.21.15.28:0/905010933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff25a800640 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 0x55adb85785b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff259400640 1 -- 172.21.15.28:0/905010933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 msgr2=0x55adb85785b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff259400640 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 0x55adb85785b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff259400640 1 -- 172.21.15.28:0/905010933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb856f9b0 msgr2=0x55adb8577b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff259400640 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb856f9b0 0x55adb8577b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff259400640 1 -- 172.21.15.28:0/905010933 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adb82b4620 con 0x55adb85702d0 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff259400640 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb85702d0 0x55adb8578070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff244011720 tx=0x7ff244011be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f38f2887a922d9a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff23f400640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff23f400640 1 -- 172.21.15.28:0/905010933 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff244010c60 con 0x55adb85702d0 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff23f400640 1 -- 172.21.15.28:0/905010933 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7ff244010e00 con 0x55adb85702d0 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.677+0000 7ff263522740 1 -- 172.21.15.28:0/905010933 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff254003aa0 con 0x55adb85702d0 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.678+0000 7ff23f400640 1 -- 172.21.15.28:0/905010933 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff244004020 con 0x55adb85702d0 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.678+0000 7ff263522740 10 client.25867 fetch_fsmap learned FSMap version 34 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.678+0000 7ff263522740 1 -- 172.21.15.28:0/905010933 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55adb8589a20 con 0x55adb85702d0 2024-04-26T01:47:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.678+0000 7ff23f400640 1 -- 172.21.15.28:0/905010933 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff254003aa0 con 0x55adb85702d0 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.678+0000 7ff23f400640 1 -- 172.21.15.28:0/905010933 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7ff24401a410 con 0x55adb85702d0 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.679+0000 7ff263522740 10 client.25867 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.679+0000 7ff263522740 10 client.25867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.679+0000 7ff263522740 20 client.25867 populate_metadata read hostname 'smithi028' 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.679+0000 7ff263522740 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55adb858afa0 0x55adb858d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.679+0000 7ff263522740 4 client.25867 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.679+0000 7ff263522740 1 -- 172.21.15.28:0/905010933 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adb84e98b0 con 0x55adb858afa0 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.679+0000 7ff25a800640 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55adb858afa0 0x55adb858d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.680+0000 7ff25a800640 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55adb858afa0 0x55adb858d3b0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a441de1a2fb43731 server_cookie=a2e6a3eb1c8f5737 in_seq=0 out_seq=0 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.680+0000 7ff23f400640 10 client.25867 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.680+0000 7ff23f400640 1 -- 172.21.15.28:0/905010933 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55adb84e98b0 con 0x55adb858afa0 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.680+0000 7ff23f400640 10 client.25867 handle_command_reply: tid=0 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 client.25867 shutdown 2024-04-26T01:47:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 -- 172.21.15.28:0/905010933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb85702d0 msgr2=0x55adb8578070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb85702d0 0x55adb8578070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff244011720 tx=0x7ff244011be0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 -- 172.21.15.28:0/905010933 shutdown_connections 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55adb858afa0 0x55adb858d3b0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb857ab90 0x55adb85785b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb85702d0 0x55adb8578070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 --2- 172.21.15.28:0/905010933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb856f9b0 0x55adb8577b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff23f400640 0 client.25867 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 -- 172.21.15.28:0/905010933 >> 172.21.15.28:0/905010933 conn(0x55adb856bec0 msgr2=0x55adb8571c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 -- 172.21.15.28:0/905010933 shutdown_connections 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 1 -- 172.21.15.28:0/905010933 wait complete. 2024-04-26T01:47:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.681+0000 7ff263522740 20 client.25867 trim_cache size 0 max 16384 2024-04-26T01:47:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff263522740 1 Processor -- start 2024-04-26T01:47:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff263522740 1 -- start start 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff263522740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8586620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff263522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8592cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff263522740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 0x55adb8593200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff263522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b24d0 con 0x55adb85702d0 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff263522740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1afd40 con 0x55adb856f9b0 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff263522740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1adbb0 con 0x55adb857ab90 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.698+0000 7ff259400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8592cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff259400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8592cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33194/0 (socket says 172.21.15.28:33194) 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff259400640 1 -- 172.21.15.28:0/1807034519 learned_addr learned my addr 172.21.15.28:0/1807034519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff25a800640 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 0x55adb8593200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff25a800640 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8586620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff259400640 1 -- 172.21.15.28:0/1807034519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 msgr2=0x55adb8593200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff259400640 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 0x55adb8593200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff259400640 1 -- 172.21.15.28:0/1807034519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 msgr2=0x55adb8586620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff259400640 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8586620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff259400640 1 -- 172.21.15.28:0/1807034519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adb8576080 con 0x55adb85702d0 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.699+0000 7ff259400640 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8592cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff24401a880 tx=0x7ff244005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f0314428a610a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff24a600640 1 -- 172.21.15.28:0/1807034519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24401a410 con 0x55adb85702d0 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff24a600640 1 -- 172.21.15.28:0/1807034519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24401a5b0 con 0x55adb85702d0 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff24a600640 1 -- 172.21.15.28:0/1807034519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff244005040 con 0x55adb85702d0 2024-04-26T01:47:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff263522740 1 -- 172.21.15.28:0/1807034519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 msgr2=0x55adb8592cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff263522740 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8592cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff24401a880 tx=0x7ff244005850 comp rx=0 tx=0).stop 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff263522740 1 -- 172.21.15.28:0/1807034519 shutdown_connections 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff263522740 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 0x55adb8593200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff263522740 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8592cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff263522740 1 --2- 172.21.15.28:0/1807034519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8586620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.700+0000 7ff263522740 1 -- 172.21.15.28:0/1807034519 >> 172.21.15.28:0/1807034519 conn(0x55adb856bec0 msgr2=0x55adb8578380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.701+0000 7ff263522740 1 -- 172.21.15.28:0/1807034519 shutdown_connections 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.701+0000 7ff263522740 1 -- 172.21.15.28:0/1807034519 wait complete. 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.701+0000 7ff263522740 1 -- start start 2024-04-26T01:47:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.701+0000 7ff263522740 1 Processor -- start 2024-04-26T01:47:12.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff263522740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8580720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff263522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8580c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff263522740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 0x55adb85811a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff263522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1ae200 con 0x55adb85702d0 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff263522740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1af670 con 0x55adb856f9b0 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff263522740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1ae8d0 con 0x55adb857ab90 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff263522740 10 client.0 _ll_register_callbacks cb 0x7ff24400bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff259400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8580c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8580720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8580720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59604/0 (socket says 172.21.15.28:59604) 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 -- 172.21.15.28:0/857822205 learned_addr learned my addr 172.21.15.28:0/857822205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff259400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8580c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33204/0 (socket says 172.21.15.28:33204) 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 0x55adb85811a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 -- 172.21.15.28:0/857822205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 msgr2=0x55adb85811a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 0x55adb85811a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 -- 172.21.15.28:0/857822205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 msgr2=0x55adb8580c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8580c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.702+0000 7ff25a800640 1 -- 172.21.15.28:0/857822205 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adb85f0640 con 0x55adb856f9b0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff25a800640 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8580720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff254009900 tx=0x7ff25404f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d7f8126a5f46550 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff23f400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff23f400640 1 -- 172.21.15.28:0/857822205 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254035040 con 0x55adb856f9b0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff263522740 1 -- 172.21.15.28:0/857822205 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff24401a5b0 con 0x55adb856f9b0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff23f400640 1 -- 172.21.15.28:0/857822205 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7ff254005020 con 0x55adb856f9b0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff23f400640 1 -- 172.21.15.28:0/857822205 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff254040040 con 0x55adb856f9b0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff263522740 10 client.25853 fetch_fsmap learned FSMap version 34 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff263522740 1 -- 172.21.15.28:0/857822205 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55adb85f1950 con 0x55adb856f9b0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.703+0000 7ff23f400640 1 -- 172.21.15.28:0/857822205 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24401a5b0 con 0x55adb856f9b0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.704+0000 7ff23f400640 1 -- 172.21.15.28:0/857822205 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7ff2540531c0 con 0x55adb856f9b0 2024-04-26T01:47:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.704+0000 7ff263522740 10 client.25853 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:47:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.704+0000 7ff263522740 10 client.25853 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.704+0000 7ff263522740 20 client.25853 populate_metadata read hostname 'smithi028' 2024-04-26T01:47:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.704+0000 7ff263522740 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55adb85f1c80 0x55adb85f4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.704+0000 7ff263522740 4 client.25853 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.704+0000 7ff263522740 1 -- 172.21.15.28:0/857822205 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55adb84e3320 con 0x55adb85f1c80 2024-04-26T01:47:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.704+0000 7ff259400640 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55adb85f1c80 0x55adb85f4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.705+0000 7ff259400640 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55adb85f1c80 0x55adb85f4070 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3fa07e6b854e7b3 server_cookie=d695463cea1af13b in_seq=0 out_seq=0 2024-04-26T01:47:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.705+0000 7ff23f400640 10 client.25853 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.727+0000 7ff23f400640 1 -- 172.21.15.28:0/857822205 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55adb84e3320 con 0x55adb85f1c80 2024-04-26T01:47:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.727+0000 7ff23f400640 10 client.25853 handle_command_reply: tid=0 2024-04-26T01:47:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.727+0000 7ff263522740 1 client.25853 shutdown 2024-04-26T01:47:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 -- 172.21.15.28:0/857822205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 msgr2=0x55adb8580720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8580720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff254009900 tx=0x7ff25404f040 comp rx=0 tx=0).stop 2024-04-26T01:47:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 -- 172.21.15.28:0/857822205 shutdown_connections 2024-04-26T01:47:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55adb85f1c80 0x55adb85f4070 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55adb857ab90 0x55adb85811a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adb85702d0 0x55adb8580c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 --2- 172.21.15.28:0/857822205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55adb856f9b0 0x55adb8580720 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 -- 172.21.15.28:0/857822205 >> 172.21.15.28:0/857822205 conn(0x55adb856bec0 msgr2=0x55adb8577af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff23f400640 0 client.25853 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 -- 172.21.15.28:0/857822205 shutdown_connections 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 1 -- 172.21.15.28:0/857822205 wait complete. 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.728+0000 7ff263522740 20 client.25853 trim_cache size 0 max 16384 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:47:12.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:47:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:47:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:47:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 -- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff2400a3d10 msgr2=0x7ff2400a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff2400a3d10 0x7ff2400a61d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0ce560 tx=0x7ff24c0104d0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 -- 172.21.15.28:0/1333588918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff25c100b20 msgr2=0x7ff25c1a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff25c100b20 0x7ff25c1a03e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00eab0 tx=0x7ff24c00ef70 comp rx=0 tx=0).stop 2024-04-26T01:47:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 -- 172.21.15.28:0/1333588918 shutdown_connections 2024-04-26T01:47:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff2400a3d10 0x7ff2400a61d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff25c10e6d0 0x7ff25c19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c1014d0 0x7ff25c1a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 --2- 172.21.15.28:0/1333588918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff25c100b20 0x7ff25c1a03e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 -- 172.21.15.28:0/1333588918 >> 172.21.15.28:0/1333588918 conn(0x7ff25c0fc770 msgr2=0x7ff25c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.729+0000 7ff25b200640 1 -- 172.21.15.28:0/1333588918 shutdown_connections 2024-04-26T01:47:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:12.730+0000 7ff25b200640 1 -- 172.21.15.28:0/1333588918 wait complete. 2024-04-26T01:47:13.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:13 smithi053 ceph-mon[32593]: pgmap v192: 97 pgs: 97 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 46 MiB/s wr, 1.44k op/s 2024-04-26T01:47:13.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:13 smithi028 ceph-mon[28247]: pgmap v192: 97 pgs: 97 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 46 MiB/s wr, 1.44k op/s 2024-04-26T01:47:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:13 smithi169 ceph-mon[32851]: pgmap v192: 97 pgs: 97 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 46 MiB/s wr, 1.44k op/s 2024-04-26T01:47:15.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:15 smithi028 ceph-mon[28247]: pgmap v193: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.46k op/s 2024-04-26T01:47:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:15 smithi169 ceph-mon[32851]: pgmap v193: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.46k op/s 2024-04-26T01:47:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:15 smithi053 ceph-mon[32593]: pgmap v193: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.46k op/s 2024-04-26T01:47:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 320.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:17.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:17 smithi028 ceph-mon[28247]: pgmap v194: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.49k op/s 2024-04-26T01:47:17.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:17 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:17 smithi169 ceph-mon[32851]: pgmap v194: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.49k op/s 2024-04-26T01:47:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:17 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:17 smithi053 ceph-mon[32593]: pgmap v194: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.49k op/s 2024-04-26T01:47:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:17 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:19.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:19 smithi028 ceph-mon[28247]: pgmap v195: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.49k op/s 2024-04-26T01:47:19.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:19 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:19.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:19 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:19.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:19 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:47:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:19 smithi169 ceph-mon[32851]: pgmap v195: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.49k op/s 2024-04-26T01:47:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:19 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:19 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:19 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:47:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:19 smithi053 ceph-mon[32593]: pgmap v195: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 43 MiB/s wr, 1.49k op/s 2024-04-26T01:47:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:19 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:19 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:19 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:47:19.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:47:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:21.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 325.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:21.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:21 smithi028 ceph-mon[28247]: pgmap v196: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 1.25k op/s 2024-04-26T01:47:21.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:21 smithi169 ceph-mon[32851]: pgmap v196: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 1.25k op/s 2024-04-26T01:47:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:21 smithi053 ceph-mon[32593]: pgmap v196: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 1.25k op/s 2024-04-26T01:47:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 -- 172.21.15.28:0/2850371566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bf010e3a0 msgr2=0x7f8bf0115f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 --2- 172.21.15.28:0/2850371566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bf010e3a0 0x7f8bf0115f70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00b0a0 tx=0x7f8bec00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:47:22.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 -- 172.21.15.28:0/2850371566 shutdown_connections 2024-04-26T01:47:22.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 --2- 172.21.15.28:0/2850371566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bf010e3a0 0x7f8bf0115f70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:22.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 --2- 172.21.15.28:0/2850371566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf0109310 0x7f8bf01096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:22.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 --2- 172.21.15.28:0/2850371566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf0109cc0 0x7f8bf010dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:22.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 -- 172.21.15.28:0/2850371566 >> 172.21.15.28:0/2850371566 conn(0x7f8bf0076210 msgr2=0x7f8bf0078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:22.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 -- 172.21.15.28:0/2850371566 shutdown_connections 2024-04-26T01:47:22.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.854+0000 7f8bf8800700 1 -- 172.21.15.28:0/2850371566 wait complete. 2024-04-26T01:47:22.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.855+0000 7f8bf8800700 1 Processor -- start 2024-04-26T01:47:22.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.855+0000 7f8bf8800700 1 -- start start 2024-04-26T01:47:22.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf8800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf0109310 0x7f8bf019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:22.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf8800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf0109cc0 0x7f8bf019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf8800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bf010e3a0 0x7f8bf01a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf8800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf01192b0 con 0x7f8bf0109cc0 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf8800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf0119130 con 0x7f8bf010e3a0 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf8800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf0119430 con 0x7f8bf0109310 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf5e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf0109310 0x7f8bf019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf5400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf0109cc0 0x7f8bf019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf5e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf0109310 0x7f8bf019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58128/0 (socket says 172.21.15.28:58128) 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf5e00700 1 -- 172.21.15.28:0/459033568 learned_addr learned my addr 172.21.15.28:0/459033568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.856+0000 7f8bf6800700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bf010e3a0 0x7f8bf01a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8bf5e00700 1 -- 172.21.15.28:0/459033568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bf010e3a0 msgr2=0x7f8bf01a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8bf5e00700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bf010e3a0 0x7f8bf01a4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8bf5e00700 1 -- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf0109cc0 msgr2=0x7f8bf019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8bf5e00700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf0109cc0 0x7f8bf019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8bf5e00700 1 -- 172.21.15.28:0/459033568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf01a4740 con 0x7f8bf0109310 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8bf5400700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf0109cc0 0x7f8bf019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8bf5e00700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf0109310 0x7f8bf019f780 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8be400e9c0 tx=0x7f8be400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8beaa00700 1 -- 172.21.15.28:0/459033568 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be400cbd0 con 0x7f8bf0109310 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.857+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf01a4a30 con 0x7f8bf0109310 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.858+0000 7f8beaa00700 1 -- 172.21.15.28:0/459033568 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8be400ee60 con 0x7f8bf0109310 2024-04-26T01:47:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.858+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf01ac2b0 con 0x7f8bf0109310 2024-04-26T01:47:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.858+0000 7f8beaa00700 1 -- 172.21.15.28:0/459033568 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4005550 con 0x7f8bf0109310 2024-04-26T01:47:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.859+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf0069730 con 0x7f8bf0109310 2024-04-26T01:47:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.861+0000 7f8beaa00700 1 -- 172.21.15.28:0/459033568 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8be4004750 con 0x7f8bf0109310 2024-04-26T01:47:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.862+0000 7f8beaa00700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8bd80a3e30 0x7f8bd80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.862+0000 7f8beaa00700 1 -- 172.21.15.28:0/459033568 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f8be40d0a80 con 0x7f8bf0109310 2024-04-26T01:47:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.863+0000 7f8bf5400700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8bd80a3e30 0x7f8bd80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.863+0000 7f8bf5400700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8bd80a3e30 0x7f8bd80a62f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009680 tx=0x7f8be0009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:22.867+0000 7f8beaa00700 1 -- 172.21.15.28:0/459033568 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8be40d2e20 con 0x7f8bf0109310 2024-04-26T01:47:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.150+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bf01a0a90 con 0x7f8bf0109310 2024-04-26T01:47:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.151+0000 7f8beaa00700 1 -- 172.21.15.28:0/459033568 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11867 (secure 0 0 0) 0x7f8be409d0c0 con 0x7f8bf0109310 2024-04-26T01:47:23.150 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:47:23.150 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:46:17.099601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8bd80a3e30 msgr2=0x7f8bd80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8bd80a3e30 0x7f8bd80a62f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009680 tx=0x7f8be0009260 comp rx=0 tx=0).stop 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf0109310 msgr2=0x7f8bf019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf0109310 0x7f8bf019f780 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8be400e9c0 tx=0x7f8be400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 shutdown_connections 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8bd80a3e30 0x7f8bd80a62f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bf010e3a0 0x7f8bf01a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf0109310 0x7f8bf019f780 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 --2- 172.21.15.28:0/459033568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf0109cc0 0x7f8bf019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.154+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 >> 172.21.15.28:0/459033568 conn(0x7f8bf0076210 msgr2=0x7f8bf0078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.155+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 shutdown_connections 2024-04-26T01:47:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:23.155+0000 7f8bf8800700 1 -- 172.21.15.28:0/459033568 wait complete. 2024-04-26T01:47:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-04-26T01:47:23.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:23 smithi028 ceph-mon[28247]: pgmap v197: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.1 MiB/s wr, 272 op/s 2024-04-26T01:47:23.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:23 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/459033568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:23.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:23 smithi169 ceph-mon[32851]: pgmap v197: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.1 MiB/s wr, 272 op/s 2024-04-26T01:47:23.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:23 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/459033568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:23.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:23 smithi053 ceph-mon[32593]: pgmap v197: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.1 MiB/s wr, 272 op/s 2024-04-26T01:47:23.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:23 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/459033568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:24.126 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id 6288c6a --release --await' 2024-04-26T01:47:24.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id 6288c6a --release --await 2024-04-26T01:47:24.585 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:25.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:25 smithi028 ceph-mon[28247]: pgmap v198: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 3.1 MiB/s wr, 277 op/s 2024-04-26T01:47:25.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:25 smithi169 ceph-mon[32851]: pgmap v198: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 3.1 MiB/s wr, 277 op/s 2024-04-26T01:47:25.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:25 smithi053 ceph-mon[32593]: pgmap v198: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 3.1 MiB/s wr, 277 op/s 2024-04-26T01:47:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.883+0000 7f0e3e800640 1 -- 172.21.15.28:0/1358128638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38102b50 msgr2=0x7f0e38106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.883+0000 7f0e3e800640 1 --2- 172.21.15.28:0/1358128638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38102b50 0x7f0e38106b90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009900 tx=0x7f0e2402a400 comp rx=0 tx=0).stop 2024-04-26T01:47:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.883+0000 7f0e3e800640 1 -- 172.21.15.28:0/1358128638 shutdown_connections 2024-04-26T01:47:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.883+0000 7f0e3e800640 1 --2- 172.21.15.28:0/1358128638 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e38107270 0x7f0e3810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.883+0000 7f0e3e800640 1 --2- 172.21.15.28:0/1358128638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38102b50 0x7f0e38106b90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.883+0000 7f0e3e800640 1 --2- 172.21.15.28:0/1358128638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e381021a0 0x7f0e38102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.883+0000 7f0e3e800640 1 -- 172.21.15.28:0/1358128638 >> 172.21.15.28:0/1358128638 conn(0x7f0e380793e0 msgr2=0x7f0e380797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.883+0000 7f0e3e800640 1 -- 172.21.15.28:0/1358128638 shutdown_connections 2024-04-26T01:47:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.884+0000 7f0e3e800640 1 -- 172.21.15.28:0/1358128638 wait complete. 2024-04-26T01:47:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.885+0000 7f0e3e800640 1 Processor -- start 2024-04-26T01:47:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.885+0000 7f0e3e800640 1 -- start start 2024-04-26T01:47:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e3e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e381021a0 0x7f0e381a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e3e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38107270 0x7f0e381a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e3e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3819a740 0x7f0e3819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e3e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381121e0 con 0x7f0e3819a740 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e3e800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38112060 con 0x7f0e38107270 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e3e800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38112360 con 0x7f0e381021a0 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e37400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e381021a0 0x7f0e381a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e37400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e381021a0 0x7f0e381a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58152/0 (socket says 172.21.15.28:58152) 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e37400640 1 -- 172.21.15.28:0/3408230514 learned_addr learned my addr 172.21.15.28:0/3408230514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e36a00640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38107270 0x7f0e381a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.886+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3819a740 0x7f0e3819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e37400640 1 -- 172.21.15.28:0/3408230514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38107270 msgr2=0x7f0e381a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e37400640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38107270 0x7f0e381a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e37400640 1 -- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3819a740 msgr2=0x7f0e3819abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e37400640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3819a740 0x7f0e3819abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e37400640 1 -- 172.21.15.28:0/3408230514 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3819b340 con 0x7f0e381021a0 2024-04-26T01:47:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e37400640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e381021a0 0x7f0e381a0630 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800e9f0 tx=0x7f0e2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e2fe00640 1 -- 172.21.15.28:0/3408230514 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2800cd80 con 0x7f0e381021a0 2024-04-26T01:47:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e2fe00640 1 -- 172.21.15.28:0/3408230514 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e280108e0 con 0x7f0e381021a0 2024-04-26T01:47:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e2fe00640 1 -- 172.21.15.28:0/3408230514 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e28018980 con 0x7f0e381021a0 2024-04-26T01:47:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.887+0000 7f0e3e800640 1 -- 172.21.15.28:0/3408230514 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e381a73d0 con 0x7f0e381021a0 2024-04-26T01:47:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.888+0000 7f0e3e800640 1 -- 172.21.15.28:0/3408230514 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e381a7780 con 0x7f0e381021a0 2024-04-26T01:47:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e4167e740 1 Processor -- start 2024-04-26T01:47:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e4167e740 1 -- start start 2024-04-26T01:47:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e2fe00640 1 -- 172.21.15.28:0/3408230514 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0e28010430 con 0x7f0e381021a0 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae74820 0x562f6ae74c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae7f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae7fa00 0x562f6ae81df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e4167e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f6adda900 con 0x562f6ae75140 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e4167e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562f6ad80ef0 con 0x562f6ae7fa00 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.891+0000 7f0e4167e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562f6add68c0 con 0x562f6ae74820 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae7f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae74820 0x562f6ae74c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae74820 0x562f6ae74c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58160/0 (socket says 172.21.15.28:58160) 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 -- 172.21.15.28:0/2947677386 learned_addr learned my addr 172.21.15.28:0/2947677386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e36a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae7fa00 0x562f6ae81df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 -- 172.21.15.28:0/2947677386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae7fa00 msgr2=0x562f6ae81df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 --2- 172.21.15.28:0/2947677386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae7fa00 0x562f6ae81df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 -- 172.21.15.28:0/2947677386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 msgr2=0x562f6ae7f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 --2- 172.21.15.28:0/2947677386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae7f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 -- 172.21.15.28:0/2947677386 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f6aaed210 con 0x562f6ae74820 2024-04-26T01:47:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.892+0000 7f0e37e00640 1 --2- 172.21.15.28:0/2947677386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae74820 0x562f6ae74c00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009f40 tx=0x7f0e300118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0e98e30c74941e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e2cc00640 1 -- 172.21.15.28:0/2947677386 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3001e070 con 0x562f6ae74820 2024-04-26T01:47:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e2cc00640 1 -- 172.21.15.28:0/2947677386 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e30003c30 con 0x562f6ae74820 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e2cc00640 1 -- 172.21.15.28:0/2947677386 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3000f8c0 con 0x562f6ae74820 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e2fe00640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e1c0a3e40 0x7f0e1c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e2fe00640 1 -- 172.21.15.28:0/3408230514 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f0e280d0ad0 con 0x7f0e381021a0 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e4167e740 1 -- 172.21.15.28:0/2947677386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae74820 msgr2=0x562f6ae74c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e4167e740 1 --2- 172.21.15.28:0/2947677386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae74820 0x562f6ae74c00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009f40 tx=0x7f0e300118d0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e37400640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e1c0a3e40 0x7f0e1c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e4167e740 1 -- 172.21.15.28:0/2947677386 shutdown_connections 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e4167e740 1 --2- 172.21.15.28:0/2947677386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae7fa00 0x562f6ae81df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e4167e740 1 --2- 172.21.15.28:0/2947677386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae7f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e4167e740 1 --2- 172.21.15.28:0/2947677386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae74820 0x562f6ae74c00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.893+0000 7f0e4167e740 1 -- 172.21.15.28:0/2947677386 >> 172.21.15.28:0/2947677386 conn(0x562f6ae70d30 msgr2=0x562f6ae71110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.894+0000 7f0e4167e740 1 -- 172.21.15.28:0/2947677386 shutdown_connections 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.894+0000 7f0e4167e740 1 -- 172.21.15.28:0/2947677386 wait complete. 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.894+0000 7f0e37400640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e1c0a3e40 0x7f0e1c0a6300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x562f6ae75dc0 tx=0x7f0e280d4580 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.894+0000 7f0e4167e740 1 -- start start 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.894+0000 7f0e4167e740 1 Processor -- start 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae74820 0x562f6ae7c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae75140 0x562f6ae7cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 0x562f6ae7d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e4167e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f6ae82630 con 0x562f6ae74820 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e4167e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562f6ad94310 con 0x562f6ae75140 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e4167e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562f6ae827b0 con 0x562f6ae7fa00 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e4167e740 10 client.0 _ll_register_callbacks cb 0x562f6ad62cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e37e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae74820 0x562f6ae7c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e37e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae74820 0x562f6ae7c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39690/0 (socket says 172.21.15.28:39690) 2024-04-26T01:47:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e37e00640 1 -- 172.21.15.28:0/3096481146 learned_addr learned my addr 172.21.15.28:0/3096481146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e36a00640 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae75140 0x562f6ae7cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 0x562f6ae7d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e37e00640 1 -- 172.21.15.28:0/3096481146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 msgr2=0x562f6ae7d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 0x562f6ae7d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.895+0000 7f0e37e00640 1 -- 172.21.15.28:0/3096481146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae75140 msgr2=0x562f6ae7cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.896+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae75140 0x562f6ae7cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.896+0000 7f0e37e00640 1 -- 172.21.15.28:0/3096481146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f6abb9620 con 0x562f6ae74820 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.896+0000 7f0e36a00640 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae75140 0x562f6ae7cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.896+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae74820 0x562f6ae7c9a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e3001c040 tx=0x7f0e3000f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330f149479e4a819 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.896+0000 7f0e21600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:47:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.896+0000 7f0e21600640 1 -- 172.21.15.28:0/3096481146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3001e030 con 0x562f6ae74820 2024-04-26T01:47:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.896+0000 7f0e21600640 1 -- 172.21.15.28:0/3096481146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f0e30004260 con 0x562f6ae74820 2024-04-26T01:47:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.896+0000 7f0e4167e740 1 -- 172.21.15.28:0/3096481146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e30003c30 con 0x562f6ae74820 2024-04-26T01:47:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e21600640 1 -- 172.21.15.28:0/3096481146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0e30035050 con 0x562f6ae74820 2024-04-26T01:47:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e4167e740 10 client.15678 fetch_fsmap learned FSMap version 34 2024-04-26T01:47:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e4167e740 1 -- 172.21.15.28:0/3096481146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562f6ae8e890 con 0x562f6ae74820 2024-04-26T01:47:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e21600640 1 -- 172.21.15.28:0/3096481146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e30003c30 con 0x562f6ae74820 2024-04-26T01:47:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e21600640 1 -- 172.21.15.28:0/3096481146 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f0e30005040 con 0x562f6ae74820 2024-04-26T01:47:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e4167e740 10 client.15678 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:47:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e4167e740 10 client.15678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e4167e740 20 client.15678 populate_metadata read hostname 'smithi028' 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562f6ae8fc90 0x562f6ae92080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e4167e740 4 client.15678 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.897+0000 7f0e4167e740 1 -- 172.21.15.28:0/3096481146 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f6adee8b0 con 0x562f6ae8fc90 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.898+0000 7f0e36a00640 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562f6ae8fc90 0x562f6ae92080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.898+0000 7f0e36a00640 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562f6ae8fc90 0x562f6ae92080 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b74801d2d3c8f41 server_cookie=b0ec5d3418074fc in_seq=0 out_seq=0 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.898+0000 7f0e21600640 10 client.15678 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e21600640 1 -- 172.21.15.28:0/3096481146 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f6adee8b0 con 0x562f6ae8fc90 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e21600640 10 client.15678 handle_command_reply: tid=0 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 client.15678 shutdown 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 -- 172.21.15.28:0/3096481146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae74820 msgr2=0x562f6ae7c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae74820 0x562f6ae7c9a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e3001c040 tx=0x7f0e3000f480 comp rx=0 tx=0).stop 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 -- 172.21.15.28:0/3096481146 shutdown_connections 2024-04-26T01:47:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562f6ae8fc90 0x562f6ae92080 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 0x562f6ae7d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae75140 0x562f6ae7cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3096481146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae74820 0x562f6ae7c9a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e21600640 0 client.15678 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.899+0000 7f0e4167e740 1 -- 172.21.15.28:0/3096481146 >> 172.21.15.28:0/3096481146 conn(0x562f6ae70d30 msgr2=0x562f6ae76ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.900+0000 7f0e4167e740 1 -- 172.21.15.28:0/3096481146 shutdown_connections 2024-04-26T01:47:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.900+0000 7f0e4167e740 1 -- 172.21.15.28:0/3096481146 wait complete. 2024-04-26T01:47:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.900+0000 7f0e4167e740 20 client.15678 trim_cache size 0 max 16384 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.916+0000 7f0e4167e740 1 Processor -- start 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.916+0000 7f0e4167e740 1 -- start start 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.916+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae8b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.916+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae97840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.916+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 0x562f6ae97d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.916+0000 7f0e4167e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381b2650 con 0x562f6ae75140 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.916+0000 7f0e4167e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381afec0 con 0x562f6ae74820 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.916+0000 7f0e4167e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381add30 con 0x562f6ae7fa00 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae8b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e36a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae97840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae8b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57650/0 (socket says 172.21.15.28:57650) 2024-04-26T01:47:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 -- 172.21.15.28:0/1465694482 learned_addr learned my addr 172.21.15.28:0/1465694482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 0x562f6ae97d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 -- 172.21.15.28:0/1465694482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 msgr2=0x562f6ae97d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 0x562f6ae97d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 -- 172.21.15.28:0/1465694482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 msgr2=0x562f6ae97840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae97840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 -- 172.21.15.28:0/1465694482 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f6ae92610 con 0x562f6ae74820 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e36a00640 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae97840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.917+0000 7f0e37e00640 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae8b4f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009f40 tx=0x7f0e3001b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=876ee22463c86284 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e23e00640 1 -- 172.21.15.28:0/1465694482 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3001e040 con 0x562f6ae74820 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e23e00640 1 -- 172.21.15.28:0/1465694482 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e3002c070 con 0x562f6ae74820 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e23e00640 1 -- 172.21.15.28:0/1465694482 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e300047d0 con 0x562f6ae74820 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 -- 172.21.15.28:0/1465694482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 msgr2=0x562f6ae8b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae8b4f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009f40 tx=0x7f0e3001b040 comp rx=0 tx=0).stop 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 -- 172.21.15.28:0/1465694482 shutdown_connections 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae7fa00 0x562f6ae97d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae75140 0x562f6ae97840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 --2- 172.21.15.28:0/1465694482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae8b4f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 -- 172.21.15.28:0/1465694482 >> 172.21.15.28:0/1465694482 conn(0x562f6ae70d30 msgr2=0x562f6ae77ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 -- 172.21.15.28:0/1465694482 shutdown_connections 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.918+0000 7f0e4167e740 1 -- 172.21.15.28:0/1465694482 wait complete. 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.919+0000 7f0e4167e740 1 -- start start 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.919+0000 7f0e4167e740 1 Processor -- start 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae85410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae75140 0x562f6ae85950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e4167e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae7fa00 0x562f6ae85e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e4167e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381ae380 con 0x562f6ae7fa00 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e4167e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381af7f0 con 0x562f6ae74820 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e4167e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381aea50 con 0x562f6ae75140 2024-04-26T01:47:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e4167e740 10 client.0 _ll_register_callbacks cb 0x7f0e300093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e37e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae7fa00 0x562f6ae85e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e36a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae75140 0x562f6ae85950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e36a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae75140 0x562f6ae85950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58184/0 (socket says 172.21.15.28:58184) 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e36a00640 1 -- 172.21.15.28:0/3865216495 learned_addr learned my addr 172.21.15.28:0/3865216495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae85410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e36a00640 1 -- 172.21.15.28:0/3865216495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 msgr2=0x562f6ae85410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e36a00640 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae85410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e36a00640 1 -- 172.21.15.28:0/3865216495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae7fa00 msgr2=0x562f6ae85e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e36a00640 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae7fa00 0x562f6ae85e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e36a00640 1 -- 172.21.15.28:0/3865216495 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f6aef50c0 con 0x562f6ae75140 2024-04-26T01:47:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.920+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae7fa00 0x562f6ae85e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:25.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.921+0000 7f0e36a00640 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae75140 0x562f6ae85950 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0e2402a9c0 tx=0x7f0e240045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1101df245fba8126 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:25.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.921+0000 7f0e21600640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:47:25.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.921+0000 7f0e21600640 1 -- 172.21.15.28:0/3865216495 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e24035040 con 0x562f6ae75140 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.921+0000 7f0e4167e740 1 -- 172.21.15.28:0/3865216495 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e3002c070 con 0x562f6ae75140 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.921+0000 7f0e21600640 1 -- 172.21.15.28:0/3865216495 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f0e240050c0 con 0x562f6ae75140 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.921+0000 7f0e21600640 1 -- 172.21.15.28:0/3865216495 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0e24054050 con 0x562f6ae75140 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e4167e740 10 client.25897 fetch_fsmap learned FSMap version 34 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e4167e740 1 -- 172.21.15.28:0/3865216495 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562f6aef6410 con 0x562f6ae75140 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e21600640 1 -- 172.21.15.28:0/3865216495 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e3002c070 con 0x562f6ae75140 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e21600640 1 -- 172.21.15.28:0/3865216495 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f0e24004750 con 0x562f6ae75140 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e4167e740 10 client.25897 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e4167e740 10 client.25897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e4167e740 20 client.25897 populate_metadata read hostname 'smithi028' 2024-04-26T01:47:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562f6aef6ba0 0x562f6aef8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e4167e740 4 client.25897 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e4167e740 1 -- 172.21.15.28:0/3865216495 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562f6ade8320 con 0x562f6aef6ba0 2024-04-26T01:47:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.922+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562f6aef6ba0 0x562f6aef8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.923+0000 7f0e37e00640 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562f6aef6ba0 0x562f6aef8fc0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb7e62d07f1ca2fe server_cookie=8efb3caee4af75cb in_seq=0 out_seq=0 2024-04-26T01:47:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:47:25.923+0000 7f0e21600640 10 client.25897 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 330.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 -- 172.21.15.28:0/4221806449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f387010f4a0 msgr2=0x7f3870111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 --2- 172.21.15.28:0/4221806449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f387010f4a0 0x7f3870111890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3868009990 tx=0x7f3868009ca0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 -- 172.21.15.28:0/4221806449 shutdown_connections 2024-04-26T01:47:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 --2- 172.21.15.28:0/4221806449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f387010f4a0 0x7f3870111890 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 --2- 172.21.15.28:0/4221806449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870074c70 0x7f3870075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 --2- 172.21.15.28:0/4221806449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3870075590 0x7f387010ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 -- 172.21.15.28:0/4221806449 >> 172.21.15.28:0/4221806449 conn(0x7f3870079b90 msgr2=0x7f3870079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 -- 172.21.15.28:0/4221806449 shutdown_connections 2024-04-26T01:47:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.293+0000 7f3876e00700 1 -- 172.21.15.28:0/4221806449 wait complete. 2024-04-26T01:47:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.294+0000 7f3876e00700 1 Processor -- start 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.294+0000 7f3876e00700 1 -- start start 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f3876e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870074c70 0x7f387019b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f3876e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3870075590 0x7f387019b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f3876e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f387010f4a0 0x7f387019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f3876e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3870114ed0 con 0x7f3870074c70 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f3876e00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3870114d50 con 0x7f387010f4a0 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f3876e00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3870115050 con 0x7f3870075590 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f386fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870074c70 0x7f387019b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f386fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870074c70 0x7f387019b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39724/0 (socket says 172.21.15.28:39724) 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f3874e00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f387010f4a0 0x7f387019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f386fe00700 1 -- 172.21.15.28:0/111731353 learned_addr learned my addr 172.21.15.28:0/111731353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.295+0000 7f3867400700 1 --2- 172.21.15.28:0/111731353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3870075590 0x7f387019b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f3874e00700 1 -- 172.21.15.28:0/111731353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3870075590 msgr2=0x7f387019b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f3874e00700 1 --2- 172.21.15.28:0/111731353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3870075590 0x7f387019b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f3874e00700 1 -- 172.21.15.28:0/111731353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870074c70 msgr2=0x7f387019b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f3874e00700 1 --2- 172.21.15.28:0/111731353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870074c70 0x7f387019b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f3874e00700 1 -- 172.21.15.28:0/111731353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38701a0430 con 0x7f387010f4a0 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f386fe00700 1 --2- 172.21.15.28:0/111731353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870074c70 0x7f387019b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f3874e00700 1 --2- 172.21.15.28:0/111731353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f387010f4a0 0x7f387019fcf0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f387006b560 tx=0x7f38680057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:27.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f386d600700 1 -- 172.21.15.28:0/111731353 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868027070 con 0x7f387010f4a0 2024-04-26T01:47:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f386d600700 1 -- 172.21.15.28:0/111731353 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3868022470 con 0x7f387010f4a0 2024-04-26T01:47:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.296+0000 7f3876e00700 1 -- 172.21.15.28:0/111731353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38701a06c0 con 0x7f387010f4a0 2024-04-26T01:47:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.297+0000 7f386d600700 1 -- 172.21.15.28:0/111731353 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868017650 con 0x7f387010f4a0 2024-04-26T01:47:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.297+0000 7f3876e00700 1 -- 172.21.15.28:0/111731353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38701a7fa0 con 0x7f387010f4a0 2024-04-26T01:47:27.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.301+0000 7f386d600700 1 -- 172.21.15.28:0/111731353 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f386802a070 con 0x7f387010f4a0 2024-04-26T01:47:27.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.302+0000 7f38816cdb80 1 Processor -- start 2024-04-26T01:47:27.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f38816cdb80 1 -- start start 2024-04-26T01:47:27.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b233ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f386d600700 1 --2- 172.21.15.28:0/111731353 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f38540a3e30 0x7f38540a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b2345bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b23460f0 0x5627b23484e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f38816cdb80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5627b228df90 con 0x5627b23460f0 2024-04-26T01:47:27.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f38816cdb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5627b2234830 con 0x5627b233a920 2024-04-26T01:47:27.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f38816cdb80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5627b2289f50 con 0x5627b233b240 2024-04-26T01:47:27.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.303+0000 7f386d600700 1 -- 172.21.15.28:0/111731353 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f38680d1530 con 0x7f387010f4a0 2024-04-26T01:47:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f386fe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b233ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3867400700 1 --2- 172.21.15.28:0/111731353 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f38540a3e30 0x7f38540a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3874e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b23460f0 0x5627b23484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3874e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b23460f0 0x5627b23484e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39728/0 (socket says 172.21.15.28:39728) 2024-04-26T01:47:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3874e00700 1 -- 172.21.15.28:0/3884960853 learned_addr learned my addr 172.21.15.28:0/3884960853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3867400700 1 --2- 172.21.15.28:0/3884960853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b2345bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3874e00700 1 -- 172.21.15.28:0/3884960853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 msgr2=0x5627b2345bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3874e00700 1 --2- 172.21.15.28:0/3884960853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b2345bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3874e00700 1 -- 172.21.15.28:0/3884960853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 msgr2=0x5627b233ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3874e00700 1 --2- 172.21.15.28:0/3884960853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b233ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.304+0000 7f3874e00700 1 -- 172.21.15.28:0/3884960853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627b222a780 con 0x5627b23460f0 2024-04-26T01:47:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.305+0000 7f3874e00700 1 --2- 172.21.15.28:0/3884960853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b23460f0 0x5627b23484e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3868006010 tx=0x7f3868022d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2e1df174aaad9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.305+0000 7f3867400700 1 --2- 172.21.15.28:0/111731353 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f38540a3e30 0x7f38540a62f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f38701112a0 tx=0x7f3858007710 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.305+0000 7f3865600700 1 -- 172.21.15.28:0/3884960853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386809dd20 con 0x5627b23460f0 2024-04-26T01:47:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.305+0000 7f3865600700 1 -- 172.21.15.28:0/3884960853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3868028260 con 0x5627b23460f0 2024-04-26T01:47:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.305+0000 7f3865600700 1 -- 172.21.15.28:0/3884960853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38680094b0 con 0x5627b23460f0 2024-04-26T01:47:27.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 -- 172.21.15.28:0/3884960853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b23460f0 msgr2=0x5627b23484e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:27.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3884960853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b23460f0 0x5627b23484e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3868006010 tx=0x7f3868022d00 comp rx=0 tx=0).stop 2024-04-26T01:47:27.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 -- 172.21.15.28:0/3884960853 shutdown_connections 2024-04-26T01:47:27.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3884960853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b23460f0 0x5627b23484e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3884960853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b233ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3884960853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b2345bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 -- 172.21.15.28:0/3884960853 >> 172.21.15.28:0/3884960853 conn(0x5627b22696a0 msgr2=0x5627b2336dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:27.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 -- 172.21.15.28:0/3884960853 shutdown_connections 2024-04-26T01:47:27.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.306+0000 7f38816cdb80 1 -- 172.21.15.28:0/3884960853 wait complete. 2024-04-26T01:47:27.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.307+0000 7f38816cdb80 1 -- start start 2024-04-26T01:47:27.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.307+0000 7f38816cdb80 1 Processor -- start 2024-04-26T01:47:27.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.307+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b2345770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.307+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233b240 0x5627b2342a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.307+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b23460f0 0x5627b2342fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.307+0000 7f38816cdb80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5627b2348a20 con 0x5627b233b240 2024-04-26T01:47:27.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.307+0000 7f38816cdb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5627b2247960 con 0x5627b233a920 2024-04-26T01:47:27.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.307+0000 7f38816cdb80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5627b2348ba0 con 0x5627b23460f0 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f38816cdb80 10 client.0 _ll_register_callbacks cb 0x5627b2226820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f3867400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233b240 0x5627b2342a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f386fe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b2345770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f386fe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b2345770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57696/0 (socket says 172.21.15.28:57696) 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f3867400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233b240 0x5627b2342a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39742/0 (socket says 172.21.15.28:39742) 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f386fe00700 1 -- 172.21.15.28:0/3724805185 learned_addr learned my addr 172.21.15.28:0/3724805185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f3874e00700 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b23460f0 0x5627b2342fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f386fe00700 1 -- 172.21.15.28:0/3724805185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b23460f0 msgr2=0x5627b2342fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f386fe00700 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b23460f0 0x5627b2342fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f386fe00700 1 -- 172.21.15.28:0/3724805185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233b240 msgr2=0x5627b2342a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f386fe00700 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233b240 0x5627b2342a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.308+0000 7f386fe00700 1 -- 172.21.15.28:0/3724805185 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627b21db9d0 con 0x5627b233a920 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.309+0000 7f386fe00700 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b2345770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3860007890 tx=0x7f3860007ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a316827fa954224 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.309+0000 7f385e000700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:47:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.309+0000 7f385e000700 1 -- 172.21.15.28:0/3724805185 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386001a070 con 0x5627b233a920 2024-04-26T01:47:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.309+0000 7f385e000700 1 -- 172.21.15.28:0/3724805185 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f3860004510 con 0x5627b233a920 2024-04-26T01:47:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.309+0000 7f38816cdb80 1 -- 172.21.15.28:0/3724805185 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3868028260 con 0x5627b233a920 2024-04-26T01:47:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.309+0000 7f385e000700 1 -- 172.21.15.28:0/3724805185 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3860004710 con 0x5627b233a920 2024-04-26T01:47:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.309+0000 7f38816cdb80 10 client.25889 fetch_fsmap learned FSMap version 34 2024-04-26T01:47:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.309+0000 7f38816cdb80 1 -- 172.21.15.28:0/3724805185 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5627b2224bf0 con 0x5627b233a920 2024-04-26T01:47:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f385e000700 1 -- 172.21.15.28:0/3724805185 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3868028260 con 0x5627b233a920 2024-04-26T01:47:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f385e000700 1 -- 172.21.15.28:0/3724805185 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f3860027510 con 0x5627b233a920 2024-04-26T01:47:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f38816cdb80 10 client.25889 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:47:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f38816cdb80 10 client.25889 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:47:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f38816cdb80 20 client.25889 populate_metadata read hostname 'smithi028' 2024-04-26T01:47:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5627b2355500 0x5627b2357970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f38816cdb80 4 client.25889 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f38816cdb80 1 -- 172.21.15.28:0/3724805185 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627b229b780 con 0x5627b2355500 2024-04-26T01:47:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.310+0000 7f3867400700 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5627b2355500 0x5627b2357970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.311+0000 7f3867400700 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5627b2355500 0x5627b2357970 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=379597699db97d4f server_cookie=e73b5f46b8ee5d33 in_seq=0 out_seq=0 2024-04-26T01:47:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:47:27.311+0000 7f385e000700 10 client.25889 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:47:27.521 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:27 smithi053 ceph-mon[32593]: pgmap v199: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-04-26T01:47:27.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:27 smithi028 ceph-mon[28247]: pgmap v199: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-04-26T01:47:27.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:27 smithi169 ceph-mon[32851]: pgmap v199: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-04-26T01:47:29.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:29 smithi028 ceph-mon[28247]: pgmap v200: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 966 KiB/s wr, 12 op/s 2024-04-26T01:47:29.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:29 smithi169 ceph-mon[32851]: pgmap v200: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 966 KiB/s wr, 12 op/s 2024-04-26T01:47:29.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:29 smithi053 ceph-mon[32593]: pgmap v200: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 966 KiB/s wr, 12 op/s 2024-04-26T01:47:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 335.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:31.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:31 smithi028 ceph-mon[28247]: pgmap v201: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 984 KiB/s wr, 18 op/s 2024-04-26T01:47:31.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:31 smithi169 ceph-mon[32851]: pgmap v201: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 984 KiB/s wr, 18 op/s 2024-04-26T01:47:31.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:31 smithi053 ceph-mon[32593]: pgmap v201: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 984 KiB/s wr, 18 op/s 2024-04-26T01:47:33.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:33 smithi028 ceph-mon[28247]: pgmap v202: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 68 KiB/s wr, 11 op/s 2024-04-26T01:47:33.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:33 smithi169 ceph-mon[32851]: pgmap v202: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 68 KiB/s wr, 11 op/s 2024-04-26T01:47:33.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:33 smithi053 ceph-mon[32593]: pgmap v202: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 68 KiB/s wr, 11 op/s 2024-04-26T01:47:35.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:35 smithi028 ceph-mon[28247]: pgmap v203: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-26T01:47:35.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:35 smithi169 ceph-mon[32851]: pgmap v203: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-26T01:47:35.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:35 smithi053 ceph-mon[32593]: pgmap v203: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-26T01:47:36.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 340.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:37 smithi028 ceph-mon[28247]: pgmap v204: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-04-26T01:47:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:37 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:37.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:37 smithi169 ceph-mon[32851]: pgmap v204: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-04-26T01:47:37.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:37 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:37.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:37 smithi053 ceph-mon[32593]: pgmap v204: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-04-26T01:47:37.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:37 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:39.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:39 smithi169 ceph-mon[32851]: pgmap v205: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 24 KiB/s wr, 10 op/s 2024-04-26T01:47:39.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:39 smithi053 ceph-mon[32593]: pgmap v205: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 24 KiB/s wr, 10 op/s 2024-04-26T01:47:39.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:39 smithi028 ceph-mon[28247]: pgmap v205: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 24 KiB/s wr, 10 op/s 2024-04-26T01:47:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 345.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:41.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:41 smithi169 ceph-mon[32851]: pgmap v206: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-04-26T01:47:41.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:41 smithi053 ceph-mon[32593]: pgmap v206: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-04-26T01:47:41.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:41 smithi028 ceph-mon[28247]: pgmap v206: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 KiB/s wr, 13 op/s 2024-04-26T01:47:43.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:43 smithi169 ceph-mon[32851]: pgmap v207: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-04-26T01:47:43.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:43 smithi053 ceph-mon[32593]: pgmap v207: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-04-26T01:47:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:43 smithi028 ceph-mon[28247]: pgmap v207: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-04-26T01:47:45.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:45 smithi169 ceph-mon[32851]: pgmap v208: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-04-26T01:47:45.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:45 smithi053 ceph-mon[32593]: pgmap v208: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-04-26T01:47:45.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:45 smithi028 ceph-mon[28247]: pgmap v208: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-04-26T01:47:46.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 350.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:47.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:47 smithi028 ceph-mon[28247]: pgmap v209: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2024-04-26T01:47:47.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:47 smithi169 ceph-mon[32851]: pgmap v209: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2024-04-26T01:47:47.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:47 smithi053 ceph-mon[32593]: pgmap v209: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2024-04-26T01:47:49.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:49 smithi169 ceph-mon[32851]: pgmap v210: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7.9 KiB/s wr, 5 op/s 2024-04-26T01:47:49.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:49 smithi053 ceph-mon[32593]: pgmap v210: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7.9 KiB/s wr, 5 op/s 2024-04-26T01:47:49.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:49 smithi028 ceph-mon[28247]: pgmap v210: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7.9 KiB/s wr, 5 op/s 2024-04-26T01:47:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 355.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:51.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:51 smithi169 ceph-mon[32851]: pgmap v211: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-26T01:47:51.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:51 smithi053 ceph-mon[32593]: pgmap v211: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-26T01:47:51.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:51 smithi028 ceph-mon[28247]: pgmap v211: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-26T01:47:52.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:52 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:52.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:52 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:52.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:52 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:53.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:53 smithi169 ceph-mon[32851]: pgmap v212: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-04-26T01:47:53.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:53 smithi053 ceph-mon[32593]: pgmap v212: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-04-26T01:47:53.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:53 smithi028 ceph-mon[28247]: pgmap v212: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-04-26T01:47:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:55 smithi169 ceph-mon[32851]: pgmap v213: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-26T01:47:55.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:55 smithi053 ceph-mon[32593]: pgmap v213: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-26T01:47:55.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:55 smithi028 ceph-mon[28247]: pgmap v213: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-26T01:47:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 360.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:47:57.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:57 smithi169 ceph-mon[32851]: pgmap v214: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-04-26T01:47:57.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:57 smithi053 ceph-mon[32593]: pgmap v214: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-04-26T01:47:57.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:57 smithi028 ceph-mon[28247]: pgmap v214: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-04-26T01:47:59.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:47:59 smithi169 ceph-mon[32851]: pgmap v215: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-04-26T01:47:59.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:47:59 smithi053 ceph-mon[32593]: pgmap v215: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-04-26T01:47:59.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:47:59 smithi028 ceph-mon[28247]: pgmap v215: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-04-26T01:48:01.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 365.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:48:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:01 smithi169 ceph-mon[32851]: pgmap v216: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-26T01:48:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:01 smithi053 ceph-mon[32593]: pgmap v216: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-26T01:48:01.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:01 smithi028 ceph-mon[28247]: pgmap v216: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-26T01:48:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:03 smithi169 ceph-mon[32851]: pgmap v217: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2024-04-26T01:48:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:03 smithi053 ceph-mon[32593]: pgmap v217: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2024-04-26T01:48:03.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:03 smithi028 ceph-mon[28247]: pgmap v217: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2024-04-26T01:48:05.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:05 smithi169 ceph-mon[32851]: pgmap v218: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-04-26T01:48:05.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:05 smithi053 ceph-mon[32593]: pgmap v218: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-04-26T01:48:05.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:05 smithi028 ceph-mon[28247]: pgmap v218: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-04-26T01:48:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 370.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:48:07.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:07 smithi169 ceph-mon[32851]: pgmap v219: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 51 KiB/s wr, 24 op/s 2024-04-26T01:48:07.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:07 smithi053 ceph-mon[32593]: pgmap v219: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 51 KiB/s wr, 24 op/s 2024-04-26T01:48:07.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:07 smithi028 ceph-mon[28247]: pgmap v219: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 51 KiB/s wr, 24 op/s 2024-04-26T01:48:08.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:08 smithi169 ceph-mon[32851]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:48:08.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:08 smithi053 ceph-mon[32593]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:48:08.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:08 smithi028 ceph-mon[28247]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:48:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:09 smithi169 ceph-mon[32851]: pgmap v221: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 62 KiB/s wr, 28 op/s 2024-04-26T01:48:09.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:09 smithi053 ceph-mon[32593]: pgmap v221: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 62 KiB/s wr, 28 op/s 2024-04-26T01:48:09.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:09 smithi028 ceph-mon[28247]: pgmap v221: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 62 KiB/s wr, 28 op/s 2024-04-26T01:48:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 375.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:48:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:11 smithi169 ceph-mon[32851]: pgmap v222: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-26T01:48:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:11 smithi053 ceph-mon[32593]: pgmap v222: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-26T01:48:11.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:11 smithi028 ceph-mon[28247]: pgmap v222: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-26T01:48:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:13 smithi169 ceph-mon[32851]: pgmap v223: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-26T01:48:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:13 smithi169 ceph-mon[32851]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:48:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:13 smithi053 ceph-mon[32593]: pgmap v223: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-26T01:48:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:13 smithi053 ceph-mon[32593]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:48:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:13 smithi028 ceph-mon[28247]: pgmap v223: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-26T01:48:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:13 smithi028 ceph-mon[28247]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:48:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:15 smithi169 ceph-mon[32851]: pgmap v225: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-04-26T01:48:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:15 smithi053 ceph-mon[32593]: pgmap v225: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-04-26T01:48:15.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:15 smithi028 ceph-mon[28247]: pgmap v225: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-04-26T01:48:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 380.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:48:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:17 smithi053 ceph-mon[32593]: pgmap v226: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-04-26T01:48:17.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:17 smithi028 ceph-mon[28247]: pgmap v226: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-04-26T01:48:17.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:17 smithi169 ceph-mon[32851]: pgmap v226: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-04-26T01:48:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.431+0000 7f0e21600640 1 -- 172.21.15.28:0/3865216495 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562f6ade8320 con 0x562f6aef6ba0 2024-04-26T01:48:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.431+0000 7f0e21600640 10 client.25897 handle_command_reply: tid=0 2024-04-26T01:48:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.431+0000 7f0e4167e740 1 client.25897 shutdown 2024-04-26T01:48:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.431+0000 7f0e4167e740 1 -- 172.21.15.28:0/3865216495 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae75140 msgr2=0x562f6ae85950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.431+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae75140 0x562f6ae85950 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0e2402a9c0 tx=0x7f0e240045d0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e4167e740 1 -- 172.21.15.28:0/3865216495 shutdown_connections 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562f6aef6ba0 0x562f6aef8fc0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f6ae7fa00 0x562f6ae85e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f6ae75140 0x562f6ae85950 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e4167e740 1 --2- 172.21.15.28:0/3865216495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562f6ae74820 0x562f6ae85410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e4167e740 1 -- 172.21.15.28:0/3865216495 >> 172.21.15.28:0/3865216495 conn(0x562f6ae70d30 msgr2=0x562f6ae6f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e21600640 0 client.25897 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e4167e740 1 -- 172.21.15.28:0/3865216495 shutdown_connections 2024-04-26T01:48:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f0e4167e740 1 -- 172.21.15.28:0/3865216495 wait complete. 2024-04-26T01:48:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f0e4167e740 20 client.25897 trim_cache size 0 max 16384 2024-04-26T01:48:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:48:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:48:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:48:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:48:18.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f385e000700 1 -- 172.21.15.28:0/3724805185 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627b229b780 con 0x5627b2355500 2024-04-26T01:48:18.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f385e000700 10 client.25889 handle_command_reply: tid=0 2024-04-26T01:48:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.434+0000 7f0e3e800640 1 -- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e1c0a3e40 msgr2=0x7f0e1c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.434+0000 7f0e3e800640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e1c0a3e40 0x7f0e1c0a6300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x562f6ae75dc0 tx=0x7f0e280d4580 comp rx=0 tx=0).stop 2024-04-26T01:48:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.434+0000 7f0e3e800640 1 -- 172.21.15.28:0/3408230514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e381021a0 msgr2=0x7f0e381a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.434+0000 7f0e3e800640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e381021a0 0x7f0e381a0630 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800e9f0 tx=0x7f0e2800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.432+0000 7f38816cdb80 1 client.25889 shutdown 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f38816cdb80 1 -- 172.21.15.28:0/3724805185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 msgr2=0x5627b2345770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b2345770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3860007890 tx=0x7f3860007ba0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f38816cdb80 1 -- 172.21.15.28:0/3724805185 shutdown_connections 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5627b2355500 0x5627b2357970 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b23460f0 0x5627b2342fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b233a920 0x5627b2345770 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3724805185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233b240 0x5627b2342a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f385e000700 0 client.25889 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.433+0000 7f38816cdb80 1 -- 172.21.15.28:0/3724805185 >> 172.21.15.28:0/3724805185 conn(0x5627b22696a0 msgr2=0x5627b23369c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.434+0000 7f38816cdb80 1 -- 172.21.15.28:0/3724805185 shutdown_connections 2024-04-26T01:48:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.434+0000 7f38816cdb80 1 -- 172.21.15.28:0/3724805185 wait complete. 2024-04-26T01:48:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.434+0000 7f38816cdb80 20 client.25889 trim_cache size 0 max 16384 2024-04-26T01:48:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.439+0000 7f0e3e800640 1 -- 172.21.15.28:0/3408230514 shutdown_connections 2024-04-26T01:48:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.439+0000 7f0e3e800640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e1c0a3e40 0x7f0e1c0a6300 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.439+0000 7f0e3e800640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3819a740 0x7f0e3819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.439+0000 7f0e3e800640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38107270 0x7f0e381a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.439+0000 7f0e3e800640 1 --2- 172.21.15.28:0/3408230514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e381021a0 0x7f0e381a0630 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.439+0000 7f0e3e800640 1 -- 172.21.15.28:0/3408230514 >> 172.21.15.28:0/3408230514 conn(0x7f0e380793e0 msgr2=0x7f0e380fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.440+0000 7f0e3e800640 1 -- 172.21.15.28:0/3408230514 shutdown_connections 2024-04-26T01:48:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:18.440+0000 7f0e3e800640 1 -- 172.21.15.28:0/3408230514 wait complete. 2024-04-26T01:48:18.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.489+0000 7f38816cdb80 1 Processor -- start 2024-04-26T01:48:18.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.490+0000 7f38816cdb80 1 -- start start 2024-04-26T01:48:18.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.490+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b235d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.490+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b235d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.490+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b233fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.490+0000 7f38816cdb80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38701b1d80 con 0x5627b233a920 2024-04-26T01:48:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.490+0000 7f38816cdb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f38701af5b0 con 0x5627b23460f0 2024-04-26T01:48:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.490+0000 7f38816cdb80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f38701ad420 con 0x5627b233b240 2024-04-26T01:48:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.491+0000 7f3867400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b235d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.491+0000 7f3867400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b235d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:34698/0 (socket says 172.21.15.28:34698) 2024-04-26T01:48:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.491+0000 7f3867400700 1 -- 172.21.15.28:0/4039014679 learned_addr learned my addr 172.21.15.28:0/4039014679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.491+0000 7f3874e00700 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b233fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.492+0000 7f386fe00700 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b235d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.492+0000 7f3867400700 1 -- 172.21.15.28:0/4039014679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 msgr2=0x5627b233fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.492+0000 7f3867400700 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b233fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.492+0000 7f3867400700 1 -- 172.21.15.28:0/4039014679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 msgr2=0x5627b235d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.493+0000 7f3867400700 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b235d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.493+0000 7f3867400700 1 -- 172.21.15.28:0/4039014679 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627b2352f50 con 0x5627b233b240 2024-04-26T01:48:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.494+0000 7f3867400700 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b235d7c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3858007b20 tx=0x7f38580047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47bf71dc349bcdc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.494+0000 7f3864c00700 1 -- 172.21.15.28:0/4039014679 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385802e070 con 0x5627b233b240 2024-04-26T01:48:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.494+0000 7f3864c00700 1 -- 172.21.15.28:0/4039014679 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f385800ec70 con 0x5627b233b240 2024-04-26T01:48:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.494+0000 7f3864c00700 1 -- 172.21.15.28:0/4039014679 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858029770 con 0x5627b233b240 2024-04-26T01:48:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.495+0000 7f386fe00700 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b235d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.496+0000 7f38816cdb80 1 -- 172.21.15.28:0/4039014679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 msgr2=0x5627b235d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.496+0000 7f38816cdb80 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b235d7c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3858007b20 tx=0x7f38580047c0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.497+0000 7f38816cdb80 1 -- 172.21.15.28:0/4039014679 shutdown_connections 2024-04-26T01:48:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.497+0000 7f38816cdb80 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b233fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.497+0000 7f38816cdb80 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b235d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.497+0000 7f38816cdb80 1 --2- 172.21.15.28:0/4039014679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b233b240 0x5627b235d7c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.497+0000 7f38816cdb80 1 -- 172.21.15.28:0/4039014679 >> 172.21.15.28:0/4039014679 conn(0x5627b22696a0 msgr2=0x5627b2344b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.497+0000 7f38816cdb80 1 -- 172.21.15.28:0/4039014679 shutdown_connections 2024-04-26T01:48:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.497+0000 7f38816cdb80 1 -- 172.21.15.28:0/4039014679 wait complete. 2024-04-26T01:48:18.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.499+0000 7f38816cdb80 1 -- start start 2024-04-26T01:48:18.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.499+0000 7f38816cdb80 1 Processor -- start 2024-04-26T01:48:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.501+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b234c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.501+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b234cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.501+0000 7f38816cdb80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b234d1c0 0x5627b23bbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.501+0000 7f38816cdb80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38701ada70 con 0x5627b233a920 2024-04-26T01:48:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.501+0000 7f38816cdb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f38701aeee0 con 0x5627b23460f0 2024-04-26T01:48:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.501+0000 7f38816cdb80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f38701ae140 con 0x5627b234d1c0 2024-04-26T01:48:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.502+0000 7f38816cdb80 10 client.0 _ll_register_callbacks cb 0x5627b2226820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.502+0000 7f386fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b234c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.503+0000 7f386fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b234c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37454/0 (socket says 172.21.15.28:37454) 2024-04-26T01:48:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.503+0000 7f386fe00700 1 -- 172.21.15.28:0/3836648223 learned_addr learned my addr 172.21.15.28:0/3836648223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.504+0000 7f3874e00700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b234d1c0 0x5627b23bbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.504+0000 7f3867400700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b234cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.505+0000 7f386fe00700 1 -- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b234d1c0 msgr2=0x5627b23bbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.505+0000 7f386fe00700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b234d1c0 0x5627b23bbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.505+0000 7f386fe00700 1 -- 172.21.15.28:0/3836648223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 msgr2=0x5627b234cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.505+0000 7f386fe00700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b234cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.505+0000 7f386fe00700 1 -- 172.21.15.28:0/3836648223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627b23bc050 con 0x5627b233a920 2024-04-26T01:48:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.505+0000 7f3867400700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b234cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.505+0000 7f3874e00700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b234d1c0 0x5627b23bbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.506+0000 7f386fe00700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b234c740 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3860007890 tx=0x7f386002b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7f5b380682200c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.506+0000 7f385e000700 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:48:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.507+0000 7f385e000700 1 -- 172.21.15.28:0/3836648223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386001a040 con 0x5627b233a920 2024-04-26T01:48:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.507+0000 7f385e000700 1 -- 172.21.15.28:0/3836648223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f3860009580 con 0x5627b233a920 2024-04-26T01:48:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.507+0000 7f38816cdb80 1 -- 172.21.15.28:0/3836648223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f385800ec70 con 0x5627b233a920 2024-04-26T01:48:18.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.508+0000 7f385e000700 1 -- 172.21.15.28:0/3836648223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f386002b070 con 0x5627b233a920 2024-04-26T01:48:18.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.509+0000 7f38816cdb80 10 client.15702 fetch_fsmap learned FSMap version 36 2024-04-26T01:48:18.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.510+0000 7f38816cdb80 1 -- 172.21.15.28:0/3836648223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5627b23bd270 con 0x5627b233a920 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.512+0000 7f385e000700 1 -- 172.21.15.28:0/3836648223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f385800ec70 con 0x5627b233a920 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.512+0000 7f385e000700 1 -- 172.21.15.28:0/3836648223 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f386000ea10 con 0x5627b233a920 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.512+0000 7f38816cdb80 10 client.15702 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.512+0000 7f38816cdb80 10 client.15702 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.512+0000 7f38816cdb80 20 client.15702 populate_metadata read hostname 'smithi028' 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.512+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5627b23bd950 0x5627b23bfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.512+0000 7f38816cdb80 4 client.15702 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6288c6a", "release": true, "await": true}] 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.512+0000 7f38816cdb80 1 -- 172.21.15.28:0/3836648223 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6288c6a", "release": true, "await": true}) -- 0x5627b2295820 con 0x5627b23bd950 2024-04-26T01:48:18.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.513+0000 7f3867400700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5627b23bd950 0x5627b23bfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.515+0000 7f385e000700 10 client.15702 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:18.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.515+0000 7f3867400700 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5627b23bd950 0x5627b23bfd70 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db1372c2475c66a server_cookie=484fed8bcdaa5d98 in_seq=0 out_seq=0 2024-04-26T01:48:18.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.516+0000 7f385e000700 1 -- 172.21.15.28:0/3836648223 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: -1 ) ==== 8+0+589 (crc 0 0 0) 0x5627b2295820 con 0x5627b23bd950 2024-04-26T01:48:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.516+0000 7f385e000700 10 client.15702 handle_command_reply: tid=0 2024-04-26T01:48:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.516+0000 7f38816cdb80 1 client.15702 shutdown 2024-04-26T01:48:18.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.517+0000 7f38816cdb80 1 -- 172.21.15.28:0/3836648223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 msgr2=0x5627b234c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.517+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b234c740 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3860007890 tx=0x7f386002b000 comp rx=0 tx=0).stop 2024-04-26T01:48:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.518+0000 7f38816cdb80 1 -- 172.21.15.28:0/3836648223 shutdown_connections 2024-04-26T01:48:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.518+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5627b23bd950 0x5627b23bfd70 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.518+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5627b234d1c0 0x5627b23bbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.518+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627b233a920 0x5627b234c740 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.518+0000 7f38816cdb80 1 --2- 172.21.15.28:0/3836648223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627b23460f0 0x5627b234cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.518+0000 7f38816cdb80 1 -- 172.21.15.28:0/3836648223 >> 172.21.15.28:0/3836648223 conn(0x5627b22696a0 msgr2=0x5627b2335f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.518+0000 7f385e000700 0 client.15702 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.519+0000 7f38816cdb80 1 -- 172.21.15.28:0/3836648223 shutdown_connections 2024-04-26T01:48:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.519+0000 7f38816cdb80 1 -- 172.21.15.28:0/3836648223 wait complete. 2024-04-26T01:48:18.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:48:18.519+0000 7f38816cdb80 20 client.15702 trim_cache size 0 max 16384 2024-04-26T01:48:18.519 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 36, 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 15, 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "6288c6a": { 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 15, 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.1, 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "EXPIRED", 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 60.0, 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 70.0, 2024-04-26T01:48:18.520 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 70.0 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:48:18.521 INFO:teuthology.orchestra.run.smithi028.stderr:Error EPERM: 2024-04-26T01:48:19.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:19 smithi028 ceph-mon[28247]: pgmap v227: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2024-04-26T01:48:19.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:19 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:48:19.566 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't release set '6288c6a' with rc: 1 (EPERM), stdout: { "epoch": 36, "leader": 24523, "set_version": 15, "sets": { "6288c6a": { "version": 15, "age_ref": 0.1, "state": { "name": "EXPIRED", "age": 0.0 }, "timeout": 60.0, "expiration": 70.0, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCED", "age": 70.0 } } } } } } 2024-04-26T01:48:19.566 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 354, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 330, in do_quiesce raise RuntimeError(f"Error releasing set '{set_id}': {rcinfo(rc)}") RuntimeError: Error releasing set '6288c6a': 1 (EPERM) 2024-04-26T01:48:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:19 smithi053 ceph-mon[32593]: pgmap v227: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2024-04-26T01:48:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:19 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:19.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:19 smithi169 ceph-mon[32851]: pgmap v227: 97 pgs: 97 active+clean; 2.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2024-04-26T01:48:19.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:19 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 385.0 s, 1198.1 tps, lat 1218.500 ms stddev 9014.022 2024-04-26T01:48:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:21 smithi053 ceph-mon[32593]: pgmap v228: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 605 KiB/s wr, 52 op/s 2024-04-26T01:48:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:21 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:21 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:21.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:21 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:48:21.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:21 smithi028 ceph-mon[28247]: pgmap v228: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 605 KiB/s wr, 52 op/s 2024-04-26T01:48:21.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:21 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:21.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:21 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:21.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:21 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:48:21.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:21 smithi169 ceph-mon[32851]: pgmap v228: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 605 KiB/s wr, 52 op/s 2024-04-26T01:48:21.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:21 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:21.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:21 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:21.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:21 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:48:23.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:23 smithi053 ceph-mon[32593]: pgmap v229: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 605 KiB/s wr, 52 op/s 2024-04-26T01:48:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:23 smithi028 ceph-mon[28247]: pgmap v229: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 605 KiB/s wr, 52 op/s 2024-04-26T01:48:23.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:23 smithi169 ceph-mon[32851]: pgmap v229: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 605 KiB/s wr, 52 op/s 2024-04-26T01:48:25.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:25 smithi053 ceph-mon[32593]: pgmap v230: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.4 MiB/s wr, 268 op/s 2024-04-26T01:48:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:25 smithi028 ceph-mon[28247]: pgmap v230: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.4 MiB/s wr, 268 op/s 2024-04-26T01:48:25.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:25 smithi169 ceph-mon[32851]: pgmap v230: 97 pgs: 97 active+clean; 2.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.4 MiB/s wr, 268 op/s 2024-04-26T01:48:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 390.0 s, 2587.6 tps, lat 37.277 ms stddev 47.364 2024-04-26T01:48:27.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:27 smithi053 ceph-mon[32593]: pgmap v231: 97 pgs: 97 active+clean; 2.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-04-26T01:48:27.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:27 smithi028 ceph-mon[28247]: pgmap v231: 97 pgs: 97 active+clean; 2.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-04-26T01:48:27.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:27 smithi169 ceph-mon[32851]: pgmap v231: 97 pgs: 97 active+clean; 2.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 11 MiB/s wr, 340 op/s 2024-04-26T01:48:28.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:28 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:28.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:28 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:28.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:28 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:29.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:29 smithi028 ceph-mon[28247]: pgmap v232: 97 pgs: 97 active+clean; 2.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 339 op/s 2024-04-26T01:48:29.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:29 smithi169 ceph-mon[32851]: pgmap v232: 97 pgs: 97 active+clean; 2.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 339 op/s 2024-04-26T01:48:29.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:29 smithi053 ceph-mon[32593]: pgmap v232: 97 pgs: 97 active+clean; 2.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 339 op/s 2024-04-26T01:48:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 395.0 s, 2182.5 tps, lat 37.354 ms stddev 63.116 2024-04-26T01:48:31.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 -- 172.21.15.28:0/4088011256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 msgr2=0x7f2d4c1032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 --2- 172.21.15.28:0/4088011256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 0x7f2d4c1032f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c009900 tx=0x7f2d3c02a480 comp rx=0 tx=0).stop 2024-04-26T01:48:31.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 -- 172.21.15.28:0/4088011256 shutdown_connections 2024-04-26T01:48:31.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 --2- 172.21.15.28:0/4088011256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c330 0x7f2d4c10e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 --2- 172.21.15.28:0/4088011256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d4c103830 0x7f2d4c10bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 --2- 172.21.15.28:0/4088011256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 0x7f2d4c1032f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 -- 172.21.15.28:0/4088011256 >> 172.21.15.28:0/4088011256 conn(0x7f2d4c0fc770 msgr2=0x7f2d4c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:31.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 -- 172.21.15.28:0/4088011256 shutdown_connections 2024-04-26T01:48:31.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.611+0000 7f2d51a00640 1 -- 172.21.15.28:0/4088011256 wait complete. 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.612+0000 7f2d51a00640 1 Processor -- start 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.612+0000 7f2d51a00640 1 -- start start 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d51a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 0x7f2d4c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d51a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d4c103830 0x7f2d4c19acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d51a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c330 0x7f2d4c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d51a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c112010 con 0x7f2d4c10c330 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d51a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c111e90 con 0x7f2d4c103830 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d51a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c112190 con 0x7f2d4c102f10 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 0x7f2d4c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 0x7f2d4c19a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:53568/0 (socket says 172.21.15.28:53568) 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d4be00640 1 -- 172.21.15.28:0/2013405959 learned_addr learned my addr 172.21.15.28:0/2013405959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:31.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d51000640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c330 0x7f2d4c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d43e00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d4c103830 0x7f2d4c19acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d4be00640 1 -- 172.21.15.28:0/2013405959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d4c103830 msgr2=0x7f2d4c19acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d4be00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d4c103830 0x7f2d4c19acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d4be00640 1 -- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c330 msgr2=0x7f2d4c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d4be00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c330 0x7f2d4c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.613+0000 7f2d4be00640 1 -- 172.21.15.28:0/2013405959 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c19f750 con 0x7f2d4c102f10 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.614+0000 7f2d4be00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 0x7f2d4c19a7b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c005ab0 tx=0x7f2d3c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.614+0000 7f2d49600640 1 -- 172.21.15.28:0/2013405959 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c03f070 con 0x7f2d4c102f10 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.614+0000 7f2d49600640 1 -- 172.21.15.28:0/2013405959 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c031b60 con 0x7f2d4c102f10 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.614+0000 7f2d49600640 1 -- 172.21.15.28:0/2013405959 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c035030 con 0x7f2d4c102f10 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.614+0000 7f2d51a00640 1 -- 172.21.15.28:0/2013405959 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4c19f9e0 con 0x7f2d4c102f10 2024-04-26T01:48:31.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.614+0000 7f2d51a00640 1 -- 172.21.15.28:0/2013405959 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d4c1a72d0 con 0x7f2d4c102f10 2024-04-26T01:48:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.617+0000 7f2d55e82740 1 Processor -- start 2024-04-26T01:48:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.617+0000 7f2d55e82740 1 -- start start 2024-04-26T01:48:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.617+0000 7f2d49600640 1 -- 172.21.15.28:0/2013405959 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2d3c042070 con 0x7f2d4c102f10 2024-04-26T01:48:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fbc9b0 0x558df2fbcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fc9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d55e82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558df2f22900 con 0x558df2fc7b90 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d55e82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558df2ec8ef0 con 0x558df2fbc9b0 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d55e82740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558df2f1e8c0 con 0x558df2fbd2d0 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fc9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d51000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fbc9b0 0x558df2fbcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fc9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52056/0 (socket says 172.21.15.28:52056) 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:53576/0 (socket says 172.21.15.28:53576) 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d43e00640 1 -- 172.21.15.28:0/2648926614 learned_addr learned my addr 172.21.15.28:0/2648926614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:31.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d51000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fbc9b0 0x558df2fbcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48160/0 (socket says 172.21.15.28:48160) 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d43e00640 1 -- 172.21.15.28:0/2648926614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 msgr2=0x558df2fc7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d43e00640 1 --2- 172.21.15.28:0/2648926614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d43e00640 1 -- 172.21.15.28:0/2648926614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fbc9b0 msgr2=0x558df2fbcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d43e00640 1 --2- 172.21.15.28:0/2648926614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fbc9b0 0x558df2fbcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d43e00640 1 -- 172.21.15.28:0/2648926614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558df2c35210 con 0x558df2fc7b90 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.618+0000 7f2d51000640 1 --2- 172.21.15.28:0/2648926614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fbc9b0 0x558df2fbcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.619+0000 7f2d43e00640 1 --2- 172.21.15.28:0/2648926614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fc9f80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009c90 tx=0x7f2d34011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66692d396cc6a1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.619+0000 7f2d41600640 1 -- 172.21.15.28:0/2648926614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34005220 con 0x558df2fc7b90 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.619+0000 7f2d41600640 1 -- 172.21.15.28:0/2648926614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d34004c50 con 0x558df2fc7b90 2024-04-26T01:48:31.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.619+0000 7f2d41600640 1 -- 172.21.15.28:0/2648926614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34019740 con 0x558df2fc7b90 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.619+0000 7f2d49600640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2d300a3d90 0x7f2d300a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.619+0000 7f2d49600640 1 -- 172.21.15.28:0/2013405959 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f2d3c0f0df0 con 0x7f2d4c102f10 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.619+0000 7f2d51000640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2d300a3d90 0x7f2d300a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 -- 172.21.15.28:0/2648926614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 msgr2=0x558df2fc9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2648926614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fc9f80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009c90 tx=0x7f2d34011810 comp rx=0 tx=0).stop 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 -- 172.21.15.28:0/2648926614 shutdown_connections 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2648926614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fc9f80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2648926614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2648926614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fbc9b0 0x558df2fbcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 -- 172.21.15.28:0/2648926614 >> 172.21.15.28:0/2648926614 conn(0x558df2fb8ec0 msgr2=0x558df2fb92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 -- 172.21.15.28:0/2648926614 shutdown_connections 2024-04-26T01:48:31.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 -- 172.21.15.28:0/2648926614 wait complete. 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d51000640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2d300a3d90 0x7f2d300a6250 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2d44005fd0 tx=0x7f2d44009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.620+0000 7f2d55e82740 1 -- start start 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d55e82740 1 Processor -- start 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbc9b0 0x558df2fc4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc7b90 0x558df2fc5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d55e82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558df2fca7c0 con 0x558df2fbc9b0 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d55e82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558df2edc310 con 0x558df2fc7b90 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d55e82740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558df2fca940 con 0x558df2fbd2d0 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d55e82740 10 client.0 _ll_register_callbacks cb 0x558df2eaacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbc9b0 0x558df2fc4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbc9b0 0x558df2fc4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52062/0 (socket says 172.21.15.28:52062) 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d4be00640 1 -- 172.21.15.28:0/2809743086 learned_addr learned my addr 172.21.15.28:0/2809743086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d51000640 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc7b90 0x558df2fc5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.621+0000 7f2d43e00640 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d4be00640 1 -- 172.21.15.28:0/2809743086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 msgr2=0x558df2fc4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d4be00640 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d4be00640 1 -- 172.21.15.28:0/2809743086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc7b90 msgr2=0x558df2fc5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d4be00640 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc7b90 0x558df2fc5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d4be00640 1 -- 172.21.15.28:0/2809743086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558df2d01620 con 0x558df2fbc9b0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d4be00640 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbc9b0 0x558df2fc4aa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0b9190 tx=0x7f2d3c03b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b26d450b8097ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d3a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d3a000640 1 -- 172.21.15.28:0/2809743086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c049070 con 0x558df2fbc9b0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d3a000640 1 -- 172.21.15.28:0/2809743086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f2d3c0b93a0 con 0x558df2fbc9b0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.622+0000 7f2d55e82740 1 -- 172.21.15.28:0/2809743086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d34004c50 con 0x558df2fbc9b0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d3a000640 1 -- 172.21.15.28:0/2809743086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2d3c0bc870 con 0x558df2fbc9b0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d55e82740 10 client.15714 fetch_fsmap learned FSMap version 37 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d55e82740 1 -- 172.21.15.28:0/2809743086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558df2fd69a0 con 0x558df2fbc9b0 2024-04-26T01:48:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d3a000640 1 -- 172.21.15.28:0/2809743086 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d34004c50 con 0x558df2fbc9b0 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d3a000640 1 -- 172.21.15.28:0/2809743086 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8342+0+0 (secure 0 0 0) 0x7f2d3c0ee760 con 0x558df2fbc9b0 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d55e82740 10 client.15714 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d55e82740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d55e82740 20 client.15714 populate_metadata read hostname 'smithi028' 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558df2fd7e00 0x558df2fda250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d55e82740 4 client.15714 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.623+0000 7f2d55e82740 1 -- 172.21.15.28:0/2809743086 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558df2f368b0 con 0x558df2fd7e00 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.624+0000 7f2d43e00640 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558df2fd7e00 0x558df2fda250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.624+0000 7f2d43e00640 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558df2fd7e00 0x558df2fda250 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78b84a41b3470127 server_cookie=9625a502aa54a16d in_seq=0 out_seq=0 2024-04-26T01:48:31.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.624+0000 7f2d3a000640 10 client.15714 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:31.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d3a000640 1 -- 172.21.15.28:0/2809743086 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558df2f368b0 con 0x558df2fd7e00 2024-04-26T01:48:31.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d3a000640 10 client.15714 handle_command_reply: tid=0 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 client.15714 shutdown 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 -- 172.21.15.28:0/2809743086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbc9b0 msgr2=0x558df2fc4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbc9b0 0x558df2fc4aa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0b9190 tx=0x7f2d3c03b790 comp rx=0 tx=0).stop 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 -- 172.21.15.28:0/2809743086 shutdown_connections 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558df2fd7e00 0x558df2fda250 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc7b90 0x558df2fc5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fc4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d3a000640 0 client.15714 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 --2- 172.21.15.28:0/2809743086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbc9b0 0x558df2fc4aa0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 -- 172.21.15.28:0/2809743086 >> 172.21.15.28:0/2809743086 conn(0x558df2fb8ec0 msgr2=0x558df2fbebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 -- 172.21.15.28:0/2809743086 shutdown_connections 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 1 -- 172.21.15.28:0/2809743086 wait complete. 2024-04-26T01:48:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.625+0000 7f2d55e82740 20 client.15714 trim_cache size 0 max 16384 2024-04-26T01:48:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d55e82740 1 Processor -- start 2024-04-26T01:48:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d55e82740 1 -- start start 2024-04-26T01:48:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbd2d0 0x558df2fd3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fc7b90 0x558df2fdfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fe00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d55e82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b2390 con 0x558df2fbd2d0 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d55e82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1afc60 con 0x558df2fc72e0 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d55e82740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1adad0 con 0x558df2fc7b90 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.641+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbd2d0 0x558df2fd3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d4be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbd2d0 0x558df2fd3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52066/0 (socket says 172.21.15.28:52066) 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d4be00640 1 -- 172.21.15.28:0/3779304219 learned_addr learned my addr 172.21.15.28:0/3779304219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d43e00640 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fc7b90 0x558df2fdfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d51000640 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fe00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d4be00640 1 -- 172.21.15.28:0/3779304219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fc7b90 msgr2=0x558df2fdfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d4be00640 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fc7b90 0x558df2fdfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d4be00640 1 -- 172.21.15.28:0/3779304219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 msgr2=0x558df2fe00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d4be00640 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fe00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d4be00640 1 -- 172.21.15.28:0/3779304219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558df2fbd100 con 0x558df2fbd2d0 2024-04-26T01:48:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d4be00640 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbd2d0 0x558df2fd3680 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0bccc0 tx=0x7f2d3c0bb240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73933baa40193d95 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d40c00640 1 -- 172.21.15.28:0/3779304219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c049040 con 0x558df2fbd2d0 2024-04-26T01:48:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d40c00640 1 -- 172.21.15.28:0/3779304219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c0bcd30 con 0x558df2fbd2d0 2024-04-26T01:48:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.642+0000 7f2d40c00640 1 -- 172.21.15.28:0/3779304219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0bc690 con 0x558df2fbd2d0 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 -- 172.21.15.28:0/3779304219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbd2d0 msgr2=0x558df2fd3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbd2d0 0x558df2fd3680 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0bccc0 tx=0x7f2d3c0bb240 comp rx=0 tx=0).stop 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 -- 172.21.15.28:0/3779304219 shutdown_connections 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fe00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fc7b90 0x558df2fdfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 --2- 172.21.15.28:0/3779304219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fbd2d0 0x558df2fd3680 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 -- 172.21.15.28:0/3779304219 >> 172.21.15.28:0/3779304219 conn(0x558df2fb8ec0 msgr2=0x558df2fb92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 -- 172.21.15.28:0/3779304219 shutdown_connections 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 -- 172.21.15.28:0/3779304219 wait complete. 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.643+0000 7f2d55e82740 1 -- start start 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d55e82740 1 Processor -- start 2024-04-26T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fcfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fcd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d55e82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fcd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d55e82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1ae120 con 0x558df2fc7b90 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d55e82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1af590 con 0x558df2fc72e0 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d55e82740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1ae7f0 con 0x558df2fbd2d0 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d55e82740 10 client.0 _ll_register_callbacks cb 0x7f2d3c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d51000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fcd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fcd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fcd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48188/0 (socket says 172.21.15.28:48188) 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d51000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fcd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52076/0 (socket says 172.21.15.28:52076) 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d43e00640 1 -- 172.21.15.28:0/55192141 learned_addr learned my addr 172.21.15.28:0/55192141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.644+0000 7f2d4be00640 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fcfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d43e00640 1 -- 172.21.15.28:0/55192141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 msgr2=0x558df2fcfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d43e00640 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fcfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d43e00640 1 -- 172.21.15.28:0/55192141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 msgr2=0x558df2fcd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d43e00640 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fcd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d43e00640 1 -- 172.21.15.28:0/55192141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558df2fcdba0 con 0x558df2fc72e0 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d43e00640 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fcd120 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009bb0 tx=0x7f2d340097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4633a0af36274e31 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d3a000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d3a000640 1 -- 172.21.15.28:0/55192141 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34004040 con 0x558df2fc72e0 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d3a000640 1 -- 172.21.15.28:0/55192141 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f2d340041e0 con 0x558df2fc72e0 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d55e82740 1 -- 172.21.15.28:0/55192141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d3c0bcd30 con 0x558df2fc72e0 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d3a000640 1 -- 172.21.15.28:0/55192141 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2d34004400 con 0x558df2fc72e0 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d55e82740 10 client.25913 fetch_fsmap learned FSMap version 37 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.645+0000 7f2d55e82740 1 -- 172.21.15.28:0/55192141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558df303e0c0 con 0x558df2fc72e0 2024-04-26T01:48:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d3a000640 1 -- 172.21.15.28:0/55192141 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c0bcd30 con 0x558df2fc72e0 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d3a000640 1 -- 172.21.15.28:0/55192141 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 37) ==== 8342+0+0 (secure 0 0 0) 0x7f2d34003890 con 0x558df2fc72e0 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d55e82740 10 client.25913 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d55e82740 10 client.25913 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d55e82740 20 client.25913 populate_metadata read hostname 'smithi028' 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d55e82740 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558df303e6e0 0x558df3040b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d55e82740 4 client.25913 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d55e82740 1 -- 172.21.15.28:0/55192141 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558df2f30320 con 0x558df303e6e0 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.646+0000 7f2d4be00640 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558df303e6e0 0x558df3040b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.647+0000 7f2d4be00640 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558df303e6e0 0x558df3040b00 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=573456a57d791c63 server_cookie=69e270d7fb34db4b in_seq=0 out_seq=0 2024-04-26T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:31.647+0000 7f2d3a000640 10 client.25913 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:31.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:31 smithi028 ceph-mon[28247]: pgmap v233: 97 pgs: 97 active+clean; 3.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 26 MiB/s wr, 1.49k op/s 2024-04-26T01:48:31.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:31 smithi169 ceph-mon[32851]: pgmap v233: 97 pgs: 97 active+clean; 3.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 26 MiB/s wr, 1.49k op/s 2024-04-26T01:48:31.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:31 smithi053 ceph-mon[32593]: pgmap v233: 97 pgs: 97 active+clean; 3.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 26 MiB/s wr, 1.49k op/s 2024-04-26T01:48:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:33 smithi028 ceph-mon[28247]: pgmap v234: 97 pgs: 97 active+clean; 3.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 26 MiB/s wr, 1.45k op/s 2024-04-26T01:48:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:33 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:33.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:33 smithi169 ceph-mon[32851]: pgmap v234: 97 pgs: 97 active+clean; 3.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 26 MiB/s wr, 1.45k op/s 2024-04-26T01:48:33.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:33 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:33.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:33 smithi053 ceph-mon[32593]: pgmap v234: 97 pgs: 97 active+clean; 3.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 26 MiB/s wr, 1.45k op/s 2024-04-26T01:48:33.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:33 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:35.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:35 smithi028 ceph-mon[28247]: pgmap v235: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 26 MiB/s wr, 1.54k op/s 2024-04-26T01:48:35.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:35 smithi169 ceph-mon[32851]: pgmap v235: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 26 MiB/s wr, 1.54k op/s 2024-04-26T01:48:35.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:35 smithi053 ceph-mon[32593]: pgmap v235: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 26 MiB/s wr, 1.54k op/s 2024-04-26T01:48:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 400.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:48:37.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:37 smithi028 ceph-mon[28247]: pgmap v236: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 21 MiB/s wr, 1.40k op/s 2024-04-26T01:48:37.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:37 smithi169 ceph-mon[32851]: pgmap v236: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 21 MiB/s wr, 1.40k op/s 2024-04-26T01:48:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:37 smithi053 ceph-mon[32593]: pgmap v236: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 21 MiB/s wr, 1.40k op/s 2024-04-26T01:48:39.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:39 smithi028 ceph-mon[28247]: pgmap v237: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 17 MiB/s wr, 1.30k op/s 2024-04-26T01:48:39.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:39 smithi169 ceph-mon[32851]: pgmap v237: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 17 MiB/s wr, 1.30k op/s 2024-04-26T01:48:39.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:39 smithi053 ceph-mon[32593]: pgmap v237: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 17 MiB/s wr, 1.30k op/s 2024-04-26T01:48:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 405.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T01:48:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.245+0000 7f2d3a000640 1 -- 172.21.15.28:0/55192141 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558df2f30320 con 0x558df303e6e0 2024-04-26T01:48:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.246+0000 7f2d3a000640 10 client.25913 handle_command_reply: tid=0 2024-04-26T01:48:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.246+0000 7f2d55e82740 1 client.25913 shutdown 2024-04-26T01:48:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.247+0000 7f2d55e82740 1 -- 172.21.15.28:0/55192141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 msgr2=0x558df2fcd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.249+0000 7f2d55e82740 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fcd120 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009bb0 tx=0x7f2d340097f0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.249+0000 7f2d55e82740 1 -- 172.21.15.28:0/55192141 shutdown_connections 2024-04-26T01:48:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.249+0000 7f2d55e82740 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558df303e6e0 0x558df3040b00 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.249+0000 7f2d55e82740 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df2fc7b90 0x558df2fcd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.249+0000 7f2d55e82740 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558df2fc72e0 0x558df2fcd120 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.249+0000 7f2d55e82740 1 --2- 172.21.15.28:0/55192141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558df2fbd2d0 0x558df2fcfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.249+0000 7f2d55e82740 1 -- 172.21.15.28:0/55192141 >> 172.21.15.28:0/55192141 conn(0x558df2fb8ec0 msgr2=0x558df2fc32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.250+0000 7f2d3a000640 0 client.25913 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.250+0000 7f2d55e82740 1 -- 172.21.15.28:0/55192141 shutdown_connections 2024-04-26T01:48:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.250+0000 7f2d55e82740 1 -- 172.21.15.28:0/55192141 wait complete. 2024-04-26T01:48:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.251+0000 7f2d55e82740 20 client.25913 trim_cache size 0 max 16384 2024-04-26T01:48:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:48:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:48:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:48:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:48:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.253+0000 7f2d51a00640 1 -- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2d300a3d90 msgr2=0x7f2d300a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.253+0000 7f2d51a00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2d300a3d90 0x7f2d300a6250 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2d44005fd0 tx=0x7f2d44009500 comp rx=0 tx=0).stop 2024-04-26T01:48:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.253+0000 7f2d51a00640 1 -- 172.21.15.28:0/2013405959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 msgr2=0x7f2d4c19a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.253+0000 7f2d51a00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 0x7f2d4c19a7b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c005ab0 tx=0x7f2d3c003c10 comp rx=0 tx=0).stop 2024-04-26T01:48:41.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.254+0000 7f2d51a00640 1 -- 172.21.15.28:0/2013405959 shutdown_connections 2024-04-26T01:48:41.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.254+0000 7f2d51a00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2d300a3d90 0x7f2d300a6250 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.254+0000 7f2d51a00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c330 0x7f2d4c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.254+0000 7f2d51a00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d4c103830 0x7f2d4c19acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.254+0000 7f2d51a00640 1 --2- 172.21.15.28:0/2013405959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d4c102f10 0x7f2d4c19a7b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:41.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.254+0000 7f2d51a00640 1 -- 172.21.15.28:0/2013405959 >> 172.21.15.28:0/2013405959 conn(0x7f2d4c0fc770 msgr2=0x7f2d4c101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.257+0000 7f2d51a00640 1 -- 172.21.15.28:0/2013405959 shutdown_connections 2024-04-26T01:48:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:41.258+0000 7f2d51a00640 1 -- 172.21.15.28:0/2013405959 wait complete. 2024-04-26T01:48:41.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:41 smithi028 ceph-mon[28247]: pgmap v238: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-26T01:48:41.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:41 smithi169 ceph-mon[32851]: pgmap v238: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-26T01:48:41.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:41 smithi053 ceph-mon[32593]: pgmap v238: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-26T01:48:43.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:43 smithi028 ceph-mon[28247]: pgmap v239: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.1 MiB/s wr, 205 op/s 2024-04-26T01:48:43.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:43 smithi169 ceph-mon[32851]: pgmap v239: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.1 MiB/s wr, 205 op/s 2024-04-26T01:48:43.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:43 smithi053 ceph-mon[32593]: pgmap v239: 97 pgs: 97 active+clean; 3.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 2.1 MiB/s wr, 205 op/s 2024-04-26T01:48:45.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:45 smithi028 ceph-mon[28247]: pgmap v240: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 5.8 MiB/s wr, 619 op/s 2024-04-26T01:48:45.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:45 smithi169 ceph-mon[32851]: pgmap v240: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 5.8 MiB/s wr, 619 op/s 2024-04-26T01:48:45.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:45 smithi053 ceph-mon[32593]: pgmap v240: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 5.8 MiB/s wr, 619 op/s 2024-04-26T01:48:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 410.0 s, 985.5 tps, lat 326.481 ms stddev 2001.362 2024-04-26T01:48:47.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:47 smithi028 ceph-mon[28247]: pgmap v241: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.7 MiB/s wr, 711 op/s 2024-04-26T01:48:47.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:47 smithi169 ceph-mon[32851]: pgmap v241: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.7 MiB/s wr, 711 op/s 2024-04-26T01:48:47.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:47 smithi053 ceph-mon[32593]: pgmap v241: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.7 MiB/s wr, 711 op/s 2024-04-26T01:48:49.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:49 smithi169 ceph-mon[32851]: pgmap v242: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.8 MiB/s wr, 654 op/s 2024-04-26T01:48:49.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:49 smithi053 ceph-mon[32593]: pgmap v242: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.8 MiB/s wr, 654 op/s 2024-04-26T01:48:50.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:49 smithi028 ceph-mon[28247]: pgmap v242: 97 pgs: 97 active+clean; 3.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.8 MiB/s wr, 654 op/s 2024-04-26T01:48:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 415.0 s, 2927.6 tps, lat 34.217 ms stddev 40.900 2024-04-26T01:48:51.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:51 smithi169 ceph-mon[32851]: pgmap v243: 97 pgs: 97 active+clean; 3.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 11 MiB/s wr, 960 op/s 2024-04-26T01:48:51.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:51 smithi053 ceph-mon[32593]: pgmap v243: 97 pgs: 97 active+clean; 3.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 11 MiB/s wr, 960 op/s 2024-04-26T01:48:52.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:51 smithi028 ceph-mon[28247]: pgmap v243: 97 pgs: 97 active+clean; 3.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 11 MiB/s wr, 960 op/s 2024-04-26T01:48:53.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:53 smithi169 ceph-mon[32851]: pgmap v244: 97 pgs: 97 active+clean; 3.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 909 op/s 2024-04-26T01:48:53.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:53 smithi053 ceph-mon[32593]: pgmap v244: 97 pgs: 97 active+clean; 3.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 909 op/s 2024-04-26T01:48:54.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:53 smithi028 ceph-mon[28247]: pgmap v244: 97 pgs: 97 active+clean; 3.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 909 op/s 2024-04-26T01:48:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.462+0000 7f5c0bc00640 1 -- 172.21.15.28:0/3488416022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c04104d90 msgr2=0x7f5c04108dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.462+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/3488416022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c04104d90 0x7f5c04108dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009860 tx=0x7f5bf002a360 comp rx=0 tx=0).stop 2024-04-26T01:48:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.463+0000 7f5c0bc00640 1 -- 172.21.15.28:0/3488416022 shutdown_connections 2024-04-26T01:48:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.463+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/3488416022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c041094b0 0x7f5c04111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.463+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/3488416022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c04104d90 0x7f5c04108dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.463+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/3488416022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c041043e0 0x7f5c041047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.463+0000 7f5c0bc00640 1 -- 172.21.15.28:0/3488416022 >> 172.21.15.28:0/3488416022 conn(0x7f5c040ffc60 msgr2=0x7f5c04102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.463+0000 7f5c0bc00640 1 -- 172.21.15.28:0/3488416022 shutdown_connections 2024-04-26T01:48:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.463+0000 7f5c0bc00640 1 -- 172.21.15.28:0/3488416022 wait complete. 2024-04-26T01:48:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.464+0000 7f5c0bc00640 1 Processor -- start 2024-04-26T01:48:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.465+0000 7f5c0bc00640 1 -- start start 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.465+0000 7f5c0bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c041043e0 0x7f5c041a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.465+0000 7f5c0bc00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c041094b0 0x7f5c041a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.465+0000 7f5c0bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c0419a7e0 0x7f5c0419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.465+0000 7f5c0bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c04114420 con 0x7f5c041043e0 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.465+0000 7f5c0bc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041142a0 con 0x7f5c0419a7e0 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.465+0000 7f5c0bc00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041145a0 con 0x7f5c041094b0 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c09200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c041043e0 0x7f5c041a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c041094b0 0x7f5c041a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c09c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c0419a7e0 0x7f5c0419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c041094b0 0x7f5c041a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:47698/0 (socket says 172.21.15.28:47698) 2024-04-26T01:48:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c09c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c0419a7e0 0x7f5c0419ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38962/0 (socket says 172.21.15.28:38962) 2024-04-26T01:48:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c03e00640 1 -- 172.21.15.28:0/496508605 learned_addr learned my addr 172.21.15.28:0/496508605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c09c00640 1 -- 172.21.15.28:0/496508605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c041094b0 msgr2=0x7f5c041a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c09c00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c041094b0 0x7f5c041a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c09c00640 1 -- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c041043e0 msgr2=0x7f5c041a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c09c00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c041043e0 0x7f5c041a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.466+0000 7f5c09c00640 1 -- 172.21.15.28:0/496508605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0419b370 con 0x7f5c0419a7e0 2024-04-26T01:48:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.467+0000 7f5c09c00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c0419a7e0 0x7f5c0419ac90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc009fd0 tx=0x7f5bfc00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.467+0000 7f5c01600640 1 -- 172.21.15.28:0/496508605 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc00cd60 con 0x7f5c0419a7e0 2024-04-26T01:48:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.468+0000 7f5c0bc00640 1 -- 172.21.15.28:0/496508605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0419b660 con 0x7f5c0419a7e0 2024-04-26T01:48:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.468+0000 7f5c01600640 1 -- 172.21.15.28:0/496508605 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bfc005020 con 0x7f5c0419a7e0 2024-04-26T01:48:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.468+0000 7f5c01600640 1 -- 172.21.15.28:0/496508605 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc00cd60 con 0x7f5c0419a7e0 2024-04-26T01:48:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.468+0000 7f5c0bc00640 1 -- 172.21.15.28:0/496508605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c041a77c0 con 0x7f5c0419a7e0 2024-04-26T01:48:54.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.473+0000 7f5c0e9ea740 1 Processor -- start 2024-04-26T01:48:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.474+0000 7f5c01600640 1 -- 172.21.15.28:0/496508605 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5bfc010430 con 0x7f5c0419a7e0 2024-04-26T01:48:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c0e9ea740 1 -- start start 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c030 0x56345c35c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c950 0x56345c366cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 0x56345c369600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56345c2b9900 con 0x56345c35c030 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56345c25fef0 con 0x56345c35c950 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x56345c2b58c0 con 0x56345c367210 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c09200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c950 0x56345c366cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c09200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c950 0x56345c366cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38978/0 (socket says 172.21.15.28:38978) 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c09200640 1 -- 172.21.15.28:0/2340342557 learned_addr learned my addr 172.21.15.28:0/2340342557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c03e00640 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c030 0x56345c35c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c09200640 1 -- 172.21.15.28:0/2340342557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 msgr2=0x56345c369600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:48:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.475+0000 7f5c09200640 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 0x56345c369600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.476+0000 7f5c09200640 1 -- 172.21.15.28:0/2340342557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c030 msgr2=0x56345c35c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.476+0000 7f5c09200640 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c030 0x56345c35c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.476+0000 7f5c09200640 1 -- 172.21.15.28:0/2340342557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56345bfcc210 con 0x56345c35c950 2024-04-26T01:48:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.476+0000 7f5c03e00640 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c030 0x56345c35c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:48:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.477+0000 7f5c01600640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5bec0a4090 0x7f5bec0a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.477+0000 7f5c03e00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5bec0a4090 0x7f5bec0a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.478+0000 7f5c01600640 1 -- 172.21.15.28:0/496508605 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f5bfc0d0a40 con 0x7f5c0419a7e0 2024-04-26T01:48:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.478+0000 7f5c09200640 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c950 0x56345c366cd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009cd0 tx=0x7f5bf8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0ede92de31bba91 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.479+0000 7f5bf6000640 1 -- 172.21.15.28:0/2340342557 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf801e070 con 0x56345c35c950 2024-04-26T01:48:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.479+0000 7f5bf6000640 1 -- 172.21.15.28:0/2340342557 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf8019ae0 con 0x56345c35c950 2024-04-26T01:48:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.479+0000 7f5bf6000640 1 -- 172.21.15.28:0/2340342557 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf800ead0 con 0x56345c35c950 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.479+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/2340342557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c950 msgr2=0x56345c366cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.479+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c950 0x56345c366cd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009cd0 tx=0x7f5bf8011810 comp rx=0 tx=0).stop 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c03e00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5bec0a4090 0x7f5bec0a6550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5c0419bae0 tx=0x7f5bf002a860 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/2340342557 shutdown_connections 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 0x56345c369600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c950 0x56345c366cd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/2340342557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c030 0x56345c35c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/2340342557 >> 172.21.15.28:0/2340342557 conn(0x56345c358540 msgr2=0x56345c358920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/2340342557 shutdown_connections 2024-04-26T01:48:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/2340342557 wait complete. 2024-04-26T01:48:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.480+0000 7f5c0e9ea740 1 -- start start 2024-04-26T01:48:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.481+0000 7f5c0e9ea740 1 Processor -- start 2024-04-26T01:48:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.481+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c364200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.482+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c35c950 0x56345c364740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.482+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c367210 0x56345c364c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.482+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56345c369e40 con 0x56345c367210 2024-04-26T01:48:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.482+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56345c273310 con 0x56345c35c030 2024-04-26T01:48:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.482+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x56345c369fc0 con 0x56345c35c950 2024-04-26T01:48:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.482+0000 7f5c0e9ea740 10 client.0 _ll_register_callbacks cb 0x56345c241cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.482+0000 7f5c09200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c364200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.482+0000 7f5c09c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c367210 0x56345c364c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c35c950 0x56345c364740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c35c950 0x56345c364740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:47700/0 (socket says 172.21.15.28:47700) 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c09200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c364200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38994/0 (socket says 172.21.15.28:38994) 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c03e00640 1 -- 172.21.15.28:0/681873894 learned_addr learned my addr 172.21.15.28:0/681873894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c03e00640 1 -- 172.21.15.28:0/681873894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 msgr2=0x56345c364200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c03e00640 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c364200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c03e00640 1 -- 172.21.15.28:0/681873894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c367210 msgr2=0x56345c364c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c03e00640 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c367210 0x56345c364c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c03e00640 1 -- 172.21.15.28:0/681873894 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56345c098620 con 0x56345c35c950 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c09200640 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c364200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.483+0000 7f5c09c00640 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c367210 0x56345c364c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.484+0000 7f5c03e00640 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c35c950 0x56345c364740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0035040 tx=0x7f5bf002a360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b88e18bf8fbff05a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.484+0000 7f5beaa00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:48:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.484+0000 7f5beaa00640 1 -- 172.21.15.28:0/681873894 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0037040 con 0x56345c35c950 2024-04-26T01:48:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.484+0000 7f5beaa00640 1 -- 172.21.15.28:0/681873894 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f5bf00035f0 con 0x56345c35c950 2024-04-26T01:48:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.484+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/681873894 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf8019ae0 con 0x56345c35c950 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.485+0000 7f5beaa00640 1 -- 172.21.15.28:0/681873894 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5bf00037f0 con 0x56345c35c950 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.486+0000 7f5c0e9ea740 10 client.25942 fetch_fsmap learned FSMap version 38 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.486+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/681873894 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56345c387080 con 0x56345c35c950 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.486+0000 7f5beaa00640 1 -- 172.21.15.28:0/681873894 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf8019ae0 con 0x56345c35c950 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.486+0000 7f5beaa00640 1 -- 172.21.15.28:0/681873894 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f5bf00329f0 con 0x56345c35c950 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.486+0000 7f5c0e9ea740 10 client.25942 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.486+0000 7f5c0e9ea740 10 client.25942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.487+0000 7f5c0e9ea740 20 client.25942 populate_metadata read hostname 'smithi028' 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.487+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56345c388a90 0x56345c38aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.487+0000 7f5c0e9ea740 4 client.25942 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.487+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/681873894 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56345c2cd8b0 con 0x56345c388a90 2024-04-26T01:48:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.487+0000 7f5c09200640 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56345c388a90 0x56345c38aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.488+0000 7f5c09200640 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56345c388a90 0x56345c38aed0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74e0a5b73dbe8dad server_cookie=10f7bc93ad1c5031 in_seq=0 out_seq=0 2024-04-26T01:48:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.488+0000 7f5beaa00640 10 client.25942 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5beaa00640 1 -- 172.21.15.28:0/681873894 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56345c2cd8b0 con 0x56345c388a90 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5beaa00640 10 client.25942 handle_command_reply: tid=0 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 client.25942 shutdown 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/681873894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c35c950 msgr2=0x56345c364740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c35c950 0x56345c364740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0035040 tx=0x7f5bf002a360 comp rx=0 tx=0).stop 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/681873894 shutdown_connections 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56345c388a90 0x56345c38aed0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c367210 0x56345c364c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c35c950 0x56345c364740 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/681873894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c364200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5beaa00640 0 client.25942 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.489+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/681873894 >> 172.21.15.28:0/681873894 conn(0x56345c358540 msgr2=0x56345c35e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.490+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/681873894 shutdown_connections 2024-04-26T01:48:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.490+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/681873894 wait complete. 2024-04-26T01:48:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.490+0000 7f5c0e9ea740 20 client.25942 trim_cache size 0 max 16384 2024-04-26T01:48:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.521+0000 7f5c0e9ea740 1 Processor -- start 2024-04-26T01:48:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c0e9ea740 1 -- start start 2024-04-26T01:48:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c372d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 0x56345c3907e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 0x56345c390d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c04072220 con 0x56345c35c950 2024-04-26T01:48:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0406faf0 con 0x56345c35c030 2024-04-26T01:48:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0406d960 con 0x56345c367210 2024-04-26T01:48:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c09200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c372d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c09200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c372d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38998/0 (socket says 172.21.15.28:38998) 2024-04-26T01:48:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c09200640 1 -- 172.21.15.28:0/1227939816 learned_addr learned my addr 172.21.15.28:0/1227939816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.522+0000 7f5c03e00640 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 0x56345c3907e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.523+0000 7f5c09c00640 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 0x56345c390d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.523+0000 7f5c09200640 1 -- 172.21.15.28:0/1227939816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 msgr2=0x56345c390d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.523+0000 7f5c09200640 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 0x56345c390d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.523+0000 7f5c09200640 1 -- 172.21.15.28:0/1227939816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 msgr2=0x56345c3907e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.523+0000 7f5c09200640 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 0x56345c3907e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.523+0000 7f5c09200640 1 -- 172.21.15.28:0/1227939816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56345c369970 con 0x56345c35c030 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.523+0000 7f5c09200640 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c372d00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8006010 tx=0x7f5bf8009890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=709543da6ce02863 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.523+0000 7f5bf5600640 1 -- 172.21.15.28:0/1227939816 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8029e60 con 0x56345c35c030 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.524+0000 7f5bf5600640 1 -- 172.21.15.28:0/1227939816 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf802d020 con 0x56345c35c030 2024-04-26T01:48:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.524+0000 7f5bf5600640 1 -- 172.21.15.28:0/1227939816 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf801e040 con 0x56345c35c030 2024-04-26T01:48:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.524+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/1227939816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 msgr2=0x56345c372d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.524+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c372d00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8006010 tx=0x7f5bf8009890 comp rx=0 tx=0).stop 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/1227939816 shutdown_connections 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c367210 0x56345c390d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 0x56345c3907e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/1227939816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c35c030 0x56345c372d00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/1227939816 >> 172.21.15.28:0/1227939816 conn(0x56345c358540 msgr2=0x56345c35f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/1227939816 shutdown_connections 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/1227939816 wait complete. 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 -- start start 2024-04-26T01:48:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.525+0000 7f5c0e9ea740 1 Processor -- start 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 0x56345c36c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c367210 0x56345c36c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c0e9ea740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c3e7020 0x56345c3e74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0406dfb0 con 0x56345c35c950 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0406f420 con 0x56345c367210 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c0e9ea740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0406e680 con 0x56345c3e7020 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c0e9ea740 10 client.0 _ll_register_callbacks cb 0x7f5bf00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c367210 0x56345c36c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c367210 0x56345c36c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39002/0 (socket says 172.21.15.28:39002) 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c03e00640 1 -- 172.21.15.28:0/81548236 learned_addr learned my addr 172.21.15.28:0/81548236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:48:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.526+0000 7f5c09200640 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 0x56345c36c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.527+0000 7f5c09c00640 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c3e7020 0x56345c3e74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.527+0000 7f5c03e00640 1 -- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c3e7020 msgr2=0x56345c3e74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.527+0000 7f5c03e00640 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c3e7020 0x56345c3e74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.527+0000 7f5c03e00640 1 -- 172.21.15.28:0/81548236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 msgr2=0x56345c36c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.527+0000 7f5c03e00640 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 0x56345c36c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.527+0000 7f5c03e00640 1 -- 172.21.15.28:0/81548236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56345c3e7a10 con 0x56345c367210 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.527+0000 7f5c09c00640 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c3e7020 0x56345c3e74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.528+0000 7f5c03e00640 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c367210 0x56345c36c950 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0032e60 tx=0x7f5bf0005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f17cb3e09e3d83e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.528+0000 7f5beaa00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.528+0000 7f5beaa00640 1 -- 172.21.15.28:0/81548236 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0037040 con 0x56345c367210 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.528+0000 7f5beaa00640 1 -- 172.21.15.28:0/81548236 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f5bf0003ba0 con 0x56345c367210 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.528+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/81548236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf802d020 con 0x56345c367210 2024-04-26T01:48:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.528+0000 7f5beaa00640 1 -- 172.21.15.28:0/81548236 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5bf0003da0 con 0x56345c367210 2024-04-26T01:48:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.528+0000 7f5c0e9ea740 10 client.25931 fetch_fsmap learned FSMap version 38 2024-04-26T01:48:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.528+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/81548236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56345c3eed80 con 0x56345c367210 2024-04-26T01:48:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.529+0000 7f5beaa00640 1 -- 172.21.15.28:0/81548236 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf802d020 con 0x56345c367210 2024-04-26T01:48:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.529+0000 7f5beaa00640 1 -- 172.21.15.28:0/81548236 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f5bf005b070 con 0x56345c367210 2024-04-26T01:48:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.529+0000 7f5c0e9ea740 10 client.25931 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:48:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.529+0000 7f5c0e9ea740 10 client.25931 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.529+0000 7f5c0e9ea740 20 client.25931 populate_metadata read hostname 'smithi028' 2024-04-26T01:48:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.529+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56345c3ef560 0x56345c3f1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.529+0000 7f5c0e9ea740 4 client.25931 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.529+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/81548236 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56345c2c7320 con 0x56345c3ef560 2024-04-26T01:48:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.530+0000 7f5c09200640 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56345c3ef560 0x56345c3f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.530+0000 7f5c09200640 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56345c3ef560 0x56345c3f1980 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c3be4ec6f75e289 server_cookie=26f9c27f12b9e0fd in_seq=0 out_seq=0 2024-04-26T01:48:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.530+0000 7f5beaa00640 10 client.25931 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.568+0000 7f5beaa00640 1 -- 172.21.15.28:0/81548236 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56345c2c7320 con 0x56345c3ef560 2024-04-26T01:48:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.568+0000 7f5beaa00640 10 client.25931 handle_command_reply: tid=0 2024-04-26T01:48:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.568+0000 7f5c0e9ea740 1 client.25931 shutdown 2024-04-26T01:48:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/81548236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c367210 msgr2=0x56345c36c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c367210 0x56345c36c950 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0032e60 tx=0x7f5bf0005d70 comp rx=0 tx=0).stop 2024-04-26T01:48:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/81548236 shutdown_connections 2024-04-26T01:48:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56345c3ef560 0x56345c3f1980 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56345c3e7020 0x56345c3e74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56345c367210 0x56345c36c950 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 --2- 172.21.15.28:0/81548236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56345c35c950 0x56345c36c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/81548236 >> 172.21.15.28:0/81548236 conn(0x56345c358540 msgr2=0x56345c34e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5beaa00640 0 client.25931 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:48:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/81548236 shutdown_connections 2024-04-26T01:48:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.569+0000 7f5c0e9ea740 1 -- 172.21.15.28:0/81548236 wait complete. 2024-04-26T01:48:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0e9ea740 20 client.25931 trim_cache size 0 max 16384 2024-04-26T01:48:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:48:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:48:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:48:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:48:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 -- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5bec0a4090 msgr2=0x7f5bec0a6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5bec0a4090 0x7f5bec0a6550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5c0419bae0 tx=0x7f5bf002a860 comp rx=0 tx=0).stop 2024-04-26T01:48:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 -- 172.21.15.28:0/496508605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c0419a7e0 msgr2=0x7f5c0419ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c0419a7e0 0x7f5c0419ac90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc009fd0 tx=0x7f5bfc00ee50 comp rx=0 tx=0).stop 2024-04-26T01:48:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 -- 172.21.15.28:0/496508605 shutdown_connections 2024-04-26T01:48:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5bec0a4090 0x7f5bec0a6550 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c0419a7e0 0x7f5c0419ac90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c041094b0 0x7f5c041a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 --2- 172.21.15.28:0/496508605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c041043e0 0x7f5c041a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.570+0000 7f5c0bc00640 1 -- 172.21.15.28:0/496508605 >> 172.21.15.28:0/496508605 conn(0x7f5c040ffc60 msgr2=0x7f5c04101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.571+0000 7f5c0bc00640 1 -- 172.21.15.28:0/496508605 shutdown_connections 2024-04-26T01:48:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:48:54.571+0000 7f5c0bc00640 1 -- 172.21.15.28:0/496508605 wait complete. 2024-04-26T01:48:55.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:55 smithi169 ceph-mon[32851]: pgmap v245: 97 pgs: 97 active+clean; 3.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-04-26T01:48:55.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:55 smithi053 ceph-mon[32593]: pgmap v245: 97 pgs: 97 active+clean; 3.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-04-26T01:48:56.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:55 smithi028 ceph-mon[28247]: pgmap v245: 97 pgs: 97 active+clean; 3.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-04-26T01:48:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 420.0 s, 2380.0 tps, lat 42.002 ms stddev 48.832 2024-04-26T01:48:57.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:57 smithi169 ceph-mon[32851]: pgmap v246: 97 pgs: 97 active+clean; 3.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 876 op/s 2024-04-26T01:48:57.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:57 smithi053 ceph-mon[32593]: pgmap v246: 97 pgs: 97 active+clean; 3.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 876 op/s 2024-04-26T01:48:58.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:57 smithi028 ceph-mon[28247]: pgmap v246: 97 pgs: 97 active+clean; 3.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 876 op/s 2024-04-26T01:48:59.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:48:59 smithi169 ceph-mon[32851]: pgmap v247: 97 pgs: 97 active+clean; 3.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 9.9 MiB/s wr, 686 op/s 2024-04-26T01:48:59.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:48:59 smithi053 ceph-mon[32593]: pgmap v247: 97 pgs: 97 active+clean; 3.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 9.9 MiB/s wr, 686 op/s 2024-04-26T01:49:00.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:48:59 smithi028 ceph-mon[28247]: pgmap v247: 97 pgs: 97 active+clean; 3.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 9.9 MiB/s wr, 686 op/s 2024-04-26T01:49:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 425.0 s, 2773.6 tps, lat 35.410 ms stddev 42.101 2024-04-26T01:49:01.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:01 smithi169 ceph-mon[32851]: pgmap v248: 97 pgs: 97 active+clean; 3.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 14 MiB/s wr, 984 op/s 2024-04-26T01:49:01.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:01 smithi053 ceph-mon[32593]: pgmap v248: 97 pgs: 97 active+clean; 3.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 14 MiB/s wr, 984 op/s 2024-04-26T01:49:02.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:01 smithi028 ceph-mon[28247]: pgmap v248: 97 pgs: 97 active+clean; 3.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 14 MiB/s wr, 984 op/s 2024-04-26T01:49:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:03 smithi169 ceph-mon[32851]: pgmap v249: 97 pgs: 97 active+clean; 3.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 678 op/s 2024-04-26T01:49:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:03 smithi169 ceph-mon[32851]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:49:03.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:03 smithi053 ceph-mon[32593]: pgmap v249: 97 pgs: 97 active+clean; 3.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 678 op/s 2024-04-26T01:49:03.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:03 smithi053 ceph-mon[32593]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:49:04.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:03 smithi028 ceph-mon[28247]: pgmap v249: 97 pgs: 97 active+clean; 3.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 678 op/s 2024-04-26T01:49:04.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:03 smithi028 ceph-mon[28247]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:49:05.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:05 smithi169 ceph-mon[32851]: pgmap v251: 97 pgs: 97 active+clean; 3.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 997 op/s 2024-04-26T01:49:05.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:05 smithi053 ceph-mon[32593]: pgmap v251: 97 pgs: 97 active+clean; 3.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 997 op/s 2024-04-26T01:49:06.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:05 smithi028 ceph-mon[28247]: pgmap v251: 97 pgs: 97 active+clean; 3.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 997 op/s 2024-04-26T01:49:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 430.0 s, 1044.6 tps, lat 97.953 ms stddev 187.592 2024-04-26T01:49:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.772+0000 7f8f2ac00640 1 -- 172.21.15.28:0/31552196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f24076480 msgr2=0x7f8f24076860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.772+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/31552196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f24076480 0x7f8f24076860 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009930 tx=0x7f8f1402a420 comp rx=0 tx=0).stop 2024-04-26T01:49:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.773+0000 7f8f2ac00640 1 -- 172.21.15.28:0/31552196 shutdown_connections 2024-04-26T01:49:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.773+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/31552196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24112de0 0x7f8f241151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.773+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/31552196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24076da0 0x7f8f241128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.773+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/31552196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f24076480 0x7f8f24076860 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.773+0000 7f8f2ac00640 1 -- 172.21.15.28:0/31552196 >> 172.21.15.28:0/31552196 conn(0x7f8f240ffc50 msgr2=0x7f8f24102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.773+0000 7f8f2ac00640 1 -- 172.21.15.28:0/31552196 shutdown_connections 2024-04-26T01:49:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.773+0000 7f8f2ac00640 1 -- 172.21.15.28:0/31552196 wait complete. 2024-04-26T01:49:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.774+0000 7f8f2ac00640 1 Processor -- start 2024-04-26T01:49:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.774+0000 7f8f2ac00640 1 -- start start 2024-04-26T01:49:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.774+0000 7f8f2ac00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f24076480 0x7f8f241a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.774+0000 7f8f2ac00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24076da0 0x7f8f241a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.774+0000 7f8f2ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24112de0 0x7f8f2419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.774+0000 7f8f2ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24118840 con 0x7f8f24112de0 2024-04-26T01:49:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.774+0000 7f8f2ac00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8f241186c0 con 0x7f8f24076da0 2024-04-26T01:49:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.774+0000 7f8f2ac00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8f241189c0 con 0x7f8f24076480 2024-04-26T01:49:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.775+0000 7f8f28c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24112de0 0x7f8f2419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.775+0000 7f8f28c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24112de0 0x7f8f2419eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42246/0 (socket says 172.21.15.28:42246) 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.775+0000 7f8f28c00640 1 -- 172.21.15.28:0/4089881810 learned_addr learned my addr 172.21.15.28:0/4089881810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.775+0000 7f8f23400640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24076da0 0x7f8f241a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.775+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f24076480 0x7f8f241a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.776+0000 7f8f23400640 1 -- 172.21.15.28:0/4089881810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f24076480 msgr2=0x7f8f241a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.776+0000 7f8f23400640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f24076480 0x7f8f241a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.776+0000 7f8f23400640 1 -- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24112de0 msgr2=0x7f8f2419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.776+0000 7f8f23400640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24112de0 0x7f8f2419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.776+0000 7f8f23400640 1 -- 172.21.15.28:0/4089881810 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2419f420 con 0x7f8f24076da0 2024-04-26T01:49:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.776+0000 7f8f28c00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24112de0 0x7f8f2419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:07.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.778+0000 7f8f23400640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24076da0 0x7f8f241a4f90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8f10005fd0 tx=0x7f8f1000dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:07.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.778+0000 7f8f20c00640 1 -- 172.21.15.28:0/4089881810 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10009c30 con 0x7f8f24076da0 2024-04-26T01:49:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.778+0000 7f8f20c00640 1 -- 172.21.15.28:0/4089881810 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f10009dd0 con 0x7f8f24076da0 2024-04-26T01:49:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.778+0000 7f8f20c00640 1 -- 172.21.15.28:0/4089881810 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10010460 con 0x7f8f24076da0 2024-04-26T01:49:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.778+0000 7f8f2ac00640 1 -- 172.21.15.28:0/4089881810 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2419f710 con 0x7f8f24076da0 2024-04-26T01:49:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.778+0000 7f8f2ac00640 1 -- 172.21.15.28:0/4089881810 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f241ab9d0 con 0x7f8f24076da0 2024-04-26T01:49:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.782+0000 7f8f2d97b740 1 Processor -- start 2024-04-26T01:49:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.782+0000 7f8f2d97b740 1 -- start start 2024-04-26T01:49:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.783+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f112820 0x557d7f112c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.783+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.783+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f11da00 0x557d7f11fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.783+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d7f078900 con 0x557d7f113140 2024-04-26T01:49:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.783+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557d7f01eef0 con 0x557d7f112820 2024-04-26T01:49:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.783+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557d7f0748c0 con 0x557d7f11da00 2024-04-26T01:49:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.783+0000 7f8f28c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f112820 0x557d7f112c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.784+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.784+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42256/0 (socket says 172.21.15.28:42256) 2024-04-26T01:49:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.784+0000 7f8f23e00640 1 -- 172.21.15.28:0/2121589699 learned_addr learned my addr 172.21.15.28:0/2121589699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.784+0000 7f8f20c00640 1 -- 172.21.15.28:0/4089881810 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8f10003e80 con 0x7f8f24076da0 2024-04-26T01:49:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f23400640 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f11da00 0x557d7f11fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f23e00640 1 -- 172.21.15.28:0/2121589699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f11da00 msgr2=0x557d7f11fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f23e00640 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f11da00 0x557d7f11fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f23e00640 1 -- 172.21.15.28:0/2121589699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f112820 msgr2=0x557d7f112c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f23e00640 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f112820 0x557d7f112c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f23e00640 1 -- 172.21.15.28:0/2121589699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d7ed8b210 con 0x557d7f113140 2024-04-26T01:49:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f23400640 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f11da00 0x557d7f11fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:49:07.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f28c00640 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f112820 0x557d7f112c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:07.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.785+0000 7f8f20c00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8f080a3e40 0x7f8f080a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.786+0000 7f8f23e00640 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11d4c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8f14006010 tx=0x7f8f14004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3bd871730ffb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.786+0000 7f8f20c00640 1 -- 172.21.15.28:0/4089881810 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8557+0+0 (secure 0 0 0) 0x7f8f100d0770 con 0x7f8f24076da0 2024-04-26T01:49:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.786+0000 7f8f28c00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8f080a3e40 0x7f8f080a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.786+0000 7f8f19200640 1 -- 172.21.15.28:0/2121589699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14035040 con 0x557d7f113140 2024-04-26T01:49:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.786+0000 7f8f19200640 1 -- 172.21.15.28:0/2121589699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f140044b0 con 0x557d7f113140 2024-04-26T01:49:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.786+0000 7f8f19200640 1 -- 172.21.15.28:0/2121589699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14003850 con 0x557d7f113140 2024-04-26T01:49:07.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.787+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2121589699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 msgr2=0x557d7f11d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.787+0000 7f8f28c00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8f080a3e40 0x7f8f080a6300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009fd0 tx=0x7f8f1c0097e0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.787+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11d4c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8f14006010 tx=0x7f8f14004200 comp rx=0 tx=0).stop 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.787+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2121589699 shutdown_connections 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.787+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f11da00 0x557d7f11fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.787+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11d4c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.787+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2121589699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f112820 0x557d7f112c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.788+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2121589699 >> 172.21.15.28:0/2121589699 conn(0x557d7f10ed30 msgr2=0x557d7f10f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.788+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2121589699 shutdown_connections 2024-04-26T01:49:07.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.788+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2121589699 wait complete. 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.788+0000 7f8f2d97b740 1 -- start start 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.788+0000 7f8f2d97b740 1 Processor -- start 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f11a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f113140 0x557d7f11aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f11da00 0x557d7f11b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d7f1205d0 con 0x557d7f11da00 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557d7f032310 con 0x557d7f113140 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557d7f120750 con 0x557d7f112820 2024-04-26T01:49:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f2d97b740 10 client.0 _ll_register_callbacks cb 0x557d7f000cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f11a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.789+0000 7f8f23400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f113140 0x557d7f11aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f11a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46720/0 (socket says 172.21.15.28:46720) 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23e00640 1 -- 172.21.15.28:0/1743203955 learned_addr learned my addr 172.21.15.28:0/1743203955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f28c00640 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f11da00 0x557d7f11b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23400640 1 -- 172.21.15.28:0/1743203955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 msgr2=0x557d7f11a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23400640 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f11a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23400640 1 -- 172.21.15.28:0/1743203955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f11da00 msgr2=0x557d7f11b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23400640 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f11da00 0x557d7f11b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23400640 1 -- 172.21.15.28:0/1743203955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d7ee57620 con 0x557d7f113140 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f11a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f28c00640 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f11da00 0x557d7f11b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:07.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f23400640 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f113140 0x557d7f11aee0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8f100cde40 tx=0x7f8f1009c0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5b12c0d6eefe36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.790+0000 7f8f0d600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:49:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f0d600640 1 -- 172.21.15.28:0/1743203955 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10020050 con 0x557d7f113140 2024-04-26T01:49:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 1 -- 172.21.15.28:0/1743203955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f140044b0 con 0x557d7f113140 2024-04-26T01:49:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f0d600640 1 -- 172.21.15.28:0/1743203955 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8557+0+0 (secure 0 0 0) 0x7f8f10085bd0 con 0x557d7f113140 2024-04-26T01:49:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f0d600640 1 -- 172.21.15.28:0/1743203955 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8f100ce2c0 con 0x557d7f113140 2024-04-26T01:49:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 10 client.25943 fetch_fsmap learned FSMap version 38 2024-04-26T01:49:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 1 -- 172.21.15.28:0/1743203955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x557d7f13d8c0 con 0x557d7f113140 2024-04-26T01:49:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f0d600640 1 -- 172.21.15.28:0/1743203955 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f140044b0 con 0x557d7f113140 2024-04-26T01:49:07.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f0d600640 1 -- 172.21.15.28:0/1743203955 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f8f100a4e70 con 0x557d7f113140 2024-04-26T01:49:07.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 10 client.25943 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:49:07.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 10 client.25943 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:07.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 20 client.25943 populate_metadata read hostname 'smithi028' 2024-04-26T01:49:07.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557d7f13eda0 0x557d7f1411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 4 client.25943 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:07.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.791+0000 7f8f2d97b740 1 -- 172.21.15.28:0/1743203955 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d7f08c8b0 con 0x557d7f13eda0 2024-04-26T01:49:07.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.792+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557d7f13eda0 0x557d7f1411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.795+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557d7f13eda0 0x557d7f1411b0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cbcba9ce20dae1b server_cookie=bf571e106393f3ba in_seq=0 out_seq=0 2024-04-26T01:49:07.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.795+0000 7f8f0d600640 10 client.25943 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:07.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.796+0000 7f8f0d600640 1 -- 172.21.15.28:0/1743203955 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d7f08c8b0 con 0x557d7f13eda0 2024-04-26T01:49:07.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.796+0000 7f8f0d600640 10 client.25943 handle_command_reply: tid=0 2024-04-26T01:49:07.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.796+0000 7f8f2d97b740 1 client.25943 shutdown 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.796+0000 7f8f2d97b740 1 -- 172.21.15.28:0/1743203955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f113140 msgr2=0x557d7f11aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.796+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f113140 0x557d7f11aee0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8f100cde40 tx=0x7f8f1009c0b0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.797+0000 7f8f2d97b740 1 -- 172.21.15.28:0/1743203955 shutdown_connections 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.797+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557d7f13eda0 0x557d7f1411b0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.797+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f11da00 0x557d7f11b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.797+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f113140 0x557d7f11aee0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.797+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/1743203955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f11a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.797+0000 7f8f2d97b740 1 -- 172.21.15.28:0/1743203955 >> 172.21.15.28:0/1743203955 conn(0x557d7f10ed30 msgr2=0x557d7f114ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.797+0000 7f8f0d600640 0 client.25943 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.798+0000 7f8f2d97b740 1 -- 172.21.15.28:0/1743203955 shutdown_connections 2024-04-26T01:49:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.798+0000 7f8f2d97b740 1 -- 172.21.15.28:0/1743203955 wait complete. 2024-04-26T01:49:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.798+0000 7f8f2d97b740 20 client.25943 trim_cache size 0 max 16384 2024-04-26T01:49:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.822+0000 7f8f2d97b740 1 Processor -- start 2024-04-26T01:49:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.822+0000 7f8f2d97b740 1 -- start start 2024-04-26T01:49:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f13a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f146b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f23400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f23400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42274/0 (socket says 172.21.15.28:42274) 2024-04-26T01:49:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f13a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f13a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46722/0 (socket says 172.21.15.28:46722) 2024-04-26T01:49:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f23400640 1 -- 172.21.15.28:0/3346253178 learned_addr learned my addr 172.21.15.28:0/3346253178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f2d97b740 1 -- 172.21.15.28:0/3346253178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f241ae320 con 0x557d7f113140 2024-04-26T01:49:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f2d97b740 1 -- 172.21.15.28:0/3346253178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8f241abbf0 con 0x557d7f11da00 2024-04-26T01:49:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.823+0000 7f8f2d97b740 1 -- 172.21.15.28:0/3346253178 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24072030 con 0x557d7f112820 2024-04-26T01:49:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f28c00640 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f23400640 1 -- 172.21.15.28:0/3346253178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 msgr2=0x557d7f13a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f23400640 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f13a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f23400640 1 -- 172.21.15.28:0/3346253178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 msgr2=0x557d7f146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f23400640 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f23400640 1 -- 172.21.15.28:0/3346253178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d7f119950 con 0x557d7f113140 2024-04-26T01:49:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f23e00640 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f13a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f28c00640 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.824+0000 7f8f23400640 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11b240 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009610 tx=0x7f8f10005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cca90c85aa5c8d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.825+0000 7f8f0fe00640 1 -- 172.21.15.28:0/3346253178 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10014030 con 0x557d7f113140 2024-04-26T01:49:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.825+0000 7f8f0fe00640 1 -- 172.21.15.28:0/3346253178 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f100a5300 con 0x557d7f113140 2024-04-26T01:49:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.825+0000 7f8f0fe00640 1 -- 172.21.15.28:0/3346253178 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f100d0e60 con 0x557d7f113140 2024-04-26T01:49:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.826+0000 7f8f2d97b740 1 -- 172.21.15.28:0/3346253178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 msgr2=0x557d7f11b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.826+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11b240 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009610 tx=0x7f8f10005bb0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.826+0000 7f8f2d97b740 1 -- 172.21.15.28:0/3346253178 shutdown_connections 2024-04-26T01:49:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.826+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.826+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f113140 0x557d7f11b240 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.826+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/3346253178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f112820 0x557d7f13a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.826+0000 7f8f2d97b740 1 -- 172.21.15.28:0/3346253178 >> 172.21.15.28:0/3346253178 conn(0x557d7f10ed30 msgr2=0x557d7f10f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.827+0000 7f8f2d97b740 1 -- 172.21.15.28:0/3346253178 shutdown_connections 2024-04-26T01:49:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.827+0000 7f8f2d97b740 1 -- 172.21.15.28:0/3346253178 wait complete. 2024-04-26T01:49:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.827+0000 7f8f2d97b740 1 -- start start 2024-04-26T01:49:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.828+0000 7f8f2d97b740 1 Processor -- start 2024-04-26T01:49:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.828+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f112820 0x557d7f133a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.828+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f113140 0x557d7f133f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.828+0000 7f8f2d97b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f1a3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.828+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24072680 con 0x557d7f112820 2024-04-26T01:49:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.828+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24073af0 con 0x557d7f11da00 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.828+0000 7f8f2d97b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24072d50 con 0x557d7f113140 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f2d97b740 10 client.0 _ll_register_callbacks cb 0x7f8f1009c530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f112820 0x557d7f133a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f112820 0x557d7f133a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42286/0 (socket says 172.21.15.28:42286) 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f23e00640 1 -- 172.21.15.28:0/2475599085 learned_addr learned my addr 172.21.15.28:0/2475599085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f28c00640 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f1a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f23400640 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f113140 0x557d7f133f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f28c00640 1 -- 172.21.15.28:0/2475599085 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f113140 msgr2=0x557d7f133f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f28c00640 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f113140 0x557d7f133f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f28c00640 1 -- 172.21.15.28:0/2475599085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f112820 msgr2=0x557d7f133a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f28c00640 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f112820 0x557d7f133a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f28c00640 1 -- 172.21.15.28:0/2475599085 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d7f1a38c0 con 0x557d7f11da00 2024-04-26T01:49:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.829+0000 7f8f23e00640 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f112820 0x557d7f133a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.830+0000 7f8f28c00640 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f1a3380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c011e30 tx=0x7f8f1c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f542da159aefd86d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.830+0000 7f8f0d600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:49:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.830+0000 7f8f0d600640 1 -- 172.21.15.28:0/2475599085 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c02f060 con 0x557d7f11da00 2024-04-26T01:49:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.830+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2475599085 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f100a5300 con 0x557d7f11da00 2024-04-26T01:49:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.830+0000 7f8f0d600640 1 -- 172.21.15.28:0/2475599085 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8557+0+0 (secure 0 0 0) 0x7f8f1c0219b0 con 0x557d7f11da00 2024-04-26T01:49:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.830+0000 7f8f0d600640 1 -- 172.21.15.28:0/2475599085 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8f1c021bb0 con 0x557d7f11da00 2024-04-26T01:49:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.830+0000 7f8f2d97b740 10 client.25949 fetch_fsmap learned FSMap version 38 2024-04-26T01:49:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.830+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2475599085 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x557d7f1a4b40 con 0x557d7f11da00 2024-04-26T01:49:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f0d600640 1 -- 172.21.15.28:0/2475599085 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f100a5300 con 0x557d7f11da00 2024-04-26T01:49:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f0d600640 1 -- 172.21.15.28:0/2475599085 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f8f1c00f040 con 0x557d7f11da00 2024-04-26T01:49:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f2d97b740 10 client.25949 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:49:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f2d97b740 10 client.25949 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f2d97b740 20 client.25949 populate_metadata read hostname 'smithi028' 2024-04-26T01:49:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557d7f1a5330 0x557d7f1a7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f2d97b740 4 client.25949 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2475599085 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557d7f086320 con 0x557d7f1a5330 2024-04-26T01:49:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.831+0000 7f8f23e00640 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557d7f1a5330 0x557d7f1a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.832+0000 7f8f0d600640 10 client.25949 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.832+0000 7f8f23e00640 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557d7f1a5330 0x557d7f1a7750 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb17d4d1c6cdc7ce server_cookie=ba49acda0cc2514a in_seq=0 out_seq=0 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.849+0000 7f8f0d600640 1 -- 172.21.15.28:0/2475599085 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557d7f086320 con 0x557d7f1a5330 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.849+0000 7f8f0d600640 10 client.25949 handle_command_reply: tid=0 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.849+0000 7f8f2d97b740 1 client.25949 shutdown 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2475599085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 msgr2=0x557d7f1a3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f1a3380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c011e30 tx=0x7f8f1c004a40 comp rx=0 tx=0).stop 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2475599085 shutdown_connections 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557d7f1a5330 0x557d7f1a7750 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d7f11da00 0x557d7f1a3380 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f0d600640 0 client.25949 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d7f113140 0x557d7f133f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 --2- 172.21.15.28:0/2475599085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d7f112820 0x557d7f133a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2475599085 >> 172.21.15.28:0/2475599085 conn(0x557d7f10ed30 msgr2=0x557d7f10d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2475599085 shutdown_connections 2024-04-26T01:49:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.850+0000 7f8f2d97b740 1 -- 172.21.15.28:0/2475599085 wait complete. 2024-04-26T01:49:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.851+0000 7f8f2d97b740 20 client.25949 trim_cache size 0 max 16384 2024-04-26T01:49:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:49:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:49:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:49:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 -- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8f080a3e40 msgr2=0x7f8f080a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8f080a3e40 0x7f8f080a6300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009fd0 tx=0x7f8f1c0097e0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 -- 172.21.15.28:0/4089881810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24076da0 msgr2=0x7f8f241a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24076da0 0x7f8f241a4f90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8f10005fd0 tx=0x7f8f1000dde0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 -- 172.21.15.28:0/4089881810 shutdown_connections 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8f080a3e40 0x7f8f080a6300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24112de0 0x7f8f2419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f24076da0 0x7f8f241a4f90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 --2- 172.21.15.28:0/4089881810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f24076480 0x7f8f241a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.852+0000 7f8f2ac00640 1 -- 172.21.15.28:0/4089881810 >> 172.21.15.28:0/4089881810 conn(0x7f8f240ffc50 msgr2=0x7f8f24114430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.853+0000 7f8f2ac00640 1 -- 172.21.15.28:0/4089881810 shutdown_connections 2024-04-26T01:49:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:07.853+0000 7f8f2ac00640 1 -- 172.21.15.28:0/4089881810 wait complete. 2024-04-26T01:49:07.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:07 smithi169 ceph-mon[32851]: pgmap v252: 97 pgs: 97 active+clean; 2.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 22 MiB/s wr, 1.73k op/s 2024-04-26T01:49:07.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:07 smithi053 ceph-mon[32593]: pgmap v252: 97 pgs: 97 active+clean; 2.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 22 MiB/s wr, 1.73k op/s 2024-04-26T01:49:08.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:07 smithi028 ceph-mon[28247]: pgmap v252: 97 pgs: 97 active+clean; 2.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 22 MiB/s wr, 1.73k op/s 2024-04-26T01:49:09.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:09 smithi028 ceph-mon[28247]: pgmap v253: 97 pgs: 97 active+clean; 2.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 22 MiB/s wr, 1.73k op/s 2024-04-26T01:49:09.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:09 smithi028 ceph-mon[28247]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:49:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:09 smithi169 ceph-mon[32851]: pgmap v253: 97 pgs: 97 active+clean; 2.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 22 MiB/s wr, 1.73k op/s 2024-04-26T01:49:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:09 smithi169 ceph-mon[32851]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:49:09.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:09 smithi053 ceph-mon[32593]: pgmap v253: 97 pgs: 97 active+clean; 2.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 22 MiB/s wr, 1.73k op/s 2024-04-26T01:49:09.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:09 smithi053 ceph-mon[32593]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:49:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 435.0 s, 2688.0 tps, lat 37.199 ms stddev 43.258 2024-04-26T01:49:11.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:11 smithi028 ceph-mon[28247]: pgmap v255: 97 pgs: 97 active+clean; 2.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 31 MiB/s wr, 2.41k op/s 2024-04-26T01:49:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:11 smithi169 ceph-mon[32851]: pgmap v255: 97 pgs: 97 active+clean; 2.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 31 MiB/s wr, 2.41k op/s 2024-04-26T01:49:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:11 smithi053 ceph-mon[32593]: pgmap v255: 97 pgs: 97 active+clean; 2.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 31 MiB/s wr, 2.41k op/s 2024-04-26T01:49:13.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:13 smithi028 ceph-mon[28247]: pgmap v256: 97 pgs: 97 active+clean; 2.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 26 MiB/s wr, 2.01k op/s 2024-04-26T01:49:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:13 smithi169 ceph-mon[32851]: pgmap v256: 97 pgs: 97 active+clean; 2.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 26 MiB/s wr, 2.01k op/s 2024-04-26T01:49:13.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:13 smithi053 ceph-mon[32593]: pgmap v256: 97 pgs: 97 active+clean; 2.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 26 MiB/s wr, 2.01k op/s 2024-04-26T01:49:15.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:15 smithi028 ceph-mon[28247]: pgmap v257: 97 pgs: 97 active+clean; 2.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 21 MiB/s wr, 1.65k op/s 2024-04-26T01:49:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:15 smithi169 ceph-mon[32851]: pgmap v257: 97 pgs: 97 active+clean; 2.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 21 MiB/s wr, 1.65k op/s 2024-04-26T01:49:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:15 smithi053 ceph-mon[32593]: pgmap v257: 97 pgs: 97 active+clean; 2.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 21 MiB/s wr, 1.65k op/s 2024-04-26T01:49:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 440.0 s, 3121.6 tps, lat 31.999 ms stddev 42.132 2024-04-26T01:49:17.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:17 smithi028 ceph-mon[28247]: pgmap v258: 97 pgs: 97 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 946 op/s 2024-04-26T01:49:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:17 smithi169 ceph-mon[32851]: pgmap v258: 97 pgs: 97 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 946 op/s 2024-04-26T01:49:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:17 smithi053 ceph-mon[32593]: pgmap v258: 97 pgs: 97 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 946 op/s 2024-04-26T01:49:19.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:19 smithi028 ceph-mon[28247]: pgmap v259: 97 pgs: 97 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 946 op/s 2024-04-26T01:49:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:19 smithi169 ceph-mon[32851]: pgmap v259: 97 pgs: 97 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 946 op/s 2024-04-26T01:49:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:19 smithi053 ceph-mon[32593]: pgmap v259: 97 pgs: 97 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 946 op/s 2024-04-26T01:49:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.063+0000 7fc227000640 1 -- 172.21.15.28:0/2745672190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc220104d70 msgr2=0x7fc220108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.063+0000 7fc227000640 1 --2- 172.21.15.28:0/2745672190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc220104d70 0x7fc220108db0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc208009900 tx=0x7fc20802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.063+0000 7fc227000640 1 -- 172.21.15.28:0/2745672190 shutdown_connections 2024-04-26T01:49:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.063+0000 7fc227000640 1 --2- 172.21.15.28:0/2745672190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc220109490 0x7fc220111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.063+0000 7fc227000640 1 --2- 172.21.15.28:0/2745672190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc220104d70 0x7fc220108db0 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc208009900 tx=0x7fc20802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.063+0000 7fc227000640 1 --2- 172.21.15.28:0/2745672190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2201043c0 0x7fc2201047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.063+0000 7fc227000640 1 -- 172.21.15.28:0/2745672190 >> 172.21.15.28:0/2745672190 conn(0x7fc2200ffc80 msgr2=0x7fc2201020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.064+0000 7fc227000640 1 -- 172.21.15.28:0/2745672190 shutdown_connections 2024-04-26T01:49:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.065+0000 7fc227000640 1 -- 172.21.15.28:0/2745672190 wait complete. 2024-04-26T01:49:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.066+0000 7fc227000640 1 Processor -- start 2024-04-26T01:49:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.066+0000 7fc227000640 1 -- start start 2024-04-26T01:49:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.066+0000 7fc227000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2201043c0 0x7fc22010fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.066+0000 7fc227000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc220109490 0x7fc2201102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2201043c0 0x7fc22010fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc227000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2201a37f0 0x7fc2201a3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc227000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc220077590 con 0x7fc2201043c0 2024-04-26T01:49:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc227000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc220077410 con 0x7fc220109490 2024-04-26T01:49:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc227000640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc220077710 con 0x7fc2201a37f0 2024-04-26T01:49:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2201043c0 0x7fc22010fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49510/0 (socket says 172.21.15.28:49510) 2024-04-26T01:49:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc21fe00640 1 -- 172.21.15.28:0/4096150869 learned_addr learned my addr 172.21.15.28:0/4096150869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc225000640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2201a37f0 0x7fc2201a3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc217e00640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc220109490 0x7fc2201102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc21fe00640 1 -- 172.21.15.28:0/4096150869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2201a37f0 msgr2=0x7fc2201a3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc21fe00640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2201a37f0 0x7fc2201a3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc21fe00640 1 -- 172.21.15.28:0/4096150869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc220109490 msgr2=0x7fc2201102b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc21fe00640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc220109490 0x7fc2201102b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.067+0000 7fc21fe00640 1 -- 172.21.15.28:0/4096150869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2201a42e0 con 0x7fc2201043c0 2024-04-26T01:49:21.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.068+0000 7fc21fe00640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2201043c0 0x7fc22010fd70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc21000d950 tx=0x7fc21000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:21.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.069+0000 7fc21d600640 1 -- 172.21.15.28:0/4096150869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210017030 con 0x7fc2201043c0 2024-04-26T01:49:21.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.069+0000 7fc21d600640 1 -- 172.21.15.28:0/4096150869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc210004020 con 0x7fc2201043c0 2024-04-26T01:49:21.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.069+0000 7fc21d600640 1 -- 172.21.15.28:0/4096150869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210017030 con 0x7fc2201043c0 2024-04-26T01:49:21.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.069+0000 7fc227000640 1 -- 172.21.15.28:0/4096150869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2201a45d0 con 0x7fc2201043c0 2024-04-26T01:49:21.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.070+0000 7fc227000640 1 -- 172.21.15.28:0/4096150869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2201abd40 con 0x7fc2201043c0 2024-04-26T01:49:21.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.075+0000 7fc229f83740 1 Processor -- start 2024-04-26T01:49:21.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.075+0000 7fc229f83740 1 -- start start 2024-04-26T01:49:21.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc229f83740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe326c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc229f83740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe327140 0x558bbe3314c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc229f83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 0x558bbe333df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc229f83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558bbe28c900 con 0x558bbe331a00 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc229f83740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558bbe232ef0 con 0x558bbe327140 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc229f83740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558bbe2888c0 con 0x558bbe326820 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc225000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe326c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc225000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe326c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58174/0 (socket says 172.21.15.28:58174) 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc225000640 1 -- 172.21.15.28:0/4125074372 learned_addr learned my addr 172.21.15.28:0/4125074372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.076+0000 7fc217e00640 1 --2- 172.21.15.28:0/4125074372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 0x558bbe333df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.077+0000 7fc21d600640 1 -- 172.21.15.28:0/4096150869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc210005450 con 0x7fc2201043c0 2024-04-26T01:49:21.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.077+0000 7fc225000640 1 -- 172.21.15.28:0/4125074372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe327140 msgr2=0x558bbe3314c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:49:21.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.077+0000 7fc225000640 1 --2- 172.21.15.28:0/4125074372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe327140 0x558bbe3314c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.077+0000 7fc225000640 1 -- 172.21.15.28:0/4125074372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 msgr2=0x558bbe333df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:21.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.077+0000 7fc225000640 1 --2- 172.21.15.28:0/4125074372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 0x558bbe333df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.077+0000 7fc225000640 1 -- 172.21.15.28:0/4125074372 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bbdf9f210 con 0x558bbe326820 2024-04-26T01:49:21.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.077+0000 7fc225000640 1 --2- 172.21.15.28:0/4125074372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe326c00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc218009f50 tx=0x7fc2180118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fc0875ff2ef0d09 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.077+0000 7fc215600640 1 -- 172.21.15.28:0/4125074372 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218004b20 con 0x558bbe326820 2024-04-26T01:49:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.078+0000 7fc215600640 1 -- 172.21.15.28:0/4125074372 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc218004cc0 con 0x558bbe326820 2024-04-26T01:49:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.078+0000 7fc215600640 1 -- 172.21.15.28:0/4125074372 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218005420 con 0x558bbe326820 2024-04-26T01:49:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 -- 172.21.15.28:0/4125074372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 msgr2=0x558bbe326c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 --2- 172.21.15.28:0/4125074372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe326c00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc218009f50 tx=0x7fc2180118d0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc21d600640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc2040a4090 0x7fc2040a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc21d600640 1 -- 172.21.15.28:0/4096150869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fc2100cfb80 con 0x7fc2201043c0 2024-04-26T01:49:21.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 -- 172.21.15.28:0/4125074372 shutdown_connections 2024-04-26T01:49:21.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 --2- 172.21.15.28:0/4125074372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 0x558bbe333df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 --2- 172.21.15.28:0/4125074372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe327140 0x558bbe3314c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 --2- 172.21.15.28:0/4125074372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe326c00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 -- 172.21.15.28:0/4125074372 >> 172.21.15.28:0/4125074372 conn(0x558bbe322d30 msgr2=0x558bbe323110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc21fe00640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc2040a4090 0x7fc2040a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 -- 172.21.15.28:0/4125074372 shutdown_connections 2024-04-26T01:49:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.079+0000 7fc229f83740 1 -- 172.21.15.28:0/4125074372 wait complete. 2024-04-26T01:49:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.080+0000 7fc229f83740 1 -- start start 2024-04-26T01:49:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.080+0000 7fc229f83740 1 Processor -- start 2024-04-26T01:49:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc229f83740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe32e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc229f83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe32ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc229f83740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe32f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc229f83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558bbe334630 con 0x558bbe327140 2024-04-26T01:49:21.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc229f83740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558bbe246310 con 0x558bbe331a00 2024-04-26T01:49:21.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc229f83740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558bbe3347b0 con 0x558bbe326820 2024-04-26T01:49:21.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc229f83740 10 client.0 _ll_register_callbacks cb 0x558bbe214cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:21.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc225000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe32e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc225000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe32e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58178/0 (socket says 172.21.15.28:58178) 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc225000640 1 -- 172.21.15.28:0/1485794848 learned_addr learned my addr 172.21.15.28:0/1485794848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc225000640 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe32f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.081+0000 7fc217e00640 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe32ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc225000640 1 -- 172.21.15.28:0/1485794848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 msgr2=0x558bbe32f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc225000640 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe32f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc225000640 1 -- 172.21.15.28:0/1485794848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 msgr2=0x558bbe32ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc225000640 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe32ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc225000640 1 -- 172.21.15.28:0/1485794848 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bbe06b620 con 0x558bbe326820 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc217e00640 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe32ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc21fe00640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc2040a4090 0x7fc2040a6550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x558bbe327dc0 tx=0x7fc210097dc0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc225000640 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe32e9e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc218012fd0 tx=0x7fc21800ff40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=788615e773f98f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc20e000640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc20e000640 1 -- 172.21.15.28:0/1485794848 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21800f830 con 0x558bbe326820 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc229f83740 1 -- 172.21.15.28:0/1485794848 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc218004cc0 con 0x558bbe326820 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.082+0000 7fc20e000640 1 -- 172.21.15.28:0/1485794848 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fc21800f9d0 con 0x558bbe326820 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc20e000640 1 -- 172.21.15.28:0/1485794848 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc218036050 con 0x558bbe326820 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc229f83740 10 client.25972 fetch_fsmap learned FSMap version 38 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc229f83740 1 -- 172.21.15.28:0/1485794848 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558bbe3518f0 con 0x558bbe326820 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc20e000640 1 -- 172.21.15.28:0/1485794848 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc218004cc0 con 0x558bbe326820 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc20e000640 1 -- 172.21.15.28:0/1485794848 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fc218025070 con 0x558bbe326820 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc229f83740 10 client.25972 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc229f83740 10 client.25972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:21.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc229f83740 20 client.25972 populate_metadata read hostname 'smithi028' 2024-04-26T01:49:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.083+0000 7fc229f83740 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bbe352cf0 0x558bbe3550e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.084+0000 7fc229f83740 4 client.25972 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.084+0000 7fc229f83740 1 -- 172.21.15.28:0/1485794848 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bbe2a08b0 con 0x558bbe352cf0 2024-04-26T01:49:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.084+0000 7fc217e00640 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bbe352cf0 0x558bbe3550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.085+0000 7fc217e00640 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bbe352cf0 0x558bbe3550e0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=291a69b0e5b8cda0 server_cookie=80d42b91cfa3e63 in_seq=0 out_seq=0 2024-04-26T01:49:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.085+0000 7fc20e000640 10 client.25972 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.085+0000 7fc20e000640 1 -- 172.21.15.28:0/1485794848 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558bbe2a08b0 con 0x558bbe352cf0 2024-04-26T01:49:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.085+0000 7fc20e000640 10 client.25972 handle_command_reply: tid=0 2024-04-26T01:49:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 client.25972 shutdown 2024-04-26T01:49:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 -- 172.21.15.28:0/1485794848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 msgr2=0x558bbe32e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe32e9e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc218012fd0 tx=0x7fc21800ff40 comp rx=0 tx=0).stop 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 -- 172.21.15.28:0/1485794848 shutdown_connections 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bbe352cf0 0x558bbe3550e0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe32f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe32ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 --2- 172.21.15.28:0/1485794848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe32e9e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 -- 172.21.15.28:0/1485794848 >> 172.21.15.28:0/1485794848 conn(0x558bbe322d30 msgr2=0x558bbe328a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc20e000640 0 client.25972 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 -- 172.21.15.28:0/1485794848 shutdown_connections 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.086+0000 7fc229f83740 1 -- 172.21.15.28:0/1485794848 wait complete. 2024-04-26T01:49:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.087+0000 7fc229f83740 20 client.25972 trim_cache size 0 max 16384 2024-04-26T01:49:21.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.115+0000 7fc229f83740 1 Processor -- start 2024-04-26T01:49:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.115+0000 7fc229f83740 1 -- start start 2024-04-26T01:49:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.115+0000 7fc229f83740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe326820 0x558bbe34e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc229f83740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe327140 0x558bbe32f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc225000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe326820 0x558bbe34e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc225000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe326820 0x558bbe34e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53500/0 (socket says 172.21.15.28:53500) 2024-04-26T01:49:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc229f83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 0x558bbe35a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc229f83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2201ae8e0 con 0x558bbe331a00 2024-04-26T01:49:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc229f83740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc2201ac1b0 con 0x558bbe326820 2024-04-26T01:49:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc229f83740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc220119a60 con 0x558bbe327140 2024-04-26T01:49:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc225000640 1 -- 172.21.15.28:0/3375725019 learned_addr learned my addr 172.21.15.28:0/3375725019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc217e00640 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe327140 0x558bbe32f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.116+0000 7fc225000640 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 0x558bbe35a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.117+0000 7fc217e00640 1 -- 172.21.15.28:0/3375725019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe326820 msgr2=0x558bbe34e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.117+0000 7fc217e00640 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe326820 0x558bbe34e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.117+0000 7fc217e00640 1 -- 172.21.15.28:0/3375725019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 msgr2=0x558bbe35a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.117+0000 7fc217e00640 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 0x558bbe35a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.117+0000 7fc217e00640 1 -- 172.21.15.28:0/3375725019 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bbe32cf30 con 0x558bbe327140 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.117+0000 7fc217e00640 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe327140 0x558bbe32f280 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc2201052c0 tx=0x7fc208005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95d021a84f65596c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:21.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.117+0000 7fc214c00640 1 -- 172.21.15.28:0/3375725019 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208035040 con 0x558bbe327140 2024-04-26T01:49:21.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.118+0000 7fc214c00640 1 -- 172.21.15.28:0/3375725019 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc208004160 con 0x558bbe327140 2024-04-26T01:49:21.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.118+0000 7fc214c00640 1 -- 172.21.15.28:0/3375725019 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208005550 con 0x558bbe327140 2024-04-26T01:49:21.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.119+0000 7fc229f83740 1 -- 172.21.15.28:0/3375725019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe327140 msgr2=0x558bbe32f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.119+0000 7fc229f83740 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe327140 0x558bbe32f280 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc2201052c0 tx=0x7fc208005fd0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.119+0000 7fc229f83740 1 -- 172.21.15.28:0/3375725019 shutdown_connections 2024-04-26T01:49:21.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.119+0000 7fc229f83740 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe331a00 0x558bbe35a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.119+0000 7fc229f83740 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe327140 0x558bbe32f280 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.119+0000 7fc229f83740 1 --2- 172.21.15.28:0/3375725019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe326820 0x558bbe34e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.119+0000 7fc229f83740 1 -- 172.21.15.28:0/3375725019 >> 172.21.15.28:0/3375725019 conn(0x558bbe322d30 msgr2=0x558bbe321e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:21.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.120+0000 7fc229f83740 1 -- 172.21.15.28:0/3375725019 shutdown_connections 2024-04-26T01:49:21.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.120+0000 7fc229f83740 1 -- 172.21.15.28:0/3375725019 wait complete. 2024-04-26T01:49:21.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.121+0000 7fc229f83740 1 -- start start 2024-04-26T01:49:21.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.121+0000 7fc229f83740 1 Processor -- start 2024-04-26T01:49:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.121+0000 7fc229f83740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe34aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.121+0000 7fc229f83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe348020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.121+0000 7fc229f83740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe348560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.121+0000 7fc229f83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc22011a0b0 con 0x558bbe327140 2024-04-26T01:49:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.121+0000 7fc229f83740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc22011b520 con 0x558bbe331a00 2024-04-26T01:49:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.121+0000 7fc229f83740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc22011a780 con 0x558bbe326820 2024-04-26T01:49:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.122+0000 7fc229f83740 10 client.0 _ll_register_callbacks cb 0x7fc218003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:21.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.122+0000 7fc217e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe348020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.122+0000 7fc225000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe348560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.122+0000 7fc217e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe348020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49550/0 (socket says 172.21.15.28:49550) 2024-04-26T01:49:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.122+0000 7fc217e00640 1 -- 172.21.15.28:0/2816792284 learned_addr learned my addr 172.21.15.28:0/2816792284 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:21.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.122+0000 7fc225000640 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe34aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.123+0000 7fc225000640 1 -- 172.21.15.28:0/2816792284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 msgr2=0x558bbe34aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.123+0000 7fc225000640 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe34aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.123+0000 7fc225000640 1 -- 172.21.15.28:0/2816792284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 msgr2=0x558bbe348020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.123+0000 7fc225000640 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe348020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.123+0000 7fc225000640 1 -- 172.21.15.28:0/2816792284 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bbe348ad0 con 0x558bbe331a00 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.123+0000 7fc217e00640 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe348020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.123+0000 7fc225000640 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe348560 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc2180040a0 tx=0x7fc218003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=219a3cb223a54738 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.123+0000 7fc20e000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.124+0000 7fc20e000640 1 -- 172.21.15.28:0/2816792284 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21800fa30 con 0x558bbe331a00 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.124+0000 7fc229f83740 1 -- 172.21.15.28:0/2816792284 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558bbe29a320 con 0x558bbe331a00 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.124+0000 7fc20e000640 1 -- 172.21.15.28:0/2816792284 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fc21803a430 con 0x558bbe331a00 2024-04-26T01:49:21.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc20e000640 1 -- 172.21.15.28:0/2816792284 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc218004480 con 0x558bbe331a00 2024-04-26T01:49:21.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc229f83740 10 client.25964 fetch_fsmap learned FSMap version 38 2024-04-26T01:49:21.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc229f83740 1 -- 172.21.15.28:0/2816792284 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558bbe3b8dd0 con 0x558bbe331a00 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc20e000640 1 -- 172.21.15.28:0/2816792284 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558bbe29a320 con 0x558bbe331a00 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc20e000640 1 -- 172.21.15.28:0/2816792284 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fc218025040 con 0x558bbe331a00 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc229f83740 10 client.25964 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc229f83740 10 client.25964 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc229f83740 20 client.25964 populate_metadata read hostname 'smithi028' 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc229f83740 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bbe3b9430 0x558bbe3bb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc229f83740 4 client.25964 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.125+0000 7fc229f83740 1 -- 172.21.15.28:0/2816792284 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558bbe294190 con 0x558bbe3b9430 2024-04-26T01:49:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.126+0000 7fc217e00640 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bbe3b9430 0x558bbe3bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:21.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.126+0000 7fc20e000640 10 client.25964 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:21.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.126+0000 7fc217e00640 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bbe3b9430 0x558bbe3bb890 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7332de4e54ae06d server_cookie=e13a28bd5dac36cb in_seq=0 out_seq=0 2024-04-26T01:49:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.144+0000 7fc20e000640 1 -- 172.21.15.28:0/2816792284 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558bbe294190 con 0x558bbe3b9430 2024-04-26T01:49:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.144+0000 7fc20e000640 10 client.25964 handle_command_reply: tid=0 2024-04-26T01:49:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.144+0000 7fc229f83740 1 client.25964 shutdown 2024-04-26T01:49:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 -- 172.21.15.28:0/2816792284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 msgr2=0x558bbe348560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe348560 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc2180040a0 tx=0x7fc218003c80 comp rx=0 tx=0).stop 2024-04-26T01:49:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 -- 172.21.15.28:0/2816792284 shutdown_connections 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bbe3b9430 0x558bbe3bb890 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc20e000640 0 client.25964 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bbe331a00 0x558bbe348560 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bbe327140 0x558bbe348020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 --2- 172.21.15.28:0/2816792284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bbe326820 0x558bbe34aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 -- 172.21.15.28:0/2816792284 >> 172.21.15.28:0/2816792284 conn(0x558bbe322d30 msgr2=0x558bbe3281a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.145+0000 7fc229f83740 1 -- 172.21.15.28:0/2816792284 shutdown_connections 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.146+0000 7fc229f83740 1 -- 172.21.15.28:0/2816792284 wait complete. 2024-04-26T01:49:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.146+0000 7fc229f83740 20 client.25964 trim_cache size 0 max 16384 2024-04-26T01:49:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:49:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:49:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:49:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:49:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.147+0000 7fc227000640 1 -- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc2040a4090 msgr2=0x7fc2040a6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.147+0000 7fc227000640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc2040a4090 0x7fc2040a6550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x558bbe327dc0 tx=0x7fc210097dc0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.147+0000 7fc227000640 1 -- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2201043c0 msgr2=0x7fc22010fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.147+0000 7fc227000640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2201043c0 0x7fc22010fd70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc21000d950 tx=0x7fc21000de10 comp rx=0 tx=0).stop 2024-04-26T01:49:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.148+0000 7fc227000640 1 -- 172.21.15.28:0/4096150869 shutdown_connections 2024-04-26T01:49:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.148+0000 7fc227000640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc2040a4090 0x7fc2040a6550 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.148+0000 7fc227000640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2201a37f0 0x7fc2201a3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.148+0000 7fc227000640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc220109490 0x7fc2201102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.148+0000 7fc227000640 1 --2- 172.21.15.28:0/4096150869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2201043c0 0x7fc22010fd70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.148+0000 7fc227000640 1 -- 172.21.15.28:0/4096150869 >> 172.21.15.28:0/4096150869 conn(0x7fc2200ffc80 msgr2=0x7fc2201020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:21.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.149+0000 7fc227000640 1 -- 172.21.15.28:0/4096150869 shutdown_connections 2024-04-26T01:49:21.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:21.149+0000 7fc227000640 1 -- 172.21.15.28:0/4096150869 wait complete. 2024-04-26T01:49:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 445.0 s, 2373.6 tps, lat 41.829 ms stddev 51.520 2024-04-26T01:49:21.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:21 smithi169 ceph-mon[32851]: pgmap v260: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-26T01:49:21.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:21 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:21 smithi053 ceph-mon[32593]: pgmap v260: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-26T01:49:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:21 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:21 smithi028 ceph-mon[28247]: pgmap v260: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 13 MiB/s wr, 1.10k op/s 2024-04-26T01:49:21.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:21 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:22.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:22 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:22 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:22 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:22 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:22 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:22 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:22 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:22 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:22 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:22 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:22 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:22.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:22 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:23.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:23 smithi169 ceph-mon[32851]: pgmap v261: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.4 MiB/s wr, 625 op/s 2024-04-26T01:49:23.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:23 smithi053 ceph-mon[32593]: pgmap v261: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.4 MiB/s wr, 625 op/s 2024-04-26T01:49:23.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:23 smithi028 ceph-mon[28247]: pgmap v261: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.4 MiB/s wr, 625 op/s 2024-04-26T01:49:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:24 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:24 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:24 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:24.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:24 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:24.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:24 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:24.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:24 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:24.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:24 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:24.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:24 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:24.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:24 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:49:25.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:25 smithi053 ceph-mon[32593]: pgmap v262: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.1 MiB/s wr, 778 op/s 2024-04-26T01:49:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:25 smithi028 ceph-mon[28247]: pgmap v262: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.1 MiB/s wr, 778 op/s 2024-04-26T01:49:25.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:25 smithi169 ceph-mon[32851]: pgmap v262: 97 pgs: 97 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.1 MiB/s wr, 778 op/s 2024-04-26T01:49:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 450.0 s, 2555.3 tps, lat 39.068 ms stddev 47.128 2024-04-26T01:49:27.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:27 smithi053 ceph-mon[32593]: pgmap v263: 97 pgs: 97 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.3 MiB/s wr, 833 op/s 2024-04-26T01:49:27.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:27 smithi028 ceph-mon[28247]: pgmap v263: 97 pgs: 97 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.3 MiB/s wr, 833 op/s 2024-04-26T01:49:27.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:27 smithi169 ceph-mon[32851]: pgmap v263: 97 pgs: 97 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.3 MiB/s wr, 833 op/s 2024-04-26T01:49:29.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:29 smithi053 ceph-mon[32593]: pgmap v264: 97 pgs: 97 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.8 MiB/s wr, 691 op/s 2024-04-26T01:49:29.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:29 smithi028 ceph-mon[28247]: pgmap v264: 97 pgs: 97 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.8 MiB/s wr, 691 op/s 2024-04-26T01:49:29.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:29 smithi169 ceph-mon[32851]: pgmap v264: 97 pgs: 97 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.8 MiB/s wr, 691 op/s 2024-04-26T01:49:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 455.0 s, 2317.4 tps, lat 43.255 ms stddev 53.160 2024-04-26T01:49:31.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:31 smithi053 ceph-mon[32593]: pgmap v265: 97 pgs: 97 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-04-26T01:49:31.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:31 smithi028 ceph-mon[28247]: pgmap v265: 97 pgs: 97 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-04-26T01:49:31.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:31 smithi169 ceph-mon[32851]: pgmap v265: 97 pgs: 97 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-04-26T01:49:33.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:33 smithi053 ceph-mon[32593]: pgmap v266: 97 pgs: 97 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.6 MiB/s wr, 684 op/s 2024-04-26T01:49:33.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:33 smithi028 ceph-mon[28247]: pgmap v266: 97 pgs: 97 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.6 MiB/s wr, 684 op/s 2024-04-26T01:49:33.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:33 smithi169 ceph-mon[32851]: pgmap v266: 97 pgs: 97 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.6 MiB/s wr, 684 op/s 2024-04-26T01:49:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.365+0000 7fc9aec00640 1 -- 172.21.15.28:0/1287661520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9a8109870 msgr2=0x7fc9a8114b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.365+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1287661520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9a8109870 0x7fc9a8114b60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc994009930 tx=0x7fc99402a4f0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.367+0000 7fc9aec00640 1 -- 172.21.15.28:0/1287661520 shutdown_connections 2024-04-26T01:49:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.367+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1287661520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a81150a0 0x7fc9a8117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.367+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1287661520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9a8109870 0x7fc9a8114b60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.367+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1287661520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a8108f50 0x7fc9a8109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.367+0000 7fc9aec00640 1 -- 172.21.15.28:0/1287661520 >> 172.21.15.28:0/1287661520 conn(0x7fc9a806d020 msgr2=0x7fc9a806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.367+0000 7fc9aec00640 1 -- 172.21.15.28:0/1287661520 shutdown_connections 2024-04-26T01:49:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.367+0000 7fc9aec00640 1 -- 172.21.15.28:0/1287661520 wait complete. 2024-04-26T01:49:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.369+0000 7fc9aec00640 1 Processor -- start 2024-04-26T01:49:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.370+0000 7fc9aec00640 1 -- start start 2024-04-26T01:49:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.370+0000 7fc9aec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9a8108f50 0x7fc9a81a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.370+0000 7fc9aec00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a8109870 0x7fc9a81a3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.370+0000 7fc9aec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a81150a0 0x7fc9a81a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.370+0000 7fc9aec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a811aa80 con 0x7fc9a81150a0 2024-04-26T01:49:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.370+0000 7fc9aec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a811a900 con 0x7fc9a8108f50 2024-04-26T01:49:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.370+0000 7fc9aec00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a811ac00 con 0x7fc9a8109870 2024-04-26T01:49:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.371+0000 7fc9a7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a8109870 0x7fc9a81a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.371+0000 7fc9a7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a8109870 0x7fc9a81a3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:44108/0 (socket says 172.21.15.28:44108) 2024-04-26T01:49:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.371+0000 7fc9acc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a81150a0 0x7fc9a81a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.371+0000 7fc9a7400640 1 -- 172.21.15.28:0/1431985511 learned_addr learned my addr 172.21.15.28:0/1431985511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.371+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9a8108f50 0x7fc9a81a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.372+0000 7fc9a7400640 1 -- 172.21.15.28:0/1431985511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9a8108f50 msgr2=0x7fc9a81a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.372+0000 7fc9a7400640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9a8108f50 0x7fc9a81a2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.372+0000 7fc9a7400640 1 -- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a81150a0 msgr2=0x7fc9a81a78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.372+0000 7fc9a7400640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a81150a0 0x7fc9a81a78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.372+0000 7fc9a7400640 1 -- 172.21.15.28:0/1431985511 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a81a7fd0 con 0x7fc9a8109870 2024-04-26T01:49:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.373+0000 7fc9a7400640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a8109870 0x7fc9a81a3520 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc99402aae0 tx=0x7fc994005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.373+0000 7fc9a4c00640 1 -- 172.21.15.28:0/1431985511 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99403f070 con 0x7fc9a8109870 2024-04-26T01:49:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.374+0000 7fc9a4c00640 1 -- 172.21.15.28:0/1431985511 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc994004460 con 0x7fc9a8109870 2024-04-26T01:49:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.374+0000 7fc9a4c00640 1 -- 172.21.15.28:0/1431985511 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99403f070 con 0x7fc9a8109870 2024-04-26T01:49:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.375+0000 7fc9aec00640 1 -- 172.21.15.28:0/1431985511 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a81a8260 con 0x7fc9a8109870 2024-04-26T01:49:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.375+0000 7fc9aec00640 1 -- 172.21.15.28:0/1431985511 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a81a8610 con 0x7fc9a8109870 2024-04-26T01:49:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.380+0000 7fc9b199f740 1 Processor -- start 2024-04-26T01:49:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.382+0000 7fc9a4c00640 1 -- 172.21.15.28:0/1431985511 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc994042070 con 0x7fc9a8109870 2024-04-26T01:49:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.384+0000 7fc9a4c00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc98c0a4040 0x7fc98c0a6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.385+0000 7fc9a4c00640 1 -- 172.21.15.28:0/1431985511 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fc9940035f0 con 0x7fc9a8109870 2024-04-26T01:49:34.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.385+0000 7fc9acc00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc98c0a4040 0x7fc98c0a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.386+0000 7fc9b199f740 1 -- start start 2024-04-26T01:49:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.386+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c216c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.386+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c217140 0x56546c2214c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.386+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c221a00 0x56546c223df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.386+0000 7fc9b199f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56546c17c900 con 0x56546c216820 2024-04-26T01:49:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.386+0000 7fc9b199f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56546c122ef0 con 0x56546c221a00 2024-04-26T01:49:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.386+0000 7fc9b199f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x56546c1788c0 con 0x56546c217140 2024-04-26T01:49:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.387+0000 7fc9a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c216c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.387+0000 7fc9acc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c221a00 0x56546c223df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.387+0000 7fc9acc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c221a00 0x56546c223df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:55918/0 (socket says 172.21.15.28:55918) 2024-04-26T01:49:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.387+0000 7fc9acc00640 1 -- 172.21.15.28:0/3749251988 learned_addr learned my addr 172.21.15.28:0/3749251988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.387+0000 7fc9a7400640 1 --2- 172.21.15.28:0/3749251988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c217140 0x56546c2214c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.388+0000 7fc9a7e00640 1 -- 172.21.15.28:0/3749251988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c217140 msgr2=0x56546c2214c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.388+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/3749251988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c217140 0x56546c2214c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.388+0000 7fc9a7e00640 1 -- 172.21.15.28:0/3749251988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c221a00 msgr2=0x56546c223df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.388+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/3749251988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c221a00 0x56546c223df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.388+0000 7fc9a7e00640 1 -- 172.21.15.28:0/3749251988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56546be8f210 con 0x56546c216820 2024-04-26T01:49:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.389+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/3749251988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c216c00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc99800b760 tx=0x7fc99800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e003fb28cab00b20 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.390+0000 7fc99d200640 1 -- 172.21.15.28:0/3749251988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99800d950 con 0x56546c216820 2024-04-26T01:49:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.391+0000 7fc9acc00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc98c0a4040 0x7fc98c0a6500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009fd0 tx=0x7fc9a001fb80 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.391+0000 7fc99d200640 1 -- 172.21.15.28:0/3749251988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc998019070 con 0x56546c216820 2024-04-26T01:49:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.391+0000 7fc99d200640 1 -- 172.21.15.28:0/3749251988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998014d40 con 0x56546c216820 2024-04-26T01:49:34.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 -- 172.21.15.28:0/3749251988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 msgr2=0x56546c216c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 --2- 172.21.15.28:0/3749251988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c216c00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc99800b760 tx=0x7fc99800bc20 comp rx=0 tx=0).stop 2024-04-26T01:49:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 -- 172.21.15.28:0/3749251988 shutdown_connections 2024-04-26T01:49:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 --2- 172.21.15.28:0/3749251988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c221a00 0x56546c223df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 --2- 172.21.15.28:0/3749251988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c217140 0x56546c2214c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 --2- 172.21.15.28:0/3749251988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c216c00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 -- 172.21.15.28:0/3749251988 >> 172.21.15.28:0/3749251988 conn(0x56546c212d30 msgr2=0x56546c213110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 -- 172.21.15.28:0/3749251988 shutdown_connections 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.392+0000 7fc9b199f740 1 -- 172.21.15.28:0/3749251988 wait complete. 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.393+0000 7fc9b199f740 1 -- start start 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.393+0000 7fc9b199f740 1 Processor -- start 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c21e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c21eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c21f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9b199f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56546c224630 con 0x56546c216820 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9b199f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56546c136310 con 0x56546c217140 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9b199f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x56546c2247b0 con 0x56546c221a00 2024-04-26T01:49:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9acc00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c21f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c21e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9acc00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c21f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:44122/0 (socket says 172.21.15.28:44122) 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.394+0000 7fc9acc00640 1 -- 172.21.15.28:0/1776088440 learned_addr learned my addr 172.21.15.28:0/1776088440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9b199f740 10 client.0 _ll_register_callbacks cb 0x56546c104cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9a7400640 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c21eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9acc00640 1 -- 172.21.15.28:0/1776088440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 msgr2=0x56546c21eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9acc00640 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c21eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9acc00640 1 -- 172.21.15.28:0/1776088440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 msgr2=0x56546c21e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9acc00640 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c21e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9acc00640 1 -- 172.21.15.28:0/1776088440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56546bf5b620 con 0x56546c221a00 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9b199f740 1 -- 172.21.15.28:0/1776088440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc998019070 con 0x56546c221a00 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc991600640 10 client.25987 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.395+0000 7fc9acc00640 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c21f430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc9a001dd50 tx=0x7fc9a0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66d978be83a8144f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.396+0000 7fc991600640 1 -- 172.21.15.28:0/1776088440 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00140b0 con 0x56546c221a00 2024-04-26T01:49:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.396+0000 7fc991600640 1 -- 172.21.15.28:0/1776088440 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fc9a0025040 con 0x56546c221a00 2024-04-26T01:49:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.396+0000 7fc991600640 1 -- 172.21.15.28:0/1776088440 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc9a00140b0 con 0x56546c221a00 2024-04-26T01:49:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.396+0000 7fc9b199f740 10 client.25987 fetch_fsmap learned FSMap version 38 2024-04-26T01:49:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.396+0000 7fc9b199f740 1 -- 172.21.15.28:0/1776088440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({fsmap=38}) -- 0x56546c230840 con 0x56546c221a00 2024-04-26T01:49:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.397+0000 7fc991600640 1 -- 172.21.15.28:0/1776088440 <== mon.1 v2:172.21.15.169:3300/0 4 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fc9a000f040 con 0x56546c221a00 2024-04-26T01:49:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.397+0000 7fc9b199f740 10 client.25987 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:49:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.397+0000 7fc9b199f740 10 client.25987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.397+0000 7fc9b199f740 20 client.25987 populate_metadata read hostname 'smithi028' 2024-04-26T01:49:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.397+0000 7fc9b199f740 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56546c2320f0 0x56546c234550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.398+0000 7fc9b199f740 4 client.25987 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.398+0000 7fc9b199f740 1 -- 172.21.15.28:0/1776088440 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56546c1908b0 con 0x56546c2320f0 2024-04-26T01:49:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.398+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56546c2320f0 0x56546c234550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.399+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56546c2320f0 0x56546c234550 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7494293983f0243 server_cookie=7ae78ab882ec4925 in_seq=0 out_seq=0 2024-04-26T01:49:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.399+0000 7fc991600640 10 client.25987 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.401+0000 7fc991600640 1 -- 172.21.15.28:0/1776088440 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56546c1908b0 con 0x56546c2320f0 2024-04-26T01:49:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.401+0000 7fc991600640 10 client.25987 handle_command_reply: tid=0 2024-04-26T01:49:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.401+0000 7fc9b199f740 1 client.25987 shutdown 2024-04-26T01:49:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc9b199f740 1 -- 172.21.15.28:0/1776088440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 msgr2=0x56546c21f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc9b199f740 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c21f430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc9a001dd50 tx=0x7fc9a0004ab0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc9b199f740 1 -- 172.21.15.28:0/1776088440 shutdown_connections 2024-04-26T01:49:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc9b199f740 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56546c2320f0 0x56546c234550 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc9b199f740 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c21f430 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc9b199f740 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c21eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc9b199f740 1 --2- 172.21.15.28:0/1776088440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c21e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc991600640 0 client.25987 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.402+0000 7fc9b199f740 1 -- 172.21.15.28:0/1776088440 >> 172.21.15.28:0/1776088440 conn(0x56546c212d30 msgr2=0x56546c218a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.404+0000 7fc9b199f740 1 -- 172.21.15.28:0/1776088440 shutdown_connections 2024-04-26T01:49:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.404+0000 7fc9b199f740 1 -- 172.21.15.28:0/1776088440 wait complete. 2024-04-26T01:49:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.404+0000 7fc9b199f740 20 client.25987 trim_cache size 0 max 16384 2024-04-26T01:49:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.436+0000 7fc9b199f740 1 Processor -- start 2024-04-26T01:49:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.437+0000 7fc9b199f740 1 -- start start 2024-04-26T01:49:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.437+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c216820 0x56546c22d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.437+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c2191e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.437+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c221a00 0x56546c239e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.437+0000 7fc9b199f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a81b6a40 con 0x56546c221a00 2024-04-26T01:49:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.437+0000 7fc9b199f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a81b4310 con 0x56546c217140 2024-04-26T01:49:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.437+0000 7fc9b199f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a81b2100 con 0x56546c216820 2024-04-26T01:49:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c216820 0x56546c22d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c216820 0x56546c22d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:44130/0 (socket says 172.21.15.28:44130) 2024-04-26T01:49:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7e00640 1 -- 172.21.15.28:0/3096377863 learned_addr learned my addr 172.21.15.28:0/3096377863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7400640 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c2191e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9acc00640 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c221a00 0x56546c239e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7400640 1 -- 172.21.15.28:0/3096377863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c216820 msgr2=0x56546c22d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7400640 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c216820 0x56546c22d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7400640 1 -- 172.21.15.28:0/3096377863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c221a00 msgr2=0x56546c239e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7400640 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c221a00 0x56546c239e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.438+0000 7fc9a7400640 1 -- 172.21.15.28:0/3096377863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56546c217ad0 con 0x56546c217140 2024-04-26T01:49:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.439+0000 7fc9acc00640 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c221a00 0x56546c239e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.439+0000 7fc9a7400640 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c2191e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc9940ee3d0 tx=0x7fc9940c11c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f006d737621f64e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.440+0000 7fc993e00640 1 -- 172.21.15.28:0/3096377863 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9940b9650 con 0x56546c217140 2024-04-26T01:49:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.440+0000 7fc993e00640 1 -- 172.21.15.28:0/3096377863 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9940c0fd0 con 0x56546c217140 2024-04-26T01:49:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.440+0000 7fc993e00640 1 -- 172.21.15.28:0/3096377863 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9940b97f0 con 0x56546c217140 2024-04-26T01:49:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 -- 172.21.15.28:0/3096377863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 msgr2=0x56546c2191e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c2191e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc9940ee3d0 tx=0x7fc9940c11c0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 -- 172.21.15.28:0/3096377863 shutdown_connections 2024-04-26T01:49:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c221a00 0x56546c239e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c2191e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 --2- 172.21.15.28:0/3096377863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c216820 0x56546c22d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 -- 172.21.15.28:0/3096377863 >> 172.21.15.28:0/3096377863 conn(0x56546c212d30 msgr2=0x56546c21f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 -- 172.21.15.28:0/3096377863 shutdown_connections 2024-04-26T01:49:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 -- 172.21.15.28:0/3096377863 wait complete. 2024-04-26T01:49:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.441+0000 7fc9b199f740 1 -- start start 2024-04-26T01:49:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.442+0000 7fc9b199f740 1 Processor -- start 2024-04-26T01:49:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.444+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c290a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.444+0000 7fc9b199f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c290fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9a7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c290fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9a7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c290fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:55936/0 (socket says 172.21.15.28:55936) 2024-04-26T01:49:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9a7400640 1 -- 172.21.15.28:0/496517428 learned_addr learned my addr 172.21.15.28:0/496517428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9b199f740 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c226d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a81b27d0 con 0x56546c216820 2024-04-26T01:49:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c290a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a81b3c40 con 0x56546c217140 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a81b2ea0 con 0x56546c221a00 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.445+0000 7fc9b199f740 10 client.0 _ll_register_callbacks cb 0x7fc9a001dc30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc9acc00640 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c226d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc9a7400640 1 -- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 msgr2=0x56546c226d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc9a7400640 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c226d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc9a7400640 1 -- 172.21.15.28:0/496517428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 msgr2=0x56546c290a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc9a7400640 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c290a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc9a7400640 1 -- 172.21.15.28:0/496517428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56546c227480 con 0x56546c217140 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc9acc00640 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c226d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc9a7400640 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c290fb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc9940037f0 tx=0x7fc9940b8fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c49e734c6ad09223 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.446+0000 7fc991600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:49:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.447+0000 7fc991600640 1 -- 172.21.15.28:0/496517428 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9940d5070 con 0x56546c217140 2024-04-26T01:49:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.447+0000 7fc991600640 1 -- 172.21.15.28:0/496517428 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fc9940dd240 con 0x56546c217140 2024-04-26T01:49:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.447+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9940c0fd0 con 0x56546c217140 2024-04-26T01:49:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.448+0000 7fc991600640 1 -- 172.21.15.28:0/496517428 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc9940cb050 con 0x56546c217140 2024-04-26T01:49:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.448+0000 7fc9b199f740 10 client.25982 fetch_fsmap learned FSMap version 38 2024-04-26T01:49:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.448+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56546c298240 con 0x56546c217140 2024-04-26T01:49:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.449+0000 7fc991600640 1 -- 172.21.15.28:0/496517428 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9940c0fd0 con 0x56546c217140 2024-04-26T01:49:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.449+0000 7fc991600640 1 -- 172.21.15.28:0/496517428 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fc9940bacc0 con 0x56546c217140 2024-04-26T01:49:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.449+0000 7fc9b199f740 10 client.25982 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:49:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.449+0000 7fc9b199f740 10 client.25982 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.449+0000 7fc9b199f740 20 client.25982 populate_metadata read hostname 'smithi028' 2024-04-26T01:49:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.449+0000 7fc9b199f740 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56546c298840 0x56546c29ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.449+0000 7fc9b199f740 4 client.25982 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.449+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56546c18a320 con 0x56546c298840 2024-04-26T01:49:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.450+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56546c298840 0x56546c29ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.453+0000 7fc9a7e00640 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56546c298840 0x56546c29ac60 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94fe760ec7fa31b6 server_cookie=a805418ac2118b12 in_seq=0 out_seq=0 2024-04-26T01:49:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.453+0000 7fc991600640 10 client.25982 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc991600640 1 -- 172.21.15.28:0/496517428 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56546c18a320 con 0x56546c298840 2024-04-26T01:49:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc991600640 10 client.25982 handle_command_reply: tid=0 2024-04-26T01:49:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 client.25982 shutdown 2024-04-26T01:49:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 msgr2=0x56546c290fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c290fb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc9940037f0 tx=0x7fc9940b8fa0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 shutdown_connections 2024-04-26T01:49:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56546c298840 0x56546c29ac60 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56546c221a00 0x56546c226d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56546c217140 0x56546c290fb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 --2- 172.21.15.28:0/496517428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56546c216820 0x56546c290a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 >> 172.21.15.28:0/496517428 conn(0x56546c212d30 msgr2=0x56546c21e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.466+0000 7fc991600640 0 client.25982 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.467+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 shutdown_connections 2024-04-26T01:49:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.467+0000 7fc9b199f740 1 -- 172.21.15.28:0/496517428 wait complete. 2024-04-26T01:49:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.467+0000 7fc9b199f740 20 client.25982 trim_cache size 0 max 16384 2024-04-26T01:49:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:49:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:49:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:49:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:49:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.468+0000 7fc9aec00640 1 -- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc98c0a4040 msgr2=0x7fc98c0a6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.468+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc98c0a4040 0x7fc98c0a6500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009fd0 tx=0x7fc9a001fb80 comp rx=0 tx=0).stop 2024-04-26T01:49:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.468+0000 7fc9aec00640 1 -- 172.21.15.28:0/1431985511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a8109870 msgr2=0x7fc9a81a3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.468+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a8109870 0x7fc9a81a3520 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc99402aae0 tx=0x7fc994005e80 comp rx=0 tx=0).stop 2024-04-26T01:49:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.469+0000 7fc9aec00640 1 -- 172.21.15.28:0/1431985511 shutdown_connections 2024-04-26T01:49:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.469+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc98c0a4040 0x7fc98c0a6500 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.469+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a81150a0 0x7fc9a81a78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.469+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a8109870 0x7fc9a81a3520 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.469+0000 7fc9aec00640 1 --2- 172.21.15.28:0/1431985511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9a8108f50 0x7fc9a81a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.469+0000 7fc9aec00640 1 -- 172.21.15.28:0/1431985511 >> 172.21.15.28:0/1431985511 conn(0x7fc9a806d020 msgr2=0x7fc9a8070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.469+0000 7fc9aec00640 1 -- 172.21.15.28:0/1431985511 shutdown_connections 2024-04-26T01:49:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:34.470+0000 7fc9aec00640 1 -- 172.21.15.28:0/1431985511 wait complete. 2024-04-26T01:49:35.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:35 smithi053 ceph-mon[32593]: pgmap v267: 97 pgs: 97 active+clean; 3.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 18 MiB/s wr, 1.27k op/s 2024-04-26T01:49:35.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:35 smithi028 ceph-mon[28247]: pgmap v267: 97 pgs: 97 active+clean; 3.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 18 MiB/s wr, 1.27k op/s 2024-04-26T01:49:35.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:35 smithi169 ceph-mon[32851]: pgmap v267: 97 pgs: 97 active+clean; 3.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 18 MiB/s wr, 1.27k op/s 2024-04-26T01:49:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 460.0 s, 2075.2 tps, lat 48.252 ms stddev 111.757 2024-04-26T01:49:37.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:37 smithi053 ceph-mon[32593]: pgmap v268: 97 pgs: 97 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 29 MiB/s wr, 1.88k op/s 2024-04-26T01:49:37.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:37 smithi028 ceph-mon[28247]: pgmap v268: 97 pgs: 97 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 29 MiB/s wr, 1.88k op/s 2024-04-26T01:49:37.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:37 smithi169 ceph-mon[32851]: pgmap v268: 97 pgs: 97 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 29 MiB/s wr, 1.88k op/s 2024-04-26T01:49:39.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:39 smithi053 ceph-mon[32593]: pgmap v269: 97 pgs: 97 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 1.64k op/s 2024-04-26T01:49:39.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:39 smithi028 ceph-mon[28247]: pgmap v269: 97 pgs: 97 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 1.64k op/s 2024-04-26T01:49:39.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:39 smithi169 ceph-mon[32851]: pgmap v269: 97 pgs: 97 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 1.64k op/s 2024-04-26T01:49:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 465.0 s, 2843.0 tps, lat 35.171 ms stddev 40.188 2024-04-26T01:49:41.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:41 smithi028 ceph-mon[28247]: pgmap v270: 97 pgs: 97 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 MiB/s wr, 1.95k op/s 2024-04-26T01:49:41.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:41 smithi169 ceph-mon[32851]: pgmap v270: 97 pgs: 97 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 MiB/s wr, 1.95k op/s 2024-04-26T01:49:41.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:41 smithi053 ceph-mon[32593]: pgmap v270: 97 pgs: 97 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 MiB/s wr, 1.95k op/s 2024-04-26T01:49:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:43 smithi028 ceph-mon[28247]: pgmap v271: 97 pgs: 97 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-26T01:49:43.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:43 smithi169 ceph-mon[32851]: pgmap v271: 97 pgs: 97 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-26T01:49:43.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:43 smithi053 ceph-mon[32593]: pgmap v271: 97 pgs: 97 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-26T01:49:45.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:45 smithi028 ceph-mon[28247]: pgmap v272: 97 pgs: 97 active+clean; 3.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 MiB/s wr, 1.75k op/s 2024-04-26T01:49:45.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:45 smithi169 ceph-mon[32851]: pgmap v272: 97 pgs: 97 active+clean; 3.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 MiB/s wr, 1.75k op/s 2024-04-26T01:49:45.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:45 smithi053 ceph-mon[32593]: pgmap v272: 97 pgs: 97 active+clean; 3.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 MiB/s wr, 1.75k op/s 2024-04-26T01:49:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 470.0 s, 2925.2 tps, lat 34.186 ms stddev 41.050 2024-04-26T01:49:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.659+0000 7fb420200640 1 -- 172.21.15.28:0/3179732806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb418108f20 msgr2=0x7fb4181153e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.660+0000 7fb420200640 1 --2- 172.21.15.28:0/3179732806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb418108f20 0x7fb4181153e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb410009cd0 tx=0x7fb41002a910 comp rx=0 tx=0).stop 2024-04-26T01:49:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.660+0000 7fb420200640 1 -- 172.21.15.28:0/3179732806 shutdown_connections 2024-04-26T01:49:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.660+0000 7fb420200640 1 --2- 172.21.15.28:0/3179732806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb418108f20 0x7fb4181153e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.660+0000 7fb420200640 1 --2- 172.21.15.28:0/3179732806 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb418108560 0x7fb4181089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.660+0000 7fb420200640 1 --2- 172.21.15.28:0/3179732806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb41810e620 0x7fb41810ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.660+0000 7fb420200640 1 -- 172.21.15.28:0/3179732806 >> 172.21.15.28:0/3179732806 conn(0x7fb4180764e0 msgr2=0x7fb418078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.661+0000 7fb420200640 1 -- 172.21.15.28:0/3179732806 shutdown_connections 2024-04-26T01:49:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.661+0000 7fb420200640 1 -- 172.21.15.28:0/3179732806 wait complete. 2024-04-26T01:49:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.662+0000 7fb420200640 1 Processor -- start 2024-04-26T01:49:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.663+0000 7fb420200640 1 -- start start 2024-04-26T01:49:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.663+0000 7fb420200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb418108560 0x7fb4181a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.663+0000 7fb420200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb418108f20 0x7fb4181a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.664+0000 7fb420200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb41810e620 0x7fb41819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.664+0000 7fb420200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb418118790 con 0x7fb418108560 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.664+0000 7fb420200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb418118610 con 0x7fb41810e620 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.664+0000 7fb420200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb418118910 con 0x7fb418108f20 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.664+0000 7fb41e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb41810e620 0x7fb41819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.664+0000 7fb41e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb41810e620 0x7fb41819ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54464/0 (socket says 172.21.15.28:54464) 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.664+0000 7fb41e200640 1 -- 172.21.15.28:0/80889804 learned_addr learned my addr 172.21.15.28:0/80889804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.664+0000 7fb41ce00640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb418108f20 0x7fb4181a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.665+0000 7fb41d800640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb418108560 0x7fb4181a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.665+0000 7fb41e200640 1 -- 172.21.15.28:0/80889804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb418108f20 msgr2=0x7fb4181a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.665+0000 7fb41e200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb418108f20 0x7fb4181a4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.665+0000 7fb41e200640 1 -- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb418108560 msgr2=0x7fb4181a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.665+0000 7fb41e200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb418108560 0x7fb4181a4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.665+0000 7fb41e200640 1 -- 172.21.15.28:0/80889804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41819f180 con 0x7fb41810e620 2024-04-26T01:49:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.666+0000 7fb41d800640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb418108560 0x7fb4181a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:49:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.666+0000 7fb41e200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb41810e620 0x7fb41819ea30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb41002f550 tx=0x7fb4100317d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.667+0000 7fb416000640 1 -- 172.21.15.28:0/80889804 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410003b70 con 0x7fb41810e620 2024-04-26T01:49:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.667+0000 7fb416000640 1 -- 172.21.15.28:0/80889804 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb410003d10 con 0x7fb41810e620 2024-04-26T01:49:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.667+0000 7fb416000640 1 -- 172.21.15.28:0/80889804 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41002f9a0 con 0x7fb41810e620 2024-04-26T01:49:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.667+0000 7fb420200640 1 -- 172.21.15.28:0/80889804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41819f410 con 0x7fb41810e620 2024-04-26T01:49:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.668+0000 7fb420200640 1 -- 172.21.15.28:0/80889804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41806b970 con 0x7fb41810e620 2024-04-26T01:49:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.673+0000 7fb416000640 1 -- 172.21.15.28:0/80889804 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb41002fb40 con 0x7fb41810e620 2024-04-26T01:49:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.675+0000 7fb4231af740 1 Processor -- start 2024-04-26T01:49:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb416000640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb4000a3f70 0x7fb4000a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb4231af740 1 -- start start 2024-04-26T01:49:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb416000640 1 -- 172.21.15.28:0/80889804 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fb4100f19a0 con 0x7fb41810e620 2024-04-26T01:49:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb41ce00640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb4000a3f70 0x7fb4000a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb4231af740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43557820 0x55cd43557c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb4231af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43558140 0x55cd435624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb4231af740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43562a00 0x55cd43564df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb4231af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd434bd900 con 0x55cd43558140 2024-04-26T01:49:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb4231af740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55cd43463ef0 con 0x55cd43562a00 2024-04-26T01:49:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb4231af740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55cd434b98c0 con 0x55cd43557820 2024-04-26T01:49:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.676+0000 7fb41e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43562a00 0x55cd43564df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43562a00 0x55cd43564df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54474/0 (socket says 172.21.15.28:54474) 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41d800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43557820 0x55cd43557c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41e200640 1 -- 172.21.15.28:0/4157259232 learned_addr learned my addr 172.21.15.28:0/4157259232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41ce00640 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43558140 0x55cd435624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41d800640 1 -- 172.21.15.28:0/4157259232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43562a00 msgr2=0x55cd43564df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41d800640 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43562a00 0x55cd43564df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41d800640 1 -- 172.21.15.28:0/4157259232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43558140 msgr2=0x55cd435624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41d800640 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43558140 0x55cd435624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41d800640 1 -- 172.21.15.28:0/4157259232 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd431d0210 con 0x55cd43557820 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41ce00640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb4000a3f70 0x7fb4000a6430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb41810dc40 tx=0x7fb404009830 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41e200640 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43562a00 0x55cd43564df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.677+0000 7fb41d800640 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43557820 0x55cd43557c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009c30 tx=0x7fb40c010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61ec69da4882d3b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.678+0000 7fb40aa00640 1 -- 172.21.15.28:0/4157259232 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c004020 con 0x55cd43557820 2024-04-26T01:49:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.678+0000 7fb40aa00640 1 -- 172.21.15.28:0/4157259232 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb40c0041c0 con 0x55cd43557820 2024-04-26T01:49:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.678+0000 7fb40aa00640 1 -- 172.21.15.28:0/4157259232 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0126b0 con 0x55cd43557820 2024-04-26T01:49:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.679+0000 7fb4231af740 1 -- 172.21.15.28:0/4157259232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43557820 msgr2=0x55cd43557c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.679+0000 7fb4231af740 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43557820 0x55cd43557c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009c30 tx=0x7fb40c010740 comp rx=0 tx=0).stop 2024-04-26T01:49:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.679+0000 7fb4231af740 1 -- 172.21.15.28:0/4157259232 shutdown_connections 2024-04-26T01:49:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.679+0000 7fb4231af740 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43562a00 0x55cd43564df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.679+0000 7fb4231af740 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43558140 0x55cd435624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.679+0000 7fb4231af740 1 --2- 172.21.15.28:0/4157259232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43557820 0x55cd43557c00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.679+0000 7fb4231af740 1 -- 172.21.15.28:0/4157259232 >> 172.21.15.28:0/4157259232 conn(0x55cd43553d30 msgr2=0x55cd43554110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.680+0000 7fb4231af740 1 -- 172.21.15.28:0/4157259232 shutdown_connections 2024-04-26T01:49:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.680+0000 7fb4231af740 1 -- 172.21.15.28:0/4157259232 wait complete. 2024-04-26T01:49:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.681+0000 7fb4231af740 1 -- start start 2024-04-26T01:49:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.681+0000 7fb4231af740 1 Processor -- start 2024-04-26T01:49:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.682+0000 7fb4231af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4355fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.682+0000 7fb4231af740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4355ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.682+0000 7fb4231af740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd43560490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.682+0000 7fb4231af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd43565630 con 0x55cd43557820 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.682+0000 7fb4231af740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55cd43477310 con 0x55cd43558140 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.682+0000 7fb4231af740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55cd435657b0 con 0x55cd43562a00 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.682+0000 7fb4231af740 10 client.0 _ll_register_callbacks cb 0x55cd43445cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4355ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4355fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4355ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54480/0 (socket says 172.21.15.28:54480) 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4355fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44632/0 (socket says 172.21.15.28:44632) 2024-04-26T01:49:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41ce00640 1 -- 172.21.15.28:0/3052036201 learned_addr learned my addr 172.21.15.28:0/3052036201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41e200640 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd43560490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41d800640 1 -- 172.21.15.28:0/3052036201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 msgr2=0x55cd43560490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41d800640 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd43560490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41d800640 1 -- 172.21.15.28:0/3052036201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 msgr2=0x55cd4355ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41d800640 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4355ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41d800640 1 -- 172.21.15.28:0/3052036201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd4329c620 con 0x55cd43557820 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.683+0000 7fb41ce00640 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4355ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.684+0000 7fb41d800640 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4355fa10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb40c000c00 tx=0x7fb40c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1746ee08065318a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.684+0000 7fb3ff400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:49:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.684+0000 7fb3ff400640 1 -- 172.21.15.28:0/3052036201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c02a060 con 0x55cd43557820 2024-04-26T01:49:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.684+0000 7fb3ff400640 1 -- 172.21.15.28:0/3052036201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fb40c009770 con 0x55cd43557820 2024-04-26T01:49:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.684+0000 7fb4231af740 1 -- 172.21.15.28:0/3052036201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb40c0041c0 con 0x55cd43557820 2024-04-26T01:49:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.685+0000 7fb3ff400640 1 -- 172.21.15.28:0/3052036201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb40c0035f0 con 0x55cd43557820 2024-04-26T01:49:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.685+0000 7fb4231af740 10 client.15795 fetch_fsmap learned FSMap version 38 2024-04-26T01:49:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.685+0000 7fb4231af740 1 -- 172.21.15.28:0/3052036201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55cd43571920 con 0x55cd43557820 2024-04-26T01:49:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.686+0000 7fb3ff400640 1 -- 172.21.15.28:0/3052036201 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb40c0041c0 con 0x55cd43557820 2024-04-26T01:49:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.686+0000 7fb3ff400640 1 -- 172.21.15.28:0/3052036201 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fb40c00f040 con 0x55cd43557820 2024-04-26T01:49:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.687+0000 7fb4231af740 10 client.15795 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:49:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.687+0000 7fb4231af740 10 client.15795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.687+0000 7fb4231af740 20 client.15795 populate_metadata read hostname 'smithi028' 2024-04-26T01:49:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.687+0000 7fb4231af740 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55cd43572d20 0x55cd43575110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.687+0000 7fb4231af740 4 client.15795 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.687+0000 7fb4231af740 1 -- 172.21.15.28:0/3052036201 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd434d18b0 con 0x55cd43572d20 2024-04-26T01:49:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.687+0000 7fb41ce00640 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55cd43572d20 0x55cd43575110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.689+0000 7fb41ce00640 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55cd43572d20 0x55cd43575110 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7fd7a7cc4f6dd96 server_cookie=c052db57fe34838e in_seq=0 out_seq=0 2024-04-26T01:49:47.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.689+0000 7fb3ff400640 10 client.15795 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:47.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.690+0000 7fb3ff400640 1 -- 172.21.15.28:0/3052036201 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd434d18b0 con 0x55cd43572d20 2024-04-26T01:49:47.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.690+0000 7fb3ff400640 10 client.15795 handle_command_reply: tid=0 2024-04-26T01:49:47.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.690+0000 7fb4231af740 1 client.15795 shutdown 2024-04-26T01:49:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 -- 172.21.15.28:0/3052036201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 msgr2=0x55cd4355fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4355fa10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb40c000c00 tx=0x7fb40c005dc0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 -- 172.21.15.28:0/3052036201 shutdown_connections 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55cd43572d20 0x55cd43575110 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd43560490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4355ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb3ff400640 0 client.15795 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 --2- 172.21.15.28:0/3052036201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4355fa10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 -- 172.21.15.28:0/3052036201 >> 172.21.15.28:0/3052036201 conn(0x55cd43553d30 msgr2=0x55cd43559ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.693+0000 7fb4231af740 1 -- 172.21.15.28:0/3052036201 shutdown_connections 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.694+0000 7fb4231af740 1 -- 172.21.15.28:0/3052036201 wait complete. 2024-04-26T01:49:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.694+0000 7fb4231af740 20 client.15795 trim_cache size 0 max 16384 2024-04-26T01:49:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.725+0000 7fb4231af740 1 Processor -- start 2024-04-26T01:49:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.725+0000 7fb4231af740 1 -- start start 2024-04-26T01:49:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.726+0000 7fb4231af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4356e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.726+0000 7fb4231af740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4357a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.726+0000 7fb4231af740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd4357ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.726+0000 7fb4231af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4181bf290 con 0x55cd43557820 2024-04-26T01:49:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.726+0000 7fb4231af740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb4181bcb60 con 0x55cd43558140 2024-04-26T01:49:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.726+0000 7fb4231af740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb418072400 con 0x55cd43562a00 2024-04-26T01:49:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4356e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41e200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd4357ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41e200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd4357ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:44500/0 (socket says 172.21.15.28:44500) 2024-04-26T01:49:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41e200640 1 -- 172.21.15.28:0/4118497919 learned_addr learned my addr 172.21.15.28:0/4118497919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41e200640 1 -- 172.21.15.28:0/4118497919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 msgr2=0x55cd4357a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41e200640 1 --2- 172.21.15.28:0/4118497919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4357a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41e200640 1 -- 172.21.15.28:0/4118497919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 msgr2=0x55cd4356e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41e200640 1 --2- 172.21.15.28:0/4118497919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4356e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.727+0000 7fb41e200640 1 -- 172.21.15.28:0/4118497919 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd43565160 con 0x55cd43562a00 2024-04-26T01:49:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.728+0000 7fb41d800640 1 --2- 172.21.15.28:0/4118497919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4356e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.728+0000 7fb41e200640 1 --2- 172.21.15.28:0/4118497919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd4357ade0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb4100b9eb0 tx=0x7fb410004820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae5ef837007aa4f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.728+0000 7fb40a000640 1 -- 172.21.15.28:0/4118497919 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4100be150 con 0x55cd43562a00 2024-04-26T01:49:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.729+0000 7fb40a000640 1 -- 172.21.15.28:0/4118497919 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4100d6020 con 0x55cd43562a00 2024-04-26T01:49:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.729+0000 7fb40a000640 1 -- 172.21.15.28:0/4118497919 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4100ef370 con 0x55cd43562a00 2024-04-26T01:49:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.730+0000 7fb4231af740 1 -- 172.21.15.28:0/4118497919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 msgr2=0x55cd4357ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.730+0000 7fb4231af740 1 --2- 172.21.15.28:0/4118497919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd4357ade0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb4100b9eb0 tx=0x7fb410004820 comp rx=0 tx=0).stop 2024-04-26T01:49:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.731+0000 7fb4231af740 1 -- 172.21.15.28:0/4118497919 shutdown_connections 2024-04-26T01:49:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.731+0000 7fb4231af740 1 --2- 172.21.15.28:0/4118497919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43562a00 0x55cd4357ade0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.731+0000 7fb4231af740 1 --2- 172.21.15.28:0/4118497919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43558140 0x55cd4357a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.731+0000 7fb4231af740 1 --2- 172.21.15.28:0/4118497919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43557820 0x55cd4356e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.731+0000 7fb4231af740 1 -- 172.21.15.28:0/4118497919 >> 172.21.15.28:0/4118497919 conn(0x55cd43553d30 msgr2=0x55cd4354edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.732+0000 7fb4231af740 1 -- 172.21.15.28:0/4118497919 shutdown_connections 2024-04-26T01:49:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.732+0000 7fb4231af740 1 -- 172.21.15.28:0/4118497919 wait complete. 2024-04-26T01:49:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.733+0000 7fb4231af740 1 -- start start 2024-04-26T01:49:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.733+0000 7fb4231af740 1 Processor -- start 2024-04-26T01:49:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb4231af740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43557820 0x55cd4356adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb4231af740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43558140 0x55cd435d1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb4231af740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43562a00 0x55cd435d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb4231af740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb418072a50 con 0x55cd43562a00 2024-04-26T01:49:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb4231af740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb418073ec0 con 0x55cd43557820 2024-04-26T01:49:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb4231af740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb418073120 con 0x55cd43558140 2024-04-26T01:49:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb4231af740 10 client.0 _ll_register_callbacks cb 0x7fb40c009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb41d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43557820 0x55cd4356adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.734+0000 7fb41d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43557820 0x55cd4356adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54494/0 (socket says 172.21.15.28:54494) 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.735+0000 7fb41d800640 1 -- 172.21.15.28:0/3440031124 learned_addr learned my addr 172.21.15.28:0/3440031124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.735+0000 7fb41e200640 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43562a00 0x55cd435d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.735+0000 7fb41ce00640 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43558140 0x55cd435d1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.735+0000 7fb41ce00640 1 -- 172.21.15.28:0/3440031124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43557820 msgr2=0x55cd4356adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.735+0000 7fb41ce00640 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43557820 0x55cd4356adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.735+0000 7fb41ce00640 1 -- 172.21.15.28:0/3440031124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43562a00 msgr2=0x55cd435d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.735+0000 7fb41ce00640 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43562a00 0x55cd435d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.735+0000 7fb41ce00640 1 -- 172.21.15.28:0/3440031124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd435d2610 con 0x55cd43558140 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.736+0000 7fb41ce00640 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43558140 0x55cd435d1880 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x55cd435757e0 tx=0x7fb4040046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76958ae24d4dcedb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.736+0000 7fb3ff400640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:49:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.739+0000 7fb3ff400640 1 -- 172.21.15.28:0/3440031124 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40401de50 con 0x55cd43558140 2024-04-26T01:49:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.740+0000 7fb3ff400640 1 -- 172.21.15.28:0/3440031124 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fb404004450 con 0x55cd43558140 2024-04-26T01:49:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.741+0000 7fb4231af740 1 -- 172.21.15.28:0/3440031124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4100d6020 con 0x55cd43558140 2024-04-26T01:49:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.741+0000 7fb3ff400640 1 -- 172.21.15.28:0/3440031124 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb4040120d0 con 0x55cd43558140 2024-04-26T01:49:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.741+0000 7fb4231af740 10 client.26011 fetch_fsmap learned FSMap version 38 2024-04-26T01:49:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.741+0000 7fb4231af740 1 -- 172.21.15.28:0/3440031124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55cd435ea240 con 0x55cd43558140 2024-04-26T01:49:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.743+0000 7fb3ff400640 1 -- 172.21.15.28:0/3440031124 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4100d6020 con 0x55cd43558140 2024-04-26T01:49:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.743+0000 7fb3ff400640 1 -- 172.21.15.28:0/3440031124 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fb40c009d20 con 0x55cd43558140 2024-04-26T01:49:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.744+0000 7fb4231af740 10 client.26011 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:49:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.744+0000 7fb4231af740 10 client.26011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.744+0000 7fb4231af740 20 client.26011 populate_metadata read hostname 'smithi028' 2024-04-26T01:49:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.744+0000 7fb4231af740 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55cd435eaa10 0x55cd435ece30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.744+0000 7fb4231af740 4 client.26011 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.744+0000 7fb4231af740 1 -- 172.21.15.28:0/3440031124 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd43553af0 con 0x55cd435eaa10 2024-04-26T01:49:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.744+0000 7fb41d800640 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55cd435eaa10 0x55cd435ece30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.746+0000 7fb41d800640 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55cd435eaa10 0x55cd435ece30 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a21924e3b92d2a6 server_cookie=420fb617a2b9f3a7 in_seq=0 out_seq=0 2024-04-26T01:49:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.749+0000 7fb3ff400640 10 client.26011 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:47.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:47 smithi028 ceph-mon[28247]: pgmap v273: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 19 MiB/s wr, 1.35k op/s 2024-04-26T01:49:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.788+0000 7fb3ff400640 1 -- 172.21.15.28:0/3440031124 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd43553af0 con 0x55cd435eaa10 2024-04-26T01:49:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb3ff400640 10 client.26011 handle_command_reply: tid=0 2024-04-26T01:49:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 client.26011 shutdown 2024-04-26T01:49:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 -- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43558140 msgr2=0x55cd435d1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43558140 0x55cd435d1880 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x55cd435757e0 tx=0x7fb4040046c0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 -- 172.21.15.28:0/3440031124 shutdown_connections 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55cd435eaa10 0x55cd435ece30 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd43562a00 0x55cd435d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd43558140 0x55cd435d1880 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 --2- 172.21.15.28:0/3440031124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55cd43557820 0x55cd4356adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.789+0000 7fb4231af740 1 -- 172.21.15.28:0/3440031124 >> 172.21.15.28:0/3440031124 conn(0x55cd43553d30 msgr2=0x55cd435591a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.790+0000 7fb3ff400640 0 client.26011 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.790+0000 7fb4231af740 1 -- 172.21.15.28:0/3440031124 shutdown_connections 2024-04-26T01:49:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.790+0000 7fb4231af740 1 -- 172.21.15.28:0/3440031124 wait complete. 2024-04-26T01:49:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.790+0000 7fb4231af740 20 client.26011 trim_cache size 0 max 16384 2024-04-26T01:49:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:49:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:49:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:49:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:49:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.791+0000 7fb420200640 1 -- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb4000a3f70 msgr2=0x7fb4000a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.791+0000 7fb420200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb4000a3f70 0x7fb4000a6430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb41810dc40 tx=0x7fb404009830 comp rx=0 tx=0).stop 2024-04-26T01:49:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 -- 172.21.15.28:0/80889804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb41810e620 msgr2=0x7fb41819ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb41810e620 0x7fb41819ea30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb41002f550 tx=0x7fb4100317d0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 -- 172.21.15.28:0/80889804 shutdown_connections 2024-04-26T01:49:47.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb4000a3f70 0x7fb4000a6430 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb41810e620 0x7fb41819ea30 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb418108f20 0x7fb4181a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 --2- 172.21.15.28:0/80889804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb418108560 0x7fb4181a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 -- 172.21.15.28:0/80889804 >> 172.21.15.28:0/80889804 conn(0x7fb4180764e0 msgr2=0x7fb418077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.792+0000 7fb420200640 1 -- 172.21.15.28:0/80889804 shutdown_connections 2024-04-26T01:49:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:49:47.793+0000 7fb420200640 1 -- 172.21.15.28:0/80889804 wait complete. 2024-04-26T01:49:47.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:47 smithi169 ceph-mon[32851]: pgmap v273: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 19 MiB/s wr, 1.35k op/s 2024-04-26T01:49:47.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:47 smithi053 ceph-mon[32593]: pgmap v273: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 19 MiB/s wr, 1.35k op/s 2024-04-26T01:49:49.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:49 smithi028 ceph-mon[28247]: pgmap v274: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.0 MiB/s wr, 587 op/s 2024-04-26T01:49:49.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:49 smithi169 ceph-mon[32851]: pgmap v274: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.0 MiB/s wr, 587 op/s 2024-04-26T01:49:49.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:49 smithi053 ceph-mon[32593]: pgmap v274: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.0 MiB/s wr, 587 op/s 2024-04-26T01:49:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 475.0 s, 2303.8 tps, lat 43.397 ms stddev 49.066 2024-04-26T01:49:51.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:51 smithi028 ceph-mon[28247]: pgmap v275: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 9.2 MiB/s wr, 898 op/s 2024-04-26T01:49:51.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:51 smithi169 ceph-mon[32851]: pgmap v275: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 9.2 MiB/s wr, 898 op/s 2024-04-26T01:49:51.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:51 smithi053 ceph-mon[32593]: pgmap v275: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 9.2 MiB/s wr, 898 op/s 2024-04-26T01:49:52.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:52 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:52.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:52 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:52.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:52 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:53.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:53 smithi028 ceph-mon[28247]: pgmap v276: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.0 MiB/s wr, 593 op/s 2024-04-26T01:49:53.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:53 smithi169 ceph-mon[32851]: pgmap v276: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.0 MiB/s wr, 593 op/s 2024-04-26T01:49:53.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:53 smithi053 ceph-mon[32593]: pgmap v276: 97 pgs: 97 active+clean; 3.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.0 MiB/s wr, 593 op/s 2024-04-26T01:49:55.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:55 smithi028 ceph-mon[28247]: pgmap v277: 97 pgs: 97 active+clean; 3.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.0 MiB/s wr, 680 op/s 2024-04-26T01:49:55.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:55 smithi169 ceph-mon[32851]: pgmap v277: 97 pgs: 97 active+clean; 3.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.0 MiB/s wr, 680 op/s 2024-04-26T01:49:55.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:55 smithi053 ceph-mon[32593]: pgmap v277: 97 pgs: 97 active+clean; 3.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.0 MiB/s wr, 680 op/s 2024-04-26T01:49:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 480.0 s, 2414.2 tps, lat 41.368 ms stddev 50.565 2024-04-26T01:49:57.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:57 smithi028 ceph-mon[28247]: pgmap v278: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.5 MiB/s wr, 830 op/s 2024-04-26T01:49:57.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:57 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:57.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:57 smithi169 ceph-mon[32851]: pgmap v278: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.5 MiB/s wr, 830 op/s 2024-04-26T01:49:57.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:57 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:57.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:57 smithi053 ceph-mon[32593]: pgmap v278: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.5 MiB/s wr, 830 op/s 2024-04-26T01:49:57.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:57 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:59.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:49:59 smithi169 ceph-mon[32851]: pgmap v279: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.4 MiB/s wr, 642 op/s 2024-04-26T01:49:59.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:49:59 smithi053 ceph-mon[32593]: pgmap v279: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.4 MiB/s wr, 642 op/s 2024-04-26T01:50:00.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:49:59 smithi028 ceph-mon[28247]: pgmap v279: 97 pgs: 97 active+clean; 3.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.4 MiB/s wr, 642 op/s 2024-04-26T01:50:00.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:00 smithi169 ceph-mon[32851]: overall HEALTH_OK 2024-04-26T01:50:00.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:00 smithi053 ceph-mon[32593]: overall HEALTH_OK 2024-04-26T01:50:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 -- 172.21.15.28:0/1903201696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c102b50 msgr2=0x7fdd7c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 --2- 172.21.15.28:0/1903201696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c102b50 0x7fdd7c106b90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdd68009860 tx=0x7fdd6802a340 comp rx=0 tx=0).stop 2024-04-26T01:50:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 -- 172.21.15.28:0/1903201696 shutdown_connections 2024-04-26T01:50:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 --2- 172.21.15.28:0/1903201696 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd7c107270 0x7fdd7c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 --2- 172.21.15.28:0/1903201696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c102b50 0x7fdd7c106b90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 --2- 172.21.15.28:0/1903201696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c1021a0 0x7fdd7c102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 -- 172.21.15.28:0/1903201696 >> 172.21.15.28:0/1903201696 conn(0x7fdd7c0fda20 msgr2=0x7fdd7c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 -- 172.21.15.28:0/1903201696 shutdown_connections 2024-04-26T01:50:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.993+0000 7fdd80e00640 1 -- 172.21.15.28:0/1903201696 wait complete. 2024-04-26T01:50:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.994+0000 7fdd80e00640 1 Processor -- start 2024-04-26T01:50:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.994+0000 7fdd80e00640 1 -- start start 2024-04-26T01:50:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd80e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c1021a0 0x7fdd7c1164e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd80e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd7c107270 0x7fdd7c116a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd80e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c111640 0x7fdd7c111af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd80e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c10ffd0 con 0x7fdd7c111640 2024-04-26T01:50:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd80e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c10fe50 con 0x7fdd7c1021a0 2024-04-26T01:50:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd80e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c110150 con 0x7fdd7c107270 2024-04-26T01:50:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd79e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c1021a0 0x7fdd7c1164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c111640 0x7fdd7c111af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd79400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd7c107270 0x7fdd7c116a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd79e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c1021a0 0x7fdd7c1164e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50720/0 (socket says 172.21.15.28:50720) 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.995+0000 7fdd79e00640 1 -- 172.21.15.28:0/326762103 learned_addr learned my addr 172.21.15.28:0/326762103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd79e00640 1 -- 172.21.15.28:0/326762103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd7c107270 msgr2=0x7fdd7c116a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd79e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd7c107270 0x7fdd7c116a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd79e00640 1 -- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c111640 msgr2=0x7fdd7c111af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd79e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c111640 0x7fdd7c111af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd79e00640 1 -- 172.21.15.28:0/326762103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7c112030 con 0x7fdd7c1021a0 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd79e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c1021a0 0x7fdd7c1164e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00e9c0 tx=0x7fdd6c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd72a00640 1 -- 172.21.15.28:0/326762103 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c00ccc0 con 0x7fdd7c1021a0 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd72a00640 1 -- 172.21.15.28:0/326762103 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd6c0054d0 con 0x7fdd7c1021a0 2024-04-26T01:50:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.996+0000 7fdd72a00640 1 -- 172.21.15.28:0/326762103 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c010430 con 0x7fdd7c1021a0 2024-04-26T01:50:00.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.997+0000 7fdd80e00640 1 -- 172.21.15.28:0/326762103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd7c112320 con 0x7fdd7c1021a0 2024-04-26T01:50:00.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:00.997+0000 7fdd80e00640 1 -- 172.21.15.28:0/326762103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd7c1b44c0 con 0x7fdd7c1021a0 2024-04-26T01:50:01.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.003+0000 7fdd72a00640 1 -- 172.21.15.28:0/326762103 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdd6c004f80 con 0x7fdd7c1021a0 2024-04-26T01:50:01.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.003+0000 7fdd83c16740 1 Processor -- start 2024-04-26T01:50:01.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.003+0000 7fdd83c16740 1 -- start start 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ec820 0x5651d32ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32f7a00 0x5651d32f9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd83c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651d3252900 con 0x5651d32ec820 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd83c16740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5651d31f8ef0 con 0x5651d32ed140 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd83c16740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5651d324e8c0 con 0x5651d32f7a00 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd79e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ec820 0x5651d32ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd79400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32f7a00 0x5651d32f9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd79400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32f7a00 0x5651d32f9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:50388/0 (socket says 172.21.15.28:50388) 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.004+0000 7fdd79400640 1 -- 172.21.15.28:0/2041075450 learned_addr learned my addr 172.21.15.28:0/2041075450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.005+0000 7fdd79e00640 1 -- 172.21.15.28:0/2041075450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32f7a00 msgr2=0x5651d32f9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.005+0000 7fdd79e00640 1 --2- 172.21.15.28:0/2041075450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32f7a00 0x5651d32f9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.005+0000 7fdd79e00640 1 -- 172.21.15.28:0/2041075450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ec820 msgr2=0x5651d32ecc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.005+0000 7fdd79e00640 1 --2- 172.21.15.28:0/2041075450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ec820 0x5651d32ecc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.005+0000 7fdd79e00640 1 -- 172.21.15.28:0/2041075450 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651d2f65210 con 0x5651d32ed140 2024-04-26T01:50:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.005+0000 7fdd72a00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd600a3f10 0x7fdd600a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.006+0000 7fdd72a00640 1 -- 172.21.15.28:0/326762103 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fdd6c0d0c90 con 0x7fdd7c1021a0 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.006+0000 7fdd79e00640 1 --2- 172.21.15.28:0/2041075450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f74c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0ce380 tx=0x7fdd6c0d96d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25ade0cede4d5123 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.006+0000 7fdd7a800640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd600a3f10 0x7fdd600a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.006+0000 7fdd67400640 1 -- 172.21.15.28:0/2041075450 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c098e70 con 0x5651d32ed140 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.006+0000 7fdd67400640 1 -- 172.21.15.28:0/2041075450 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd6c09d2d0 con 0x5651d32ed140 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.006+0000 7fdd67400640 1 -- 172.21.15.28:0/2041075450 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c09c530 con 0x5651d32ed140 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.007+0000 7fdd83c16740 1 -- 172.21.15.28:0/2041075450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 msgr2=0x5651d32f74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.007+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2041075450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f74c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0ce380 tx=0x7fdd6c0d96d0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.007+0000 7fdd83c16740 1 -- 172.21.15.28:0/2041075450 shutdown_connections 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.007+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2041075450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32f7a00 0x5651d32f9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.007+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2041075450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f74c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.007+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2041075450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ec820 0x5651d32ecc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.007+0000 7fdd83c16740 1 -- 172.21.15.28:0/2041075450 >> 172.21.15.28:0/2041075450 conn(0x5651d32e8d30 msgr2=0x5651d32e9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.008+0000 7fdd83c16740 1 -- 172.21.15.28:0/2041075450 shutdown_connections 2024-04-26T01:50:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.008+0000 7fdd7a800640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd600a3f10 0x7fdd600a63d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd740092f0 tx=0x7fdd74009280 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.008+0000 7fdd83c16740 1 -- 172.21.15.28:0/2041075450 wait complete. 2024-04-26T01:50:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.008+0000 7fdd83c16740 1 -- start start 2024-04-26T01:50:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.009+0000 7fdd83c16740 1 Processor -- start 2024-04-26T01:50:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.009+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32f4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.009+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.009+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d32f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd79e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd79e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32f4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:50402/0 (socket says 172.21.15.28:50402) 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d32f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd79400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd79400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50734/0 (socket says 172.21.15.28:50734) 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd83c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651d32fa630 con 0x5651d32f7a00 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd83c16740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5651d320c310 con 0x5651d32ed140 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd83c16740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5651d32fa7b0 con 0x5651d32ec820 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd83c16740 10 client.0 _ll_register_callbacks cb 0x5651d31dacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd79e00640 1 -- 172.21.15.28:0/2979759855 learned_addr learned my addr 172.21.15.28:0/2979759855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd7a800640 1 -- 172.21.15.28:0/2979759855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 msgr2=0x5651d32f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd7a800640 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd7a800640 1 -- 172.21.15.28:0/2979759855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 msgr2=0x5651d32f4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd7a800640 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.010+0000 7fdd7a800640 1 -- 172.21.15.28:0/2979759855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651d3031620 con 0x5651d32f7a00 2024-04-26T01:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.011+0000 7fdd79400640 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.011+0000 7fdd83c16740 1 -- 172.21.15.28:0/2979759855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd6c09d2d0 con 0x5651d32f7a00 2024-04-26T01:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.011+0000 7fdd7a800640 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d32f53d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdd740127a0 tx=0x7fdd74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d223bfe30b027d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.011+0000 7fdd5be00640 10 client.15813 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:50:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.012+0000 7fdd5be00640 1 -- 172.21.15.28:0/2979759855 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7400ea50 con 0x5651d32f7a00 2024-04-26T01:50:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.012+0000 7fdd5be00640 1 -- 172.21.15.28:0/2979759855 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fdd7400ebf0 con 0x5651d32f7a00 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.012+0000 7fdd5be00640 1 -- 172.21.15.28:0/2979759855 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdd7400ee10 con 0x5651d32f7a00 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.012+0000 7fdd83c16740 10 client.15813 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.012+0000 7fdd83c16740 1 -- 172.21.15.28:0/2979759855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=40}) -- 0x5651d3306810 con 0x5651d32f7a00 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.014+0000 7fdd5be00640 1 -- 172.21.15.28:0/2979759855 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fdd7400f040 con 0x5651d32f7a00 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.014+0000 7fdd83c16740 10 client.15813 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.014+0000 7fdd83c16740 10 client.15813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.014+0000 7fdd83c16740 20 client.15813 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.014+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5651d3307bc0 0x5651d3309fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.014+0000 7fdd83c16740 4 client.15813 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.014+0000 7fdd83c16740 1 -- 172.21.15.28:0/2979759855 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651d32668b0 con 0x5651d3307bc0 2024-04-26T01:50:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.015+0000 7fdd79e00640 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5651d3307bc0 0x5651d3309fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.015+0000 7fdd79e00640 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5651d3307bc0 0x5651d3309fd0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c06562d07ee2979c server_cookie=bbcf91d701a3fafc in_seq=0 out_seq=0 2024-04-26T01:50:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.015+0000 7fdd5be00640 10 client.15813 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.016+0000 7fdd5be00640 1 -- 172.21.15.28:0/2979759855 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651d32668b0 con 0x5651d3307bc0 2024-04-26T01:50:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.016+0000 7fdd5be00640 10 client.15813 handle_command_reply: tid=0 2024-04-26T01:50:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 client.15813 shutdown 2024-04-26T01:50:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 -- 172.21.15.28:0/2979759855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 msgr2=0x5651d32f53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d32f53d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdd740127a0 tx=0x7fdd74004a40 comp rx=0 tx=0).stop 2024-04-26T01:50:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 -- 172.21.15.28:0/2979759855 shutdown_connections 2024-04-26T01:50:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5651d3307bc0 0x5651d3309fd0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd5be00640 0 client.15813 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d32f53d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d32f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 --2- 172.21.15.28:0/2979759855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.017+0000 7fdd83c16740 1 -- 172.21.15.28:0/2979759855 >> 172.21.15.28:0/2979759855 conn(0x5651d32e8d30 msgr2=0x5651d32eeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.018+0000 7fdd83c16740 1 -- 172.21.15.28:0/2979759855 shutdown_connections 2024-04-26T01:50:01.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.020+0000 7fdd83c16740 1 -- 172.21.15.28:0/2979759855 wait complete. 2024-04-26T01:50:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.020+0000 7fdd83c16740 20 client.15813 trim_cache size 0 max 16384 2024-04-26T01:50:01.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:00 smithi028 ceph-mon[28247]: overall HEALTH_OK 2024-04-26T01:50:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.047+0000 7fdd83c16740 1 Processor -- start 2024-04-26T01:50:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.047+0000 7fdd83c16740 1 -- start start 2024-04-26T01:50:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32e8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d330f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d330fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd83c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c127260 con 0x5651d32f7a00 2024-04-26T01:50:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd83c16740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c124ad0 con 0x5651d32ed140 2024-04-26T01:50:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd83c16740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c122940 con 0x5651d32ec820 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd79400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d330f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd79400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d330f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50750/0 (socket says 172.21.15.28:50750) 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd79400640 1 -- 172.21.15.28:0/233060764 learned_addr learned my addr 172.21.15.28:0/233060764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.048+0000 7fdd79e00640 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.049+0000 7fdd7a800640 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d330fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.049+0000 7fdd79400640 1 -- 172.21.15.28:0/233060764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 msgr2=0x5651d32e8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.049+0000 7fdd79400640 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32e8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.049+0000 7fdd79400640 1 -- 172.21.15.28:0/233060764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 msgr2=0x5651d330fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.049+0000 7fdd79400640 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d330fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.049+0000 7fdd79400640 1 -- 172.21.15.28:0/233060764 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651d32f2ea0 con 0x5651d32ed140 2024-04-26T01:50:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.049+0000 7fdd7a800640 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d330fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:01.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.049+0000 7fdd79400640 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d330f8e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdd68035040 tx=0x7fdd6802a840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26cfe00197b8300b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.050+0000 7fdd66a00640 1 -- 172.21.15.28:0/233060764 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68065070 con 0x5651d32ed140 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.050+0000 7fdd66a00640 1 -- 172.21.15.28:0/233060764 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd68030900 con 0x5651d32ed140 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.050+0000 7fdd66a00640 1 -- 172.21.15.28:0/233060764 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68037040 con 0x5651d32ed140 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.051+0000 7fdd83c16740 1 -- 172.21.15.28:0/233060764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 msgr2=0x5651d330f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.051+0000 7fdd83c16740 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d330f8e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdd68035040 tx=0x7fdd6802a840 comp rx=0 tx=0).stop 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.051+0000 7fdd83c16740 1 -- 172.21.15.28:0/233060764 shutdown_connections 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.051+0000 7fdd83c16740 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32f7a00 0x5651d330fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.051+0000 7fdd83c16740 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32ed140 0x5651d330f8e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.051+0000 7fdd83c16740 1 --2- 172.21.15.28:0/233060764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.051+0000 7fdd83c16740 1 -- 172.21.15.28:0/233060764 >> 172.21.15.28:0/233060764 conn(0x5651d32e8d30 msgr2=0x5651d32eea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.052+0000 7fdd83c16740 1 -- 172.21.15.28:0/233060764 shutdown_connections 2024-04-26T01:50:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.052+0000 7fdd83c16740 1 -- 172.21.15.28:0/233060764 wait complete. 2024-04-26T01:50:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.052+0000 7fdd83c16740 1 -- start start 2024-04-26T01:50:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.053+0000 7fdd83c16740 1 Processor -- start 2024-04-26T01:50:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.053+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.053+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ed140 0x5651d32fd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.053+0000 7fdd83c16740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32f7a00 0x5651d32fd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.053+0000 7fdd83c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c122f90 con 0x5651d32ed140 2024-04-26T01:50:01.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.053+0000 7fdd83c16740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c124400 con 0x5651d32f7a00 2024-04-26T01:50:01.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.053+0000 7fdd83c16740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c123660 con 0x5651d32ec820 2024-04-26T01:50:01.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.053+0000 7fdd83c16740 10 client.0 _ll_register_callbacks cb 0x7fdd74012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.054+0000 7fdd79e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.054+0000 7fdd79e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:50420/0 (socket says 172.21.15.28:50420) 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.054+0000 7fdd79e00640 1 -- 172.21.15.28:0/251657986 learned_addr learned my addr 172.21.15.28:0/251657986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.054+0000 7fdd7a800640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32f7a00 0x5651d32fd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.054+0000 7fdd79400640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ed140 0x5651d32fd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd79e00640 1 -- 172.21.15.28:0/251657986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32f7a00 msgr2=0x5651d32fd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd79e00640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32f7a00 0x5651d32fd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd79e00640 1 -- 172.21.15.28:0/251657986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ed140 msgr2=0x5651d32fd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd79e00640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ed140 0x5651d32fd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd79e00640 1 -- 172.21.15.28:0/251657986 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651d32fde20 con 0x5651d32ec820 2024-04-26T01:50:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd7a800640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32f7a00 0x5651d32fd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd79400640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ed140 0x5651d32fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd79e00640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32ffd50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009530 tx=0x7fdd6c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6d46b202c3bb3b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.055+0000 7fdd5be00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.056+0000 7fdd5be00640 1 -- 172.21.15.28:0/251657986 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c09c710 con 0x5651d32ec820 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.056+0000 7fdd5be00640 1 -- 172.21.15.28:0/251657986 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fdd6c09aed0 con 0x5651d32ec820 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.056+0000 7fdd83c16740 1 -- 172.21.15.28:0/251657986 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd68030900 con 0x5651d32ec820 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.056+0000 7fdd5be00640 1 -- 172.21.15.28:0/251657986 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdd6c081050 con 0x5651d32ec820 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.056+0000 7fdd83c16740 10 client.26029 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.056+0000 7fdd83c16740 1 -- 172.21.15.28:0/251657986 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5651d336e100 con 0x5651d32ec820 2024-04-26T01:50:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.056+0000 7fdd5be00640 1 -- 172.21.15.28:0/251657986 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd68030900 con 0x5651d32ec820 2024-04-26T01:50:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.057+0000 7fdd5be00640 1 -- 172.21.15.28:0/251657986 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fdd6c084020 con 0x5651d32ec820 2024-04-26T01:50:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.057+0000 7fdd83c16740 10 client.26029 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.057+0000 7fdd83c16740 10 client.26029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.057+0000 7fdd83c16740 20 client.26029 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.057+0000 7fdd83c16740 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5651d3307bc0 0x5651d33703a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.057+0000 7fdd83c16740 4 client.26029 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.057+0000 7fdd83c16740 1 -- 172.21.15.28:0/251657986 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651d3260320 con 0x5651d3307bc0 2024-04-26T01:50:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.057+0000 7fdd79400640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5651d3307bc0 0x5651d33703a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.058+0000 7fdd79400640 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5651d3307bc0 0x5651d33703a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5b7b002e62de5fd server_cookie=1a67f467541df928 in_seq=0 out_seq=0 2024-04-26T01:50:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.058+0000 7fdd5be00640 10 client.26029 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd5be00640 1 -- 172.21.15.28:0/251657986 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5651d3260320 con 0x5651d3307bc0 2024-04-26T01:50:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd5be00640 10 client.26029 handle_command_reply: tid=0 2024-04-26T01:50:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 client.26029 shutdown 2024-04-26T01:50:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 -- 172.21.15.28:0/251657986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 msgr2=0x5651d32ffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32ffd50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009530 tx=0x7fdd6c009f90 comp rx=0 tx=0).stop 2024-04-26T01:50:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 -- 172.21.15.28:0/251657986 shutdown_connections 2024-04-26T01:50:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5651d3307bc0 0x5651d33703a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd5be00640 0 client.26029 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5651d32f7a00 0x5651d32fd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651d32ed140 0x5651d32fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 --2- 172.21.15.28:0/251657986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651d32ec820 0x5651d32ffd50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.073+0000 7fdd83c16740 1 -- 172.21.15.28:0/251657986 >> 172.21.15.28:0/251657986 conn(0x5651d32e8d30 msgr2=0x5651d32f6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.074+0000 7fdd83c16740 1 -- 172.21.15.28:0/251657986 shutdown_connections 2024-04-26T01:50:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.074+0000 7fdd83c16740 1 -- 172.21.15.28:0/251657986 wait complete. 2024-04-26T01:50:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.074+0000 7fdd83c16740 20 client.26029 trim_cache size 0 max 16384 2024-04-26T01:50:01.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:50:01.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:50:01.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:50:01.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:50:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.074+0000 7fdd80e00640 1 -- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd600a3f10 msgr2=0x7fdd600a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd600a3f10 0x7fdd600a63d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd740092f0 tx=0x7fdd74009280 comp rx=0 tx=0).stop 2024-04-26T01:50:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 -- 172.21.15.28:0/326762103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c1021a0 msgr2=0x7fdd7c1164e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c1021a0 0x7fdd7c1164e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00e9c0 tx=0x7fdd6c00ee80 comp rx=0 tx=0).stop 2024-04-26T01:50:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 -- 172.21.15.28:0/326762103 shutdown_connections 2024-04-26T01:50:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdd600a3f10 0x7fdd600a63d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c111640 0x7fdd7c111af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd7c107270 0x7fdd7c116a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 --2- 172.21.15.28:0/326762103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd7c1021a0 0x7fdd7c1164e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 -- 172.21.15.28:0/326762103 >> 172.21.15.28:0/326762103 conn(0x7fdd7c0fda20 msgr2=0x7fdd7c0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 -- 172.21.15.28:0/326762103 shutdown_connections 2024-04-26T01:50:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:01.075+0000 7fdd80e00640 1 -- 172.21.15.28:0/326762103 wait complete. 2024-04-26T01:50:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 485.0 s, 2283.1 tps, lat 42.637 ms stddev 50.112 2024-04-26T01:50:01.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:01 smithi169 ceph-mon[32851]: pgmap v280: 97 pgs: 97 active+clean; 3.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 10 MiB/s wr, 914 op/s 2024-04-26T01:50:01.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:01 smithi053 ceph-mon[32593]: pgmap v280: 97 pgs: 97 active+clean; 3.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 10 MiB/s wr, 914 op/s 2024-04-26T01:50:02.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:01 smithi028 ceph-mon[28247]: pgmap v280: 97 pgs: 97 active+clean; 3.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 10 MiB/s wr, 914 op/s 2024-04-26T01:50:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:03 smithi169 ceph-mon[32851]: pgmap v281: 97 pgs: 97 active+clean; 3.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.0 MiB/s wr, 603 op/s 2024-04-26T01:50:03.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:03 smithi053 ceph-mon[32593]: pgmap v281: 97 pgs: 97 active+clean; 3.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.0 MiB/s wr, 603 op/s 2024-04-26T01:50:04.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:03 smithi028 ceph-mon[28247]: pgmap v281: 97 pgs: 97 active+clean; 3.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.0 MiB/s wr, 603 op/s 2024-04-26T01:50:05.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:05 smithi028 ceph-mon[28247]: pgmap v282: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 14 MiB/s wr, 971 op/s 2024-04-26T01:50:05.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:05 smithi169 ceph-mon[32851]: pgmap v282: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 14 MiB/s wr, 971 op/s 2024-04-26T01:50:05.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:05 smithi053 ceph-mon[32593]: pgmap v282: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 14 MiB/s wr, 971 op/s 2024-04-26T01:50:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 490.0 s, 689.0 tps, lat 148.590 ms stddev 305.155 2024-04-26T01:50:07.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:07 smithi169 ceph-mon[32851]: pgmap v283: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 27 MiB/s wr, 1.69k op/s 2024-04-26T01:50:07.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:07 smithi053 ceph-mon[32593]: pgmap v283: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 27 MiB/s wr, 1.69k op/s 2024-04-26T01:50:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:07 smithi028 ceph-mon[28247]: pgmap v283: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 27 MiB/s wr, 1.69k op/s 2024-04-26T01:50:08.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:08 smithi169 ceph-mon[32851]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:50:08.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:08 smithi053 ceph-mon[32593]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:50:09.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:08 smithi028 ceph-mon[28247]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:50:09.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:09 smithi169 ceph-mon[32851]: pgmap v285: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-04-26T01:50:09.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:09 smithi053 ceph-mon[32593]: pgmap v285: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-04-26T01:50:10.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:09 smithi028 ceph-mon[28247]: pgmap v285: 97 pgs: 97 active+clean; 3.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-04-26T01:50:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 495.0 s, 2575.2 tps, lat 38.628 ms stddev 46.287 2024-04-26T01:50:11.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:11 smithi169 ceph-mon[32851]: pgmap v286: 97 pgs: 97 active+clean; 3.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 30 MiB/s wr, 1.87k op/s 2024-04-26T01:50:11.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:11 smithi053 ceph-mon[32593]: pgmap v286: 97 pgs: 97 active+clean; 3.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 30 MiB/s wr, 1.87k op/s 2024-04-26T01:50:12.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:11 smithi028 ceph-mon[28247]: pgmap v286: 97 pgs: 97 active+clean; 3.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 30 MiB/s wr, 1.87k op/s 2024-04-26T01:50:13.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:13 smithi169 ceph-mon[32851]: pgmap v287: 97 pgs: 97 active+clean; 3.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 30 MiB/s wr, 1.86k op/s 2024-04-26T01:50:13.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:13 smithi169 ceph-mon[32851]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:50:13.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:13 smithi053 ceph-mon[32593]: pgmap v287: 97 pgs: 97 active+clean; 3.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 30 MiB/s wr, 1.86k op/s 2024-04-26T01:50:13.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:13 smithi053 ceph-mon[32593]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:50:14.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:13 smithi028 ceph-mon[28247]: pgmap v287: 97 pgs: 97 active+clean; 3.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 30 MiB/s wr, 1.86k op/s 2024-04-26T01:50:14.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:13 smithi028 ceph-mon[28247]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:50:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.268+0000 7fe8ba000640 1 -- 172.21.15.28:0/3985436620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4109490 msgr2=0x7fe8b41098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.268+0000 7fe8ba000640 1 --2- 172.21.15.28:0/3985436620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4109490 0x7fe8b41098f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009900 tx=0x7fe8a002a420 comp rx=0 tx=0).stop 2024-04-26T01:50:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.270+0000 7fe8ba000640 1 -- 172.21.15.28:0/3985436620 shutdown_connections 2024-04-26T01:50:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.270+0000 7fe8ba000640 1 --2- 172.21.15.28:0/3985436620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4116f50 0x7fe8b4119340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.270+0000 7fe8ba000640 1 --2- 172.21.15.28:0/3985436620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4109490 0x7fe8b41098f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.270+0000 7fe8ba000640 1 --2- 172.21.15.28:0/3985436620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b410c540 0x7fe8b4108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.270+0000 7fe8ba000640 1 -- 172.21.15.28:0/3985436620 >> 172.21.15.28:0/3985436620 conn(0x7fe8b406ce90 msgr2=0x7fe8b406d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.270+0000 7fe8ba000640 1 -- 172.21.15.28:0/3985436620 shutdown_connections 2024-04-26T01:50:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.270+0000 7fe8ba000640 1 -- 172.21.15.28:0/3985436620 wait complete. 2024-04-26T01:50:14.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.272+0000 7fe8ba000640 1 Processor -- start 2024-04-26T01:50:14.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.272+0000 7fe8ba000640 1 -- start start 2024-04-26T01:50:14.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.273+0000 7fe8ba000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4109490 0x7fe8b41111e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.273+0000 7fe8ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b410c540 0x7fe8b4111720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.273+0000 7fe8ba000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4116f50 0x7fe8b4115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.273+0000 7fe8b8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4109490 0x7fe8b41111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.274+0000 7fe8b8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4109490 0x7fe8b41111e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48498/0 (socket says 172.21.15.28:48498) 2024-04-26T01:50:14.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.274+0000 7fe8b8c00640 1 -- 172.21.15.28:0/2739588582 learned_addr learned my addr 172.21.15.28:0/2739588582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:14.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.274+0000 7fe8b9600640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4116f50 0x7fe8b4115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.274+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b411cb60 con 0x7fe8b410c540 2024-04-26T01:50:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.274+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b411c9e0 con 0x7fe8b4109490 2024-04-26T01:50:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.274+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b411cce0 con 0x7fe8b4116f50 2024-04-26T01:50:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.274+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b410c540 0x7fe8b4111720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.275+0000 7fe8b9600640 1 -- 172.21.15.28:0/2739588582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4109490 msgr2=0x7fe8b41111e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.275+0000 7fe8b9600640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4109490 0x7fe8b41111e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.275+0000 7fe8b9600640 1 -- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b410c540 msgr2=0x7fe8b4111720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.275+0000 7fe8b9600640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b410c540 0x7fe8b4111720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.275+0000 7fe8b9600640 1 -- 172.21.15.28:0/2739588582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b4116820 con 0x7fe8b4116f50 2024-04-26T01:50:14.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.275+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b410c540 0x7fe8b4111720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:14.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.276+0000 7fe8b9600640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4116f50 0x7fe8b4115f90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00ca80 tx=0x7fe8ac00cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:14.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.278+0000 7fe8b1600640 1 -- 172.21.15.28:0/2739588582 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac00ae20 con 0x7fe8b4116f50 2024-04-26T01:50:14.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.278+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b4116b10 con 0x7fe8b4116f50 2024-04-26T01:50:14.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.278+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b41a9120 con 0x7fe8b4116f50 2024-04-26T01:50:14.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.278+0000 7fe8b1600640 1 -- 172.21.15.28:0/2739588582 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8ac00f900 con 0x7fe8b4116f50 2024-04-26T01:50:14.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.278+0000 7fe8b1600640 1 -- 172.21.15.28:0/2739588582 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac00ae20 con 0x7fe8b4116f50 2024-04-26T01:50:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.283+0000 7fe8b1600640 1 -- 172.21.15.28:0/2739588582 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe8ac00e7b0 con 0x7fe8b4116f50 2024-04-26T01:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.287+0000 7fe8b1600640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe89c0a4110 0x7fe89c0a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.287+0000 7fe8b1600640 1 -- 172.21.15.28:0/2739588582 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fe8ac00ac20 con 0x7fe8b4116f50 2024-04-26T01:50:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.291+0000 7fe8be364740 1 Processor -- start 2024-04-26T01:50:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.292+0000 7fe8be364740 1 -- start start 2024-04-26T01:50:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.292+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe89c0a4110 0x7fe89c0a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.294+0000 7fe8be364740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399bd9b0 0x55f2399bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.294+0000 7fe8be364740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399be2d0 0x55f2399c8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.294+0000 7fe8be364740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399c8b90 0x55f2399caf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.294+0000 7fe8be364740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f239923900 con 0x55f2399bd9b0 2024-04-26T01:50:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.294+0000 7fe8be364740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f2398c9ef0 con 0x55f2399c8b90 2024-04-26T01:50:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.294+0000 7fe8be364740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f23991f8c0 con 0x55f2399be2d0 2024-04-26T01:50:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.295+0000 7fe8b9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399c8b90 0x55f2399caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.295+0000 7fe8b9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399c8b90 0x55f2399caf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48506/0 (socket says 172.21.15.28:48506) 2024-04-26T01:50:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.295+0000 7fe8b9600640 1 -- 172.21.15.28:0/1524039133 learned_addr learned my addr 172.21.15.28:0/1524039133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.295+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399be2d0 0x55f2399c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.295+0000 7fe8b9600640 1 -- 172.21.15.28:0/1524039133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399be2d0 msgr2=0x55f2399c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.295+0000 7fe8b9600640 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399be2d0 0x55f2399c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.295+0000 7fe8b9600640 1 -- 172.21.15.28:0/1524039133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399bd9b0 msgr2=0x55f2399bdd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:50:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.296+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399bd9b0 0x55f2399bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.296+0000 7fe8b9600640 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399bd9b0 0x55f2399bdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.296+0000 7fe8b9600640 1 -- 172.21.15.28:0/1524039133 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f239636210 con 0x55f2399c8b90 2024-04-26T01:50:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.296+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399bd9b0 0x55f2399bdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:50:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.296+0000 7fe8b9600640 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399c8b90 0x55f2399caf80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac098e70 tx=0x7fe8ac00aee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1005f2a82ce46ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.296+0000 7fe8a5c00640 1 -- 172.21.15.28:0/1524039133 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0a5110 con 0x55f2399c8b90 2024-04-26T01:50:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.297+0000 7fe8a5c00640 1 -- 172.21.15.28:0/1524039133 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8ac0cded0 con 0x55f2399c8b90 2024-04-26T01:50:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.297+0000 7fe8a5c00640 1 -- 172.21.15.28:0/1524039133 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac09b8f0 con 0x55f2399c8b90 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.297+0000 7fe8be364740 1 -- 172.21.15.28:0/1524039133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399c8b90 msgr2=0x55f2399caf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.297+0000 7fe8be364740 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399c8b90 0x55f2399caf80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac098e70 tx=0x7fe8ac00aee0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.297+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe89c0a4110 0x7fe89c0a65d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8009a30 tx=0x7fe8a8009190 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.298+0000 7fe8be364740 1 -- 172.21.15.28:0/1524039133 shutdown_connections 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.298+0000 7fe8be364740 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399c8b90 0x55f2399caf80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.298+0000 7fe8be364740 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399be2d0 0x55f2399c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.298+0000 7fe8be364740 1 --2- 172.21.15.28:0/1524039133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399bd9b0 0x55f2399bdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.298+0000 7fe8be364740 1 -- 172.21.15.28:0/1524039133 >> 172.21.15.28:0/1524039133 conn(0x55f2399b9ec0 msgr2=0x55f2399ba2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.298+0000 7fe8be364740 1 -- 172.21.15.28:0/1524039133 shutdown_connections 2024-04-26T01:50:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.298+0000 7fe8be364740 1 -- 172.21.15.28:0/1524039133 wait complete. 2024-04-26T01:50:14.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.300+0000 7fe8be364740 1 -- start start 2024-04-26T01:50:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.301+0000 7fe8be364740 1 Processor -- start 2024-04-26T01:50:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.301+0000 7fe8be364740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399bd9b0 0x55f2399c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.301+0000 7fe8be364740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399be2d0 0x55f2399c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.301+0000 7fe8be364740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399c8b90 0x55f2399c6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.301+0000 7fe8be364740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2399cb7c0 con 0x55f2399c8b90 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.301+0000 7fe8be364740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f2398dd310 con 0x55f2399be2d0 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.301+0000 7fe8be364740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f2399cb940 con 0x55f2399bd9b0 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8be364740 10 client.0 _ll_register_callbacks cb 0x55f2398abcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399be2d0 0x55f2399c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399be2d0 0x55f2399c5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48516/0 (socket says 172.21.15.28:48516) 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b3e00640 1 -- 172.21.15.28:0/1180750050 learned_addr learned my addr 172.21.15.28:0/1180750050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b9600640 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399c8b90 0x55f2399c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399bd9b0 0x55f2399c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b3e00640 1 -- 172.21.15.28:0/1180750050 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399bd9b0 msgr2=0x55f2399c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399bd9b0 0x55f2399c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b3e00640 1 -- 172.21.15.28:0/1180750050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399c8b90 msgr2=0x55f2399c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.302+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399c8b90 0x55f2399c6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.303+0000 7fe8b3e00640 1 -- 172.21.15.28:0/1180750050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f239702620 con 0x55f2399be2d0 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.303+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399be2d0 0x55f2399c5ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009900 tx=0x7fe8a0003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21cfbb3512d8e24d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.303+0000 7fe89a000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.303+0000 7fe89a000640 1 -- 172.21.15.28:0/1180750050 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0035040 con 0x55f2399be2d0 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.304+0000 7fe8be364740 1 -- 172.21.15.28:0/1180750050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8ac0cded0 con 0x55f2399be2d0 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.304+0000 7fe89a000640 1 -- 172.21.15.28:0/1180750050 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fe8a0003c50 con 0x55f2399be2d0 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.304+0000 7fe89a000640 1 -- 172.21.15.28:0/1180750050 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe8a0003e50 con 0x55f2399be2d0 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.304+0000 7fe8be364740 10 client.26024 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.304+0000 7fe8be364740 1 -- 172.21.15.28:0/1180750050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f2399d79b0 con 0x55f2399be2d0 2024-04-26T01:50:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.304+0000 7fe89a000640 1 -- 172.21.15.28:0/1180750050 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8ac0cded0 con 0x55f2399be2d0 2024-04-26T01:50:14.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.305+0000 7fe89a000640 1 -- 172.21.15.28:0/1180750050 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fe8a003c6e0 con 0x55f2399be2d0 2024-04-26T01:50:14.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.305+0000 7fe8be364740 10 client.26024 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:14.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.305+0000 7fe8be364740 10 client.26024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:14.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.305+0000 7fe8be364740 20 client.26024 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.305+0000 7fe8be364740 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f2399d8e80 0x55f2399db290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.305+0000 7fe8be364740 4 client.26024 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.305+0000 7fe8be364740 1 -- 172.21.15.28:0/1180750050 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2399378b0 con 0x55f2399d8e80 2024-04-26T01:50:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.306+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f2399d8e80 0x55f2399db290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.307+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f2399d8e80 0x55f2399db290 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=143140f70a7b82c3 server_cookie=ae79f048d5379043 in_seq=0 out_seq=0 2024-04-26T01:50:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.307+0000 7fe89a000640 10 client.26024 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.309+0000 7fe89a000640 1 -- 172.21.15.28:0/1180750050 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2399378b0 con 0x55f2399d8e80 2024-04-26T01:50:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.309+0000 7fe89a000640 10 client.26024 handle_command_reply: tid=0 2024-04-26T01:50:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.309+0000 7fe8be364740 1 client.26024 shutdown 2024-04-26T01:50:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe8be364740 1 -- 172.21.15.28:0/1180750050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399be2d0 msgr2=0x55f2399c5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe8be364740 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399be2d0 0x55f2399c5ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009900 tx=0x7fe8a0003820 comp rx=0 tx=0).stop 2024-04-26T01:50:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe8be364740 1 -- 172.21.15.28:0/1180750050 shutdown_connections 2024-04-26T01:50:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe8be364740 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f2399d8e80 0x55f2399db290 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe8be364740 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399c8b90 0x55f2399c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe8be364740 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399be2d0 0x55f2399c5ff0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe8be364740 1 --2- 172.21.15.28:0/1180750050 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399bd9b0 0x55f2399c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe8be364740 1 -- 172.21.15.28:0/1180750050 >> 172.21.15.28:0/1180750050 conn(0x55f2399b9ec0 msgr2=0x55f2399bfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.310+0000 7fe89a000640 0 client.26024 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:14.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.312+0000 7fe8be364740 1 -- 172.21.15.28:0/1180750050 shutdown_connections 2024-04-26T01:50:14.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.312+0000 7fe8be364740 1 -- 172.21.15.28:0/1180750050 wait complete. 2024-04-26T01:50:14.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.312+0000 7fe8be364740 20 client.26024 trim_cache size 0 max 16384 2024-04-26T01:50:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.337+0000 7fe8be364740 1 Processor -- start 2024-04-26T01:50:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.337+0000 7fe8be364740 1 -- start start 2024-04-26T01:50:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.337+0000 7fe8be364740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.337+0000 7fe8be364740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.337+0000 7fe8be364740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 0x55f2399e0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.337+0000 7fe8be364740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe894011080 con 0x55f2399be2d0 2024-04-26T01:50:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.337+0000 7fe8be364740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe89400e8f0 con 0x55f2399bd9b0 2024-04-26T01:50:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.337+0000 7fe8be364740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe89400c760 con 0x55f2399c8b90 2024-04-26T01:50:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.338+0000 7fe8b8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.338+0000 7fe8b8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48532/0 (socket says 172.21.15.28:48532) 2024-04-26T01:50:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.338+0000 7fe8b8c00640 1 -- 172.21.15.28:0/3067507863 learned_addr learned my addr 172.21.15.28:0/3067507863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.339+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.339+0000 7fe8b9600640 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 0x55f2399e0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.339+0000 7fe8b8c00640 1 -- 172.21.15.28:0/3067507863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 msgr2=0x55f2399e0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.339+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 0x55f2399e0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.339+0000 7fe8b8c00640 1 -- 172.21.15.28:0/3067507863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 msgr2=0x55f2399c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.339+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399c79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.339+0000 7fe8b8c00640 1 -- 172.21.15.28:0/3067507863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2399db910 con 0x55f2399bd9b0 2024-04-26T01:50:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.339+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:50:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.340+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d4680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55f2399dbb40 tx=0x7fe8a8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d441e26291e7422f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.340+0000 7fe8a5200640 1 -- 172.21.15.28:0/3067507863 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8013d40 con 0x55f2399bd9b0 2024-04-26T01:50:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.340+0000 7fe8a5200640 1 -- 172.21.15.28:0/3067507863 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a802b070 con 0x55f2399bd9b0 2024-04-26T01:50:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.340+0000 7fe8a5200640 1 -- 172.21.15.28:0/3067507863 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8022940 con 0x55f2399bd9b0 2024-04-26T01:50:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.341+0000 7fe8be364740 1 -- 172.21.15.28:0/3067507863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 msgr2=0x55f2399d4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.341+0000 7fe8be364740 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d4680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55f2399dbb40 tx=0x7fe8a8017000 comp rx=0 tx=0).stop 2024-04-26T01:50:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.342+0000 7fe8be364740 1 -- 172.21.15.28:0/3067507863 shutdown_connections 2024-04-26T01:50:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.342+0000 7fe8be364740 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 0x55f2399e0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.342+0000 7fe8be364740 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.342+0000 7fe8be364740 1 --2- 172.21.15.28:0/3067507863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d4680 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.342+0000 7fe8be364740 1 -- 172.21.15.28:0/3067507863 >> 172.21.15.28:0/3067507863 conn(0x55f2399b9ec0 msgr2=0x55f2399c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.342+0000 7fe8be364740 1 -- 172.21.15.28:0/3067507863 shutdown_connections 2024-04-26T01:50:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.342+0000 7fe8be364740 1 -- 172.21.15.28:0/3067507863 wait complete. 2024-04-26T01:50:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.343+0000 7fe8be364740 1 -- start start 2024-04-26T01:50:14.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.343+0000 7fe8be364740 1 Processor -- start 2024-04-26T01:50:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8be364740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8be364740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8be364740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 0x55f2399ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8be364740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe89400cdb0 con 0x55f2399be2d0 2024-04-26T01:50:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8be364740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe89400e220 con 0x55f2399bd9b0 2024-04-26T01:50:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8be364740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe89400d480 con 0x55f2399c8b90 2024-04-26T01:50:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8be364740 10 client.0 _ll_register_callbacks cb 0x7fe8a0004500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8b8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8b8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48546/0 (socket says 172.21.15.28:48546) 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.344+0000 7fe8b8c00640 1 -- 172.21.15.28:0/1913615983 learned_addr learned my addr 172.21.15.28:0/1913615983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe8b9600640 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 0x55f2399ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe8b8c00640 1 -- 172.21.15.28:0/1913615983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 msgr2=0x55f2399ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 0x55f2399ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe8b8c00640 1 -- 172.21.15.28:0/1913615983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 msgr2=0x55f2399ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399ce180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe8b8c00640 1 -- 172.21.15.28:0/1913615983 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2399ced60 con 0x55f2399bd9b0 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d0df0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8009810 tx=0x7fe8a80224b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd2bcb3d97ad4fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.345+0000 7fe89a000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:50:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.346+0000 7fe89a000640 1 -- 172.21.15.28:0/1913615983 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8005e40 con 0x55f2399bd9b0 2024-04-26T01:50:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.346+0000 7fe89a000640 1 -- 172.21.15.28:0/1913615983 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fe8a80271d0 con 0x55f2399bd9b0 2024-04-26T01:50:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.346+0000 7fe8be364740 1 -- 172.21.15.28:0/1913615983 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8a802b070 con 0x55f2399bd9b0 2024-04-26T01:50:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.347+0000 7fe89a000640 1 -- 172.21.15.28:0/1913615983 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe8a80273d0 con 0x55f2399bd9b0 2024-04-26T01:50:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.347+0000 7fe8be364740 10 client.26030 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.347+0000 7fe8be364740 1 -- 172.21.15.28:0/1913615983 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f239a3f230 con 0x55f2399bd9b0 2024-04-26T01:50:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.347+0000 7fe89a000640 1 -- 172.21.15.28:0/1913615983 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a802b070 con 0x55f2399bd9b0 2024-04-26T01:50:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.347+0000 7fe89a000640 1 -- 172.21.15.28:0/1913615983 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fe8a80225c0 con 0x55f2399bd9b0 2024-04-26T01:50:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.347+0000 7fe8be364740 10 client.26030 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.347+0000 7fe8be364740 10 client.26030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.347+0000 7fe8be364740 20 client.26030 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.348+0000 7fe8be364740 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f239a3f920 0x55f239a41d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.348+0000 7fe8be364740 4 client.26030 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.348+0000 7fe8be364740 1 -- 172.21.15.28:0/1913615983 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f239931320 con 0x55f239a3f920 2024-04-26T01:50:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.348+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f239a3f920 0x55f239a41d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.349+0000 7fe89a000640 10 client.26030 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.349+0000 7fe8b3e00640 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f239a3f920 0x55f239a41d40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=280449d3c3053dfb server_cookie=fdb13f6988b3275b in_seq=0 out_seq=0 2024-04-26T01:50:14.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.364+0000 7fe89a000640 1 -- 172.21.15.28:0/1913615983 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f239931320 con 0x55f239a3f920 2024-04-26T01:50:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.364+0000 7fe89a000640 10 client.26030 handle_command_reply: tid=0 2024-04-26T01:50:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.365+0000 7fe8be364740 1 client.26030 shutdown 2024-04-26T01:50:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 -- 172.21.15.28:0/1913615983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 msgr2=0x55f2399d0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d0df0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8009810 tx=0x7fe8a80224b0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 -- 172.21.15.28:0/1913615983 shutdown_connections 2024-04-26T01:50:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f239a3f920 0x55f239a41d40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f2399c8b90 0x55f2399ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2399be2d0 0x55f2399ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 --2- 172.21.15.28:0/1913615983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f2399bd9b0 0x55f2399d0df0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 -- 172.21.15.28:0/1913615983 >> 172.21.15.28:0/1913615983 conn(0x55f2399b9ec0 msgr2=0x55f2399c7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe89a000640 0 client.26030 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 -- 172.21.15.28:0/1913615983 shutdown_connections 2024-04-26T01:50:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.367+0000 7fe8be364740 1 -- 172.21.15.28:0/1913615983 wait complete. 2024-04-26T01:50:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.368+0000 7fe8be364740 20 client.26030 trim_cache size 0 max 16384 2024-04-26T01:50:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:50:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:50:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:50:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:50:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.369+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe89c0a4110 msgr2=0x7fe89c0a65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.369+0000 7fe8ba000640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe89c0a4110 0x7fe89c0a65d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8009a30 tx=0x7fe8a8009190 comp rx=0 tx=0).stop 2024-04-26T01:50:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.369+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4116f50 msgr2=0x7fe8b4115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.369+0000 7fe8ba000640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4116f50 0x7fe8b4115f90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00ca80 tx=0x7fe8ac00cf40 comp rx=0 tx=0).stop 2024-04-26T01:50:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.370+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 shutdown_connections 2024-04-26T01:50:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.370+0000 7fe8ba000640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fe89c0a4110 0x7fe89c0a65d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.370+0000 7fe8ba000640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8b4116f50 0x7fe8b4115f90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.370+0000 7fe8ba000640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b410c540 0x7fe8b4111720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.370+0000 7fe8ba000640 1 --2- 172.21.15.28:0/2739588582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4109490 0x7fe8b41111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.370+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 >> 172.21.15.28:0/2739588582 conn(0x7fe8b406ce90 msgr2=0x7fe8b4117330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.370+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 shutdown_connections 2024-04-26T01:50:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:14.371+0000 7fe8ba000640 1 -- 172.21.15.28:0/2739588582 wait complete. 2024-04-26T01:50:15.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:15 smithi169 ceph-mon[32851]: pgmap v289: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.7 MiB/s wr, 687 op/s 2024-04-26T01:50:15.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:15 smithi053 ceph-mon[32593]: pgmap v289: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.7 MiB/s wr, 687 op/s 2024-04-26T01:50:16.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:15 smithi028 ceph-mon[28247]: pgmap v289: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.7 MiB/s wr, 687 op/s 2024-04-26T01:50:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 500.0 s, 2534.6 tps, lat 39.810 ms stddev 45.889 2024-04-26T01:50:17.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:17 smithi169 ceph-mon[32851]: pgmap v290: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 930 op/s 2024-04-26T01:50:17.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:17 smithi053 ceph-mon[32593]: pgmap v290: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 930 op/s 2024-04-26T01:50:18.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:17 smithi028 ceph-mon[28247]: pgmap v290: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 930 op/s 2024-04-26T01:50:19.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:19 smithi169 ceph-mon[32851]: pgmap v291: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.7 MiB/s wr, 799 op/s 2024-04-26T01:50:19.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:19 smithi053 ceph-mon[32593]: pgmap v291: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.7 MiB/s wr, 799 op/s 2024-04-26T01:50:20.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:19 smithi028 ceph-mon[28247]: pgmap v291: 97 pgs: 97 active+clean; 3.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.7 MiB/s wr, 799 op/s 2024-04-26T01:50:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 505.0 s, 2572.6 tps, lat 38.711 ms stddev 44.782 2024-04-26T01:50:21.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:21 smithi169 ceph-mon[32851]: pgmap v292: 97 pgs: 97 active+clean; 3.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.6 MiB/s wr, 680 op/s 2024-04-26T01:50:21.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:21 smithi053 ceph-mon[32593]: pgmap v292: 97 pgs: 97 active+clean; 3.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.6 MiB/s wr, 680 op/s 2024-04-26T01:50:22.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:21 smithi028 ceph-mon[28247]: pgmap v292: 97 pgs: 97 active+clean; 3.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.6 MiB/s wr, 680 op/s 2024-04-26T01:50:23.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:23 smithi169 ceph-mon[32851]: pgmap v293: 97 pgs: 97 active+clean; 3.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.6 MiB/s wr, 681 op/s 2024-04-26T01:50:23.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:23 smithi053 ceph-mon[32593]: pgmap v293: 97 pgs: 97 active+clean; 3.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.6 MiB/s wr, 681 op/s 2024-04-26T01:50:24.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:23 smithi028 ceph-mon[28247]: pgmap v293: 97 pgs: 97 active+clean; 3.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.6 MiB/s wr, 681 op/s 2024-04-26T01:50:24.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:24 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:25.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:24 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:25.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:24 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:25.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:25 smithi053 ceph-mon[32593]: pgmap v294: 97 pgs: 97 active+clean; 3.5 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.9 MiB/s wr, 815 op/s 2024-04-26T01:50:25.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:25 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:25.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:25 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:26.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:25 smithi028 ceph-mon[28247]: pgmap v294: 97 pgs: 97 active+clean; 3.5 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.9 MiB/s wr, 815 op/s 2024-04-26T01:50:26.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:25 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:26.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:25 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:26.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:25 smithi169 ceph-mon[32851]: pgmap v294: 97 pgs: 97 active+clean; 3.5 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.9 MiB/s wr, 815 op/s 2024-04-26T01:50:26.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:25 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:26.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:25 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 510.0 s, 2102.1 tps, lat 47.575 ms stddev 59.005 2024-04-26T01:50:26.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:26 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:26.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:26 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:26.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:26 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:26.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:26 smithi053 ceph-mon[32593]: pgmap v295: 97 pgs: 97 active+clean; 3.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.4 MiB/s wr, 851 op/s 2024-04-26T01:50:27.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:26 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:27.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:26 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:27.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:26 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:27.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:26 smithi028 ceph-mon[28247]: pgmap v295: 97 pgs: 97 active+clean; 3.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.4 MiB/s wr, 851 op/s 2024-04-26T01:50:27.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:26 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:27.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:26 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:27.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:26 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:50:27.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:26 smithi169 ceph-mon[32851]: pgmap v295: 97 pgs: 97 active+clean; 3.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.4 MiB/s wr, 851 op/s 2024-04-26T01:50:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.574+0000 7fb6c3800640 1 -- 172.21.15.28:0/2637701468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc1063d0 msgr2=0x7fb6bc106880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.574+0000 7fb6c3800640 1 --2- 172.21.15.28:0/2637701468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc1063d0 0x7fb6bc106880 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009be0 tx=0x7fb6b402a750 comp rx=0 tx=0).stop 2024-04-26T01:50:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.574+0000 7fb6c3800640 1 -- 172.21.15.28:0/2637701468 shutdown_connections 2024-04-26T01:50:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.574+0000 7fb6c3800640 1 --2- 172.21.15.28:0/2637701468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc1063d0 0x7fb6bc106880 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.574+0000 7fb6c3800640 1 --2- 172.21.15.28:0/2637701468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc1018f0 0x7fb6bc105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.574+0000 7fb6c3800640 1 --2- 172.21.15.28:0/2637701468 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6bc100fd0 0x7fb6bc1013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.574+0000 7fb6c3800640 1 -- 172.21.15.28:0/2637701468 >> 172.21.15.28:0/2637701468 conn(0x7fb6bc077f60 msgr2=0x7fb6bc078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.575+0000 7fb6c3800640 1 -- 172.21.15.28:0/2637701468 shutdown_connections 2024-04-26T01:50:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.575+0000 7fb6c3800640 1 -- 172.21.15.28:0/2637701468 wait complete. 2024-04-26T01:50:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.576+0000 7fb6c3800640 1 Processor -- start 2024-04-26T01:50:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.576+0000 7fb6c3800640 1 -- start start 2024-04-26T01:50:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.576+0000 7fb6c3800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc100fd0 0x7fb6bc1a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.576+0000 7fb6c3800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6bc1018f0 0x7fb6bc1a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.576+0000 7fb6c3800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc1063d0 0x7fb6bc19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.576+0000 7fb6c3800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc1187b0 con 0x7fb6bc1063d0 2024-04-26T01:50:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.576+0000 7fb6c3800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc118630 con 0x7fb6bc100fd0 2024-04-26T01:50:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.576+0000 7fb6c3800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc118930 con 0x7fb6bc1018f0 2024-04-26T01:50:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.577+0000 7fb6c1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc1063d0 0x7fb6bc19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.577+0000 7fb6c1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc1063d0 0x7fb6bc19eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49962/0 (socket says 172.21.15.28:49962) 2024-04-26T01:50:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.577+0000 7fb6c1800640 1 -- 172.21.15.28:0/152989921 learned_addr learned my addr 172.21.15.28:0/152989921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.577+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6bc1018f0 0x7fb6bc1a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.577+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc100fd0 0x7fb6bc1a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.578+0000 7fb6c1800640 1 -- 172.21.15.28:0/152989921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6bc1018f0 msgr2=0x7fb6bc1a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.578+0000 7fb6c1800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6bc1018f0 0x7fb6bc1a4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.578+0000 7fb6c1800640 1 -- 172.21.15.28:0/152989921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc100fd0 msgr2=0x7fb6bc1a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.578+0000 7fb6c1800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc100fd0 0x7fb6bc1a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.578+0000 7fb6c1800640 1 -- 172.21.15.28:0/152989921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc19f3d0 con 0x7fb6bc1063d0 2024-04-26T01:50:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.579+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc100fd0 0x7fb6bc1a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.579+0000 7fb6c1800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc1063d0 0x7fb6bc19eb10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4030900 tx=0x7fb6b4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.580+0000 7fb6b9600640 1 -- 172.21.15.28:0/152989921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4041020 con 0x7fb6bc1063d0 2024-04-26T01:50:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.580+0000 7fb6c3800640 1 -- 172.21.15.28:0/152989921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6bc19f660 con 0x7fb6bc1063d0 2024-04-26T01:50:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.580+0000 7fb6c3800640 1 -- 172.21.15.28:0/152989921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6bc1ab7f0 con 0x7fb6bc1063d0 2024-04-26T01:50:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.580+0000 7fb6b9600640 1 -- 172.21.15.28:0/152989921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b40318f0 con 0x7fb6bc1063d0 2024-04-26T01:50:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.580+0000 7fb6b9600640 1 -- 172.21.15.28:0/152989921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4036040 con 0x7fb6bc1063d0 2024-04-26T01:50:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.585+0000 7fb6b9600640 1 -- 172.21.15.28:0/152989921 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb6b4044020 con 0x7fb6bc1063d0 2024-04-26T01:50:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.587+0000 7fb6c65f2740 1 Processor -- start 2024-04-26T01:50:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.587+0000 7fb6b9600640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6a40a3f10 0x7fb6a40a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.588+0000 7fb6c65f2740 1 -- start start 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.588+0000 7fb6b9600640 1 -- 172.21.15.28:0/152989921 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fb6b40f13a0 con 0x7fb6bc1063d0 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.588+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6dd820 0x556e9d6ddc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.588+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6de140 0x556e9d6e84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.588+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6eadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.588+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e9d643900 con 0x556e9d6e8a00 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.588+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556e9d5e9ef0 con 0x556e9d6dd820 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.588+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x556e9d63f8c0 con 0x556e9d6de140 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.589+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6a40a3f10 0x7fb6a40a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.589+0000 7fb6c0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6dd820 0x556e9d6ddc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.589+0000 7fb6c0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6dd820 0x556e9d6ddc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37774/0 (socket says 172.21.15.28:37774) 2024-04-26T01:50:27.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.589+0000 7fb6c0e00640 1 -- 172.21.15.28:0/3609316747 learned_addr learned my addr 172.21.15.28:0/3609316747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:27.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.589+0000 7fb6c1800640 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6eadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.589+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6de140 0x556e9d6e84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.590+0000 7fb6c0e00640 1 -- 172.21.15.28:0/3609316747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6de140 msgr2=0x556e9d6e84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.590+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6de140 0x556e9d6e84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.590+0000 7fb6c0e00640 1 -- 172.21.15.28:0/3609316747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 msgr2=0x556e9d6eadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.590+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6eadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.590+0000 7fb6c0e00640 1 -- 172.21.15.28:0/3609316747 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e9d356210 con 0x556e9d6dd820 2024-04-26T01:50:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.591+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6dd820 0x556e9d6ddc00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0011510 tx=0x7fb6b00119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d967d7c1fb701392 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.591+0000 7fb6ae000640 1 -- 172.21.15.28:0/3609316747 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0004020 con 0x556e9d6dd820 2024-04-26T01:50:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.591+0000 7fb6ae000640 1 -- 172.21.15.28:0/3609316747 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b00041c0 con 0x556e9d6dd820 2024-04-26T01:50:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.591+0000 7fb6ae000640 1 -- 172.21.15.28:0/3609316747 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000f890 con 0x556e9d6dd820 2024-04-26T01:50:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.591+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6a40a3f10 0x7fb6a40a63d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x556e9d6dedc0 tx=0x7fb6a8009740 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.592+0000 7fb6c65f2740 1 -- 172.21.15.28:0/3609316747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6dd820 msgr2=0x556e9d6ddc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.592+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6dd820 0x556e9d6ddc00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0011510 tx=0x7fb6b00119d0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.593+0000 7fb6c65f2740 1 -- 172.21.15.28:0/3609316747 shutdown_connections 2024-04-26T01:50:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.593+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6eadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.593+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6de140 0x556e9d6e84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.593+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/3609316747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6dd820 0x556e9d6ddc00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.593+0000 7fb6c65f2740 1 -- 172.21.15.28:0/3609316747 >> 172.21.15.28:0/3609316747 conn(0x556e9d6d9d30 msgr2=0x556e9d6da110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.595+0000 7fb6c65f2740 1 -- 172.21.15.28:0/3609316747 shutdown_connections 2024-04-26T01:50:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.595+0000 7fb6c65f2740 1 -- 172.21.15.28:0/3609316747 wait complete. 2024-04-26T01:50:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.596+0000 7fb6c65f2740 1 -- start start 2024-04-26T01:50:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.596+0000 7fb6c65f2740 1 Processor -- start 2024-04-26T01:50:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.597+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 0x556e9d6e5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.597+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d6e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.597+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6e6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.597+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e9d6eb630 con 0x556e9d6e8a00 2024-04-26T01:50:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.597+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556e9d5fd310 con 0x556e9d6de140 2024-04-26T01:50:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.597+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x556e9d6eb7b0 con 0x556e9d6dd820 2024-04-26T01:50:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.597+0000 7fb6c65f2740 10 client.0 _ll_register_callbacks cb 0x556e9d5cbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:27.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.598+0000 7fb6c1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6e6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.599+0000 7fb6c1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6e6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49978/0 (socket says 172.21.15.28:49978) 2024-04-26T01:50:27.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.599+0000 7fb6c1800640 1 -- 172.21.15.28:0/2120065213 learned_addr learned my addr 172.21.15.28:0/2120065213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:27.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.599+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 0x556e9d6e5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.599+0000 7fb6c1800640 1 -- 172.21.15.28:0/2120065213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 msgr2=0x556e9d6e5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.599+0000 7fb6c1800640 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 0x556e9d6e5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.600+0000 7fb6c1800640 1 -- 172.21.15.28:0/2120065213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 msgr2=0x556e9d6e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.600+0000 7fb6c1800640 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d6e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.600+0000 7fb6c1800640 1 -- 172.21.15.28:0/2120065213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e9d422620 con 0x556e9d6e8a00 2024-04-26T01:50:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.601+0000 7fb6c1800640 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6e6390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40eeaf0 tx=0x7fb6b40bce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d23e65a6b6ae0b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.601+0000 7fb6a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:50:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.602+0000 7fb6a2a00640 1 -- 172.21.15.28:0/2120065213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40d5020 con 0x556e9d6e8a00 2024-04-26T01:50:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.602+0000 7fb6a2a00640 1 -- 172.21.15.28:0/2120065213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fb6b40eed10 con 0x556e9d6e8a00 2024-04-26T01:50:27.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.602+0000 7fb6c65f2740 1 -- 172.21.15.28:0/2120065213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b00041c0 con 0x556e9d6e8a00 2024-04-26T01:50:27.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.602+0000 7fb6a2a00640 1 -- 172.21.15.28:0/2120065213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb6b40eef10 con 0x556e9d6e8a00 2024-04-26T01:50:27.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.602+0000 7fb6c65f2740 10 client.15840 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:27.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.602+0000 7fb6c65f2740 1 -- 172.21.15.28:0/2120065213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556e9d6f7810 con 0x556e9d6e8a00 2024-04-26T01:50:27.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.603+0000 7fb6a2a00640 1 -- 172.21.15.28:0/2120065213 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b409e030 con 0x556e9d6e8a00 2024-04-26T01:50:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.603+0000 7fb6a2a00640 1 -- 172.21.15.28:0/2120065213 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fb6b40c5ac0 con 0x556e9d6e8a00 2024-04-26T01:50:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.603+0000 7fb6c65f2740 10 client.15840 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.603+0000 7fb6c65f2740 10 client.15840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.603+0000 7fb6c65f2740 20 client.15840 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.604+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e9d6f8d20 0x556e9d6fb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.604+0000 7fb6c65f2740 4 client.15840 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.604+0000 7fb6c65f2740 1 -- 172.21.15.28:0/2120065213 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e9d6578b0 con 0x556e9d6f8d20 2024-04-26T01:50:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.605+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e9d6f8d20 0x556e9d6fb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.607+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e9d6f8d20 0x556e9d6fb1c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eebbde7aa52ac5ed server_cookie=59e752a5dc2a38cd in_seq=0 out_seq=0 2024-04-26T01:50:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.607+0000 7fb6a2a00640 10 client.15840 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:27.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.609+0000 7fb6a2a00640 1 -- 172.21.15.28:0/2120065213 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e9d6578b0 con 0x556e9d6f8d20 2024-04-26T01:50:27.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.609+0000 7fb6a2a00640 10 client.15840 handle_command_reply: tid=0 2024-04-26T01:50:27.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.609+0000 7fb6c65f2740 1 client.15840 shutdown 2024-04-26T01:50:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.610+0000 7fb6c65f2740 1 -- 172.21.15.28:0/2120065213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 msgr2=0x556e9d6e6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.610+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6e6390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40eeaf0 tx=0x7fb6b40bce20 comp rx=0 tx=0).stop 2024-04-26T01:50:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.612+0000 7fb6c65f2740 1 -- 172.21.15.28:0/2120065213 shutdown_connections 2024-04-26T01:50:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.612+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e9d6f8d20 0x556e9d6fb1c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.612+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6e6390 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.612+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d6e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.612+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/2120065213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 0x556e9d6e5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.612+0000 7fb6c65f2740 1 -- 172.21.15.28:0/2120065213 >> 172.21.15.28:0/2120065213 conn(0x556e9d6d9d30 msgr2=0x556e9d6dfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:27.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.613+0000 7fb6a2a00640 0 client.15840 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:27.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.614+0000 7fb6c65f2740 1 -- 172.21.15.28:0/2120065213 shutdown_connections 2024-04-26T01:50:27.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.614+0000 7fb6c65f2740 1 -- 172.21.15.28:0/2120065213 wait complete. 2024-04-26T01:50:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.614+0000 7fb6c65f2740 20 client.15840 trim_cache size 0 max 16384 2024-04-26T01:50:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.646+0000 7fb6c65f2740 1 Processor -- start 2024-04-26T01:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.646+0000 7fb6c65f2740 1 -- start start 2024-04-26T01:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.646+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6dd820 0x556e9d6f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.646+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d700a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.646+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6e8a00 0x556e9d700fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.646+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc1b6ad0 con 0x556e9d6dd820 2024-04-26T01:50:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.646+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc1b4340 con 0x556e9d6de140 2024-04-26T01:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.646+0000 7fb6c65f2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc1b21b0 con 0x556e9d6e8a00 2024-04-26T01:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c1800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6e8a00 0x556e9d700fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6dd820 0x556e9d6f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c1800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6e8a00 0x556e9d700fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59944/0 (socket says 172.21.15.28:59944) 2024-04-26T01:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c1800640 1 -- 172.21.15.28:0/1738849331 learned_addr learned my addr 172.21.15.28:0/1738849331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d700a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c0e00640 1 -- 172.21.15.28:0/1738849331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6e8a00 msgr2=0x556e9d700fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6e8a00 0x556e9d700fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c0e00640 1 -- 172.21.15.28:0/1738849331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 msgr2=0x556e9d700a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d700a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c0e00640 1 -- 172.21.15.28:0/1738849331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e9d6ddf70 con 0x556e9d6dd820 2024-04-26T01:50:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6c1800640 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6e8a00 0x556e9d700fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.647+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d700a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.648+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6dd820 0x556e9d6f44f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0011ed0 tx=0x7fb6b0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a984b5826940fa4c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.648+0000 7fb6ad600640 1 -- 172.21.15.28:0/1738849331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b002bbc0 con 0x556e9d6dd820 2024-04-26T01:50:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.648+0000 7fb6ad600640 1 -- 172.21.15.28:0/1738849331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556e9d6578b0 con 0x556e9d6dd820 2024-04-26T01:50:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.648+0000 7fb6ad600640 1 -- 172.21.15.28:0/1738849331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b002b350 con 0x556e9d6dd820 2024-04-26T01:50:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.649+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1738849331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6dd820 msgr2=0x556e9d6f44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.649+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6dd820 0x556e9d6f44f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0011ed0 tx=0x7fb6b0009260 comp rx=0 tx=0).stop 2024-04-26T01:50:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.650+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1738849331 shutdown_connections 2024-04-26T01:50:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.650+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6e8a00 0x556e9d700fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.650+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d700a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.650+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1738849331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6dd820 0x556e9d6f44f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.650+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1738849331 >> 172.21.15.28:0/1738849331 conn(0x556e9d6d9d30 msgr2=0x556e9d6e0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.651+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1738849331 shutdown_connections 2024-04-26T01:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.651+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1738849331 wait complete. 2024-04-26T01:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.652+0000 7fb6c65f2740 1 -- start start 2024-04-26T01:50:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.652+0000 7fb6c65f2740 1 Processor -- start 2024-04-26T01:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.654+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 0x556e9d757ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.654+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d757ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.654+0000 7fb6c65f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6efa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.654+0000 7fb6bbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d757ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.654+0000 7fb6c1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.654+0000 7fb6c1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6efa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50004/0 (socket says 172.21.15.28:50004) 2024-04-26T01:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.654+0000 7fb6bbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d757ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37794/0 (socket says 172.21.15.28:37794) 2024-04-26T01:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.655+0000 7fb6c1800640 1 -- 172.21.15.28:0/1057483257 learned_addr learned my addr 172.21.15.28:0/1057483257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.655+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc1b2800 con 0x556e9d6e8a00 2024-04-26T01:50:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.655+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 0x556e9d757ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.655+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc1b3c70 con 0x556e9d6de140 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.655+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc1b2ed0 con 0x556e9d6dd820 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.656+0000 7fb6c65f2740 10 client.0 _ll_register_callbacks cb 0x7fb6b40b9580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.656+0000 7fb6bbe00640 1 -- 172.21.15.28:0/1057483257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 msgr2=0x556e9d757ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.656+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 0x556e9d757ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.656+0000 7fb6bbe00640 1 -- 172.21.15.28:0/1057483257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 msgr2=0x556e9d6efa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.656+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6efa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.656+0000 7fb6bbe00640 1 -- 172.21.15.28:0/1057483257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e9d6eff70 con 0x556e9d6de140 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.656+0000 7fb6c1800640 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6efa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.657+0000 7fb6bbe00640 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d757ff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009920 tx=0x7fb6a80099d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=921efbb417c43f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:27.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.657+0000 7fb6a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.657+0000 7fb6a2a00640 1 -- 172.21.15.28:0/1057483257 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80035f0 con 0x556e9d6de140 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.657+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556e9d6578b0 con 0x556e9d6de140 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.657+0000 7fb6a2a00640 1 -- 172.21.15.28:0/1057483257 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fb6a8003790 con 0x556e9d6de140 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.658+0000 7fb6a2a00640 1 -- 172.21.15.28:0/1057483257 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb6a80039b0 con 0x556e9d6de140 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.658+0000 7fb6c65f2740 10 client.26045 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.658+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556e9d6f12d0 con 0x556e9d6de140 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.658+0000 7fb6a2a00640 1 -- 172.21.15.28:0/1057483257 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556e9d6578b0 con 0x556e9d6de140 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.658+0000 7fb6a2a00640 1 -- 172.21.15.28:0/1057483257 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fb6a801f040 con 0x556e9d6de140 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.659+0000 7fb6c65f2740 10 client.26045 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.659+0000 7fb6c65f2740 10 client.26045 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.659+0000 7fb6c65f2740 20 client.26045 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.659+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e9d7708a0 0x556e9d772cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.659+0000 7fb6c65f2740 4 client.26045 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.659+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e9d651320 con 0x556e9d7708a0 2024-04-26T01:50:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.659+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e9d7708a0 0x556e9d772cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.660+0000 7fb6c0e00640 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e9d7708a0 0x556e9d772cc0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8775c08469dbb3 server_cookie=faa4707e191d134d in_seq=0 out_seq=0 2024-04-26T01:50:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.660+0000 7fb6a2a00640 10 client.26045 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6a2a00640 1 -- 172.21.15.28:0/1057483257 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e9d651320 con 0x556e9d7708a0 2024-04-26T01:50:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6a2a00640 10 client.26045 handle_command_reply: tid=0 2024-04-26T01:50:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 client.26045 shutdown 2024-04-26T01:50:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 msgr2=0x556e9d757ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d757ff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009920 tx=0x7fb6a80099d0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 shutdown_connections 2024-04-26T01:50:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x556e9d7708a0 0x556e9d772cc0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e9d6e8a00 0x556e9d6efa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556e9d6de140 0x556e9d757ff0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 --2- 172.21.15.28:0/1057483257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556e9d6dd820 0x556e9d757ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 >> 172.21.15.28:0/1057483257 conn(0x556e9d6d9d30 msgr2=0x556e9d6d8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.672+0000 7fb6a2a00640 0 client.26045 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.673+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 shutdown_connections 2024-04-26T01:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.673+0000 7fb6c65f2740 1 -- 172.21.15.28:0/1057483257 wait complete. 2024-04-26T01:50:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.673+0000 7fb6c65f2740 20 client.26045 trim_cache size 0 max 16384 2024-04-26T01:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.674+0000 7fb6c3800640 1 -- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6a40a3f10 msgr2=0x7fb6a40a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.674+0000 7fb6c3800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6a40a3f10 0x7fb6a40a63d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x556e9d6dedc0 tx=0x7fb6a8009740 comp rx=0 tx=0).stop 2024-04-26T01:50:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.674+0000 7fb6c3800640 1 -- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc1063d0 msgr2=0x7fb6bc19eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.674+0000 7fb6c3800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc1063d0 0x7fb6bc19eb10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4030900 tx=0x7fb6b4003ba0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.675+0000 7fb6c3800640 1 -- 172.21.15.28:0/152989921 shutdown_connections 2024-04-26T01:50:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.675+0000 7fb6c3800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6a40a3f10 0x7fb6a40a63d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.675+0000 7fb6c3800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc1063d0 0x7fb6bc19eb10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.675+0000 7fb6c3800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6bc1018f0 0x7fb6bc1a4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.675+0000 7fb6c3800640 1 --2- 172.21.15.28:0/152989921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6bc100fd0 0x7fb6bc1a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.675+0000 7fb6c3800640 1 -- 172.21.15.28:0/152989921 >> 172.21.15.28:0/152989921 conn(0x7fb6bc077f60 msgr2=0x7fb6bc103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.676+0000 7fb6c3800640 1 -- 172.21.15.28:0/152989921 shutdown_connections 2024-04-26T01:50:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:27.676+0000 7fb6c3800640 1 -- 172.21.15.28:0/152989921 wait complete. 2024-04-26T01:50:29.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:29 smithi028 ceph-mon[28247]: pgmap v296: 97 pgs: 97 active+clean; 3.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.3 MiB/s wr, 643 op/s 2024-04-26T01:50:29.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:29 smithi169 ceph-mon[32851]: pgmap v296: 97 pgs: 97 active+clean; 3.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.3 MiB/s wr, 643 op/s 2024-04-26T01:50:29.671 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:29 smithi053 ceph-mon[32593]: pgmap v296: 97 pgs: 97 active+clean; 3.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.3 MiB/s wr, 643 op/s 2024-04-26T01:50:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 515.0 s, 1180.8 tps, lat 84.481 ms stddev 273.003 2024-04-26T01:50:31.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:31 smithi028 ceph-mon[28247]: pgmap v297: 97 pgs: 97 active+clean; 3.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.9 MiB/s wr, 906 op/s 2024-04-26T01:50:31.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:31 smithi169 ceph-mon[32851]: pgmap v297: 97 pgs: 97 active+clean; 3.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.9 MiB/s wr, 906 op/s 2024-04-26T01:50:31.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:31 smithi053 ceph-mon[32593]: pgmap v297: 97 pgs: 97 active+clean; 3.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.9 MiB/s wr, 906 op/s 2024-04-26T01:50:33.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:33 smithi028 ceph-mon[28247]: pgmap v298: 97 pgs: 97 active+clean; 3.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 625 op/s 2024-04-26T01:50:33.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:33 smithi169 ceph-mon[32851]: pgmap v298: 97 pgs: 97 active+clean; 3.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 625 op/s 2024-04-26T01:50:33.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:33 smithi053 ceph-mon[32593]: pgmap v298: 97 pgs: 97 active+clean; 3.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 625 op/s 2024-04-26T01:50:35.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:35 smithi169 ceph-mon[32851]: pgmap v299: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.5 MiB/s wr, 754 op/s 2024-04-26T01:50:35.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:35 smithi053 ceph-mon[32593]: pgmap v299: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.5 MiB/s wr, 754 op/s 2024-04-26T01:50:35.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:35 smithi028 ceph-mon[28247]: pgmap v299: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.5 MiB/s wr, 754 op/s 2024-04-26T01:50:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 520.0 s, 2845.0 tps, lat 35.224 ms stddev 40.185 2024-04-26T01:50:37.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:37 smithi169 ceph-mon[32851]: pgmap v300: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.6 MiB/s wr, 680 op/s 2024-04-26T01:50:37.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:37 smithi053 ceph-mon[32593]: pgmap v300: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.6 MiB/s wr, 680 op/s 2024-04-26T01:50:37.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:37 smithi028 ceph-mon[28247]: pgmap v300: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.6 MiB/s wr, 680 op/s 2024-04-26T01:50:39.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:39 smithi169 ceph-mon[32851]: pgmap v301: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-26T01:50:39.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:39 smithi053 ceph-mon[32593]: pgmap v301: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-26T01:50:39.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:39 smithi028 ceph-mon[28247]: pgmap v301: 97 pgs: 97 active+clean; 3.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-26T01:50:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.891+0000 7f6625600640 1 -- 172.21.15.28:0/1036346363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 msgr2=0x7f66201007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.891+0000 7f6625600640 1 --2- 172.21.15.28:0/1036346363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 0x7f66201007c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6610009860 tx=0x7f661002a340 comp rx=0 tx=0).stop 2024-04-26T01:50:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.892+0000 7f6625600640 1 -- 172.21.15.28:0/1036346363 shutdown_connections 2024-04-26T01:50:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.892+0000 7f6625600640 1 --2- 172.21.15.28:0/1036346363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66201054b0 0x7f662010d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.892+0000 7f6625600640 1 --2- 172.21.15.28:0/1036346363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6620100d90 0x7f6620104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.892+0000 7f6625600640 1 --2- 172.21.15.28:0/1036346363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 0x7f66201007c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.892+0000 7f6625600640 1 -- 172.21.15.28:0/1036346363 >> 172.21.15.28:0/1036346363 conn(0x7f6620076f50 msgr2=0x7f66200fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.892+0000 7f6625600640 1 -- 172.21.15.28:0/1036346363 shutdown_connections 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.892+0000 7f6625600640 1 -- 172.21.15.28:0/1036346363 wait complete. 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.894+0000 7f6625600640 1 Processor -- start 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.895+0000 7f6625600640 1 -- start start 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f6625600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 0x7f662010bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f6625600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66201054b0 0x7f662010c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f6625600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66201a4790 0x7f66201a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f6625600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f662010f580 con 0x7f66201a4790 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f6625600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f662010f400 con 0x7f66201054b0 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f6625600640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f662010f700 con 0x7f66201003e0 2024-04-26T01:50:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f661ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 0x7f662010bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f661ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 0x7f662010bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51516/0 (socket says 172.21.15.28:51516) 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f661ea00640 1 -- 172.21.15.28:0/2055613216 learned_addr learned my addr 172.21.15.28:0/2055613216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.896+0000 7f661f400640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66201a4790 0x7f66201a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.897+0000 7f661ea00640 1 -- 172.21.15.28:0/2055613216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66201054b0 msgr2=0x7f662010c1b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.897+0000 7f661ea00640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66201054b0 0x7f662010c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.897+0000 7f661ea00640 1 -- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66201a4790 msgr2=0x7f66201a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.897+0000 7f661ea00640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66201a4790 0x7f66201a4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.897+0000 7f661ea00640 1 -- 172.21.15.28:0/2055613216 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66201a5280 con 0x7f66201003e0 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.897+0000 7f661f400640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66201a4790 0x7f66201a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.897+0000 7f661ea00640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 0x7f662010bc70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6610035040 tx=0x7f6610032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.897+0000 7f6617400640 1 -- 172.21.15.28:0/2055613216 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610037040 con 0x7f66201003e0 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.898+0000 7f6617400640 1 -- 172.21.15.28:0/2055613216 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6610041070 con 0x7f66201003e0 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.898+0000 7f6617400640 1 -- 172.21.15.28:0/2055613216 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610003830 con 0x7f66201003e0 2024-04-26T01:50:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.898+0000 7f6625600640 1 -- 172.21.15.28:0/2055613216 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66201a5510 con 0x7f66201003e0 2024-04-26T01:50:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.899+0000 7f6625600640 1 -- 172.21.15.28:0/2055613216 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66201aba40 con 0x7f66201003e0 2024-04-26T01:50:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.903+0000 7f6628574740 1 Processor -- start 2024-04-26T01:50:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.903+0000 7f6628574740 1 -- start start 2024-04-26T01:50:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.904+0000 7f6628574740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea9431c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.904+0000 7f6617400640 1 -- 172.21.15.28:0/2055613216 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6610004050 con 0x7f66201003e0 2024-04-26T01:50:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.904+0000 7f6628574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea943c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.904+0000 7f6628574740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 0x564ea943edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.904+0000 7f6628574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ea9397900 con 0x564ea9432140 2024-04-26T01:50:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.904+0000 7f6628574740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564ea933def0 con 0x564ea943ca00 2024-04-26T01:50:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.904+0000 7f6628574740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x564ea93938c0 con 0x564ea9431820 2024-04-26T01:50:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.905+0000 7f661f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea9431c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.905+0000 7f661f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea9431c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51522/0 (socket says 172.21.15.28:51522) 2024-04-26T01:50:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.905+0000 7f661f400640 1 -- 172.21.15.28:0/1425267039 learned_addr learned my addr 172.21.15.28:0/1425267039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.905+0000 7f661f400640 1 -- 172.21.15.28:0/1425267039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 msgr2=0x564ea943edf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:50:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.905+0000 7f661f400640 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 0x564ea943edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.905+0000 7f661f400640 1 -- 172.21.15.28:0/1425267039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 msgr2=0x564ea943c4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:50:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.906+0000 7f661ea00640 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea943c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.906+0000 7f661f400640 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea943c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.906+0000 7f661f400640 1 -- 172.21.15.28:0/1425267039 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ea90aa210 con 0x564ea9431820 2024-04-26T01:50:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.906+0000 7f661ea00640 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea943c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:50:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.907+0000 7f661f400640 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea9431c00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6618008cd0 tx=0x7f661800d710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d3c53b272464164 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.907+0000 7f6617400640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f66040a3e40 0x7f66040a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.908+0000 7f6617400640 1 -- 172.21.15.28:0/2055613216 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f66100f03b0 con 0x7f66201003e0 2024-04-26T01:50:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.908+0000 7f660b400640 1 -- 172.21.15.28:0/1425267039 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618018680 con 0x564ea9431820 2024-04-26T01:50:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.908+0000 7f660b400640 1 -- 172.21.15.28:0/1425267039 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6618018cd0 con 0x564ea9431820 2024-04-26T01:50:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.908+0000 7f660b400640 1 -- 172.21.15.28:0/1425267039 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661800fda0 con 0x564ea9431820 2024-04-26T01:50:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.908+0000 7f661ea00640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f66040a3e40 0x7f66040a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.909+0000 7f6628574740 1 -- 172.21.15.28:0/1425267039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 msgr2=0x564ea9431c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.909+0000 7f6628574740 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea9431c00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6618008cd0 tx=0x7f661800d710 comp rx=0 tx=0).stop 2024-04-26T01:50:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.909+0000 7f6628574740 1 -- 172.21.15.28:0/1425267039 shutdown_connections 2024-04-26T01:50:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.909+0000 7f6628574740 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 0x564ea943edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.909+0000 7f6628574740 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea943c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.909+0000 7f6628574740 1 --2- 172.21.15.28:0/1425267039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea9431c00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.909+0000 7f6628574740 1 -- 172.21.15.28:0/1425267039 >> 172.21.15.28:0/1425267039 conn(0x564ea942dd30 msgr2=0x564ea942e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.910+0000 7f6628574740 1 -- 172.21.15.28:0/1425267039 shutdown_connections 2024-04-26T01:50:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.911+0000 7f661ea00640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f66040a3e40 0x7f66040a6300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f66100edcc0 tx=0x7f66100b8040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.911+0000 7f6628574740 1 -- 172.21.15.28:0/1425267039 wait complete. 2024-04-26T01:50:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.911+0000 7f6628574740 1 -- start start 2024-04-26T01:50:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.911+0000 7f6628574740 1 Processor -- start 2024-04-26T01:50:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.912+0000 7f6628574740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea94399c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.912+0000 7f6628574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9439f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.912+0000 7f661f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea94399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.912+0000 7f661f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea94399c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51534/0 (socket says 172.21.15.28:51534) 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.913+0000 7f661f400640 1 -- 172.21.15.28:0/3139515151 learned_addr learned my addr 172.21.15.28:0/3139515151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.913+0000 7f6628574740 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 0x564ea943a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.913+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ea943f630 con 0x564ea9432140 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.913+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564ea9351310 con 0x564ea943ca00 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.913+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x564ea943f7b0 con 0x564ea9431820 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.913+0000 7f6628574740 10 client.0 _ll_register_callbacks cb 0x564ea931fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.913+0000 7f661e000640 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9439f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.914+0000 7f661f400640 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 0x564ea943a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.914+0000 7f661f400640 1 -- 172.21.15.28:0/3139515151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 msgr2=0x564ea943a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.914+0000 7f661f400640 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 0x564ea943a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.914+0000 7f661f400640 1 -- 172.21.15.28:0/3139515151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 msgr2=0x564ea9439f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.914+0000 7f661f400640 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9439f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.914+0000 7f661f400640 1 -- 172.21.15.28:0/3139515151 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ea9176620 con 0x564ea9431820 2024-04-26T01:50:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.915+0000 7f661f400640 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea94399c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f661800fad0 tx=0x7f661800f900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d092933a4bb8e4d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:40.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.915+0000 7f65ffe00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:50:40.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.916+0000 7f65ffe00640 1 -- 172.21.15.28:0/3139515151 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618018680 con 0x564ea9431820 2024-04-26T01:50:40.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.917+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6618018cd0 con 0x564ea9431820 2024-04-26T01:50:40.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.917+0000 7f65ffe00640 1 -- 172.21.15.28:0/3139515151 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f6618018480 con 0x564ea9431820 2024-04-26T01:50:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.918+0000 7f65ffe00640 1 -- 172.21.15.28:0/3139515151 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6618018680 con 0x564ea9431820 2024-04-26T01:50:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.918+0000 7f6628574740 10 client.26077 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.918+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564ea944b850 con 0x564ea9431820 2024-04-26T01:50:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.918+0000 7f65ffe00640 1 -- 172.21.15.28:0/3139515151 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6618018cd0 con 0x564ea9431820 2024-04-26T01:50:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.918+0000 7f65ffe00640 1 -- 172.21.15.28:0/3139515151 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f6618027a50 con 0x564ea9431820 2024-04-26T01:50:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.918+0000 7f6628574740 10 client.26077 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.918+0000 7f6628574740 10 client.26077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.919+0000 7f6628574740 20 client.26077 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.919+0000 7f6628574740 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x564ea944cc50 0x564ea944f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.919+0000 7f6628574740 4 client.26077 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.919+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ea93ab8b0 con 0x564ea944cc50 2024-04-26T01:50:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.919+0000 7f661e000640 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x564ea944cc50 0x564ea944f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.920+0000 7f661e000640 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x564ea944cc50 0x564ea944f040 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cca06e7b3ef163f server_cookie=9cbb0dd2862722a8 in_seq=0 out_seq=0 2024-04-26T01:50:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.920+0000 7f65ffe00640 10 client.26077 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:40.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.920+0000 7f65ffe00640 1 -- 172.21.15.28:0/3139515151 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ea93ab8b0 con 0x564ea944cc50 2024-04-26T01:50:40.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.920+0000 7f65ffe00640 10 client.26077 handle_command_reply: tid=0 2024-04-26T01:50:40.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.921+0000 7f6628574740 1 client.26077 shutdown 2024-04-26T01:50:40.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.921+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 msgr2=0x564ea94399c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:40.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.921+0000 7f6628574740 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea94399c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f661800fad0 tx=0x7f661800f900 comp rx=0 tx=0).stop 2024-04-26T01:50:40.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.922+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 shutdown_connections 2024-04-26T01:50:40.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.922+0000 7f6628574740 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x564ea944cc50 0x564ea944f040 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.922+0000 7f6628574740 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea943ca00 0x564ea943a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.922+0000 7f6628574740 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9439f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.922+0000 7f6628574740 1 --2- 172.21.15.28:0/3139515151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea9431820 0x564ea94399c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.922+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 >> 172.21.15.28:0/3139515151 conn(0x564ea942dd30 msgr2=0x564ea9433ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:40.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.922+0000 7f65ffe00640 0 client.26077 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:40.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.922+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 shutdown_connections 2024-04-26T01:50:40.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.923+0000 7f6628574740 1 -- 172.21.15.28:0/3139515151 wait complete. 2024-04-26T01:50:40.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.923+0000 7f6628574740 20 client.26077 trim_cache size 0 max 16384 2024-04-26T01:50:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.953+0000 7f6628574740 1 Processor -- start 2024-04-26T01:50:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.953+0000 7f6628574740 1 -- start start 2024-04-26T01:50:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.953+0000 7f6628574740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea94484f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.953+0000 7f6628574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9434230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.953+0000 7f6628574740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 0x564ea94546f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f6628574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66201b6a60 con 0x564ea9432140 2024-04-26T01:50:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f6628574740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f66201b42d0 con 0x564ea9431820 2024-04-26T01:50:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f6628574740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f66201b2140 con 0x564ea943ca00 2024-04-26T01:50:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea94484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea94484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:58674/0 (socket says 172.21.15.28:58674) 2024-04-26T01:50:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9434230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661f400640 1 -- 172.21.15.28:0/2606223934 learned_addr learned my addr 172.21.15.28:0/2606223934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661f400640 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 0x564ea94546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661f400640 1 -- 172.21.15.28:0/2606223934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 msgr2=0x564ea94546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661f400640 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 0x564ea94546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661f400640 1 -- 172.21.15.28:0/2606223934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 msgr2=0x564ea9434230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.954+0000 7f661f400640 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9434230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.955+0000 7f661f400640 1 -- 172.21.15.28:0/2606223934 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ea9434770 con 0x564ea9431820 2024-04-26T01:50:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.955+0000 7f661e000640 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9434230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.955+0000 7f661f400640 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea94484f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6618008930 tx=0x7f66180264e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e284cfc7f0bae0a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:40.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.956+0000 7f660aa00640 1 -- 172.21.15.28:0/2606223934 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618041070 con 0x564ea9431820 2024-04-26T01:50:40.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.956+0000 7f660aa00640 1 -- 172.21.15.28:0/2606223934 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f661800fe40 con 0x564ea9431820 2024-04-26T01:50:40.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.956+0000 7f660aa00640 1 -- 172.21.15.28:0/2606223934 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618041070 con 0x564ea9431820 2024-04-26T01:50:40.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.957+0000 7f6628574740 1 -- 172.21.15.28:0/2606223934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 msgr2=0x564ea94484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.958+0000 7f6628574740 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea94484f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6618008930 tx=0x7f66180264e0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.958+0000 7f6628574740 1 -- 172.21.15.28:0/2606223934 shutdown_connections 2024-04-26T01:50:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.958+0000 7f6628574740 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 0x564ea94546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.958+0000 7f6628574740 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9434230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.958+0000 7f6628574740 1 --2- 172.21.15.28:0/2606223934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea94484f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.958+0000 7f6628574740 1 -- 172.21.15.28:0/2606223934 >> 172.21.15.28:0/2606223934 conn(0x564ea942dd30 msgr2=0x564ea942ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.958+0000 7f6628574740 1 -- 172.21.15.28:0/2606223934 shutdown_connections 2024-04-26T01:50:40.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.958+0000 7f6628574740 1 -- 172.21.15.28:0/2606223934 wait complete. 2024-04-26T01:50:40.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.959+0000 7f6628574740 1 -- start start 2024-04-26T01:50:40.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.959+0000 7f6628574740 1 Processor -- start 2024-04-26T01:50:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.960+0000 7f6628574740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea9441bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.960+0000 7f6628574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9442100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.960+0000 7f6628574740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 0x564ea94b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.960+0000 7f6628574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66201b2790 con 0x564ea9432140 2024-04-26T01:50:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.960+0000 7f6628574740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f66201b3c00 con 0x564ea9431820 2024-04-26T01:50:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.960+0000 7f6628574740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f66201b2e60 con 0x564ea943ca00 2024-04-26T01:50:40.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.960+0000 7f6628574740 10 client.0 _ll_register_callbacks cb 0x7f66180089f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:40.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.960+0000 7f661f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea9441bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.961+0000 7f661e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9442100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.961+0000 7f661f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea9441bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:58680/0 (socket says 172.21.15.28:58680) 2024-04-26T01:50:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.961+0000 7f661e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9442100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44352/0 (socket says 172.21.15.28:44352) 2024-04-26T01:50:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.961+0000 7f661e000640 1 -- 172.21.15.28:0/2256615484 learned_addr learned my addr 172.21.15.28:0/2256615484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.961+0000 7f661f400640 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 0x564ea94b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.962+0000 7f661e000640 1 -- 172.21.15.28:0/2256615484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 msgr2=0x564ea94b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.962+0000 7f661e000640 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 0x564ea94b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.962+0000 7f661e000640 1 -- 172.21.15.28:0/2256615484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 msgr2=0x564ea9441bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.962+0000 7f661e000640 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea9441bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.962+0000 7f661e000640 1 -- 172.21.15.28:0/2256615484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ea94c24b0 con 0x564ea9432140 2024-04-26T01:50:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.962+0000 7f661f400640 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea9441bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.962+0000 7f661e000640 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9442100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f660c004700 tx=0x7f660c00a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d646d2762e323b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.963+0000 7f65ffe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:50:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.963+0000 7f65ffe00640 1 -- 172.21.15.28:0/2256615484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c026060 con 0x564ea9432140 2024-04-26T01:50:40.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.963+0000 7f6628574740 1 -- 172.21.15.28:0/2256615484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564ea93a5320 con 0x564ea9432140 2024-04-26T01:50:40.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.963+0000 7f65ffe00640 1 -- 172.21.15.28:0/2256615484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f660c009040 con 0x564ea9432140 2024-04-26T01:50:40.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.965+0000 7f65ffe00640 1 -- 172.21.15.28:0/2256615484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f660c026060 con 0x564ea9432140 2024-04-26T01:50:40.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.965+0000 7f6628574740 10 client.15855 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:40.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.965+0000 7f6628574740 1 -- 172.21.15.28:0/2256615484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564ea94c3790 con 0x564ea9432140 2024-04-26T01:50:40.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.966+0000 7f65ffe00640 1 -- 172.21.15.28:0/2256615484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564ea93a5320 con 0x564ea9432140 2024-04-26T01:50:40.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.966+0000 7f65ffe00640 1 -- 172.21.15.28:0/2256615484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f660c02a3a0 con 0x564ea9432140 2024-04-26T01:50:40.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.966+0000 7f6628574740 10 client.15855 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:40.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.966+0000 7f6628574740 10 client.15855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:40.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.966+0000 7f6628574740 20 client.15855 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:40.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.966+0000 7f6628574740 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x564ea94c3f00 0x564ea94c6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:40.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.966+0000 7f6628574740 4 client.15855 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:40.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.966+0000 7f6628574740 1 -- 172.21.15.28:0/2256615484 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ea939f190 con 0x564ea94c3f00 2024-04-26T01:50:40.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.967+0000 7f661f400640 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x564ea94c3f00 0x564ea94c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:40.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.967+0000 7f661f400640 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x564ea94c3f00 0x564ea94c6320 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eae89b0fee99c5b8 server_cookie=e241b3d30606ae36 in_seq=0 out_seq=0 2024-04-26T01:50:40.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.967+0000 7f65ffe00640 10 client.15855 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.981+0000 7f65ffe00640 1 -- 172.21.15.28:0/2256615484 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ea939f190 con 0x564ea94c3f00 2024-04-26T01:50:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.981+0000 7f65ffe00640 10 client.15855 handle_command_reply: tid=0 2024-04-26T01:50:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.981+0000 7f6628574740 1 client.15855 shutdown 2024-04-26T01:50:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.981+0000 7f6628574740 1 -- 172.21.15.28:0/2256615484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 msgr2=0x564ea9442100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.981+0000 7f6628574740 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9442100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f660c004700 tx=0x7f660c00a980 comp rx=0 tx=0).stop 2024-04-26T01:50:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 1 -- 172.21.15.28:0/2256615484 shutdown_connections 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x564ea94c3f00 0x564ea94c6320 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564ea943ca00 0x564ea94b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ea9432140 0x564ea9442100 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f65ffe00640 0 client.15855 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 1 --2- 172.21.15.28:0/2256615484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ea9431820 0x564ea9441bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 1 -- 172.21.15.28:0/2256615484 >> 172.21.15.28:0/2256615484 conn(0x564ea942dd30 msgr2=0x564ea9439960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 1 -- 172.21.15.28:0/2256615484 shutdown_connections 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 1 -- 172.21.15.28:0/2256615484 wait complete. 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.982+0000 7f6628574740 20 client.15855 trim_cache size 0 max 16384 2024-04-26T01:50:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:50:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:50:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:50:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:50:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 -- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f66040a3e40 msgr2=0x7f66040a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f66040a3e40 0x7f66040a6300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f66100edcc0 tx=0x7f66100b8040 comp rx=0 tx=0).stop 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 -- 172.21.15.28:0/2055613216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 msgr2=0x7f662010bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 0x7f662010bc70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6610035040 tx=0x7f6610032dc0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 -- 172.21.15.28:0/2055613216 shutdown_connections 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f66040a3e40 0x7f66040a6300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66201a4790 0x7f66201a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66201054b0 0x7f662010c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 --2- 172.21.15.28:0/2055613216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66201003e0 0x7f662010bc70 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6610035040 tx=0x7f6610032dc0 comp rx=0 tx=0).stop 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.983+0000 7f6625600640 1 -- 172.21.15.28:0/2055613216 >> 172.21.15.28:0/2055613216 conn(0x7f6620076f50 msgr2=0x7f6620111c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.984+0000 7f6625600640 1 -- 172.21.15.28:0/2055613216 shutdown_connections 2024-04-26T01:50:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:40.984+0000 7f6625600640 1 -- 172.21.15.28:0/2055613216 wait complete. 2024-04-26T01:50:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 525.0 s, 2041.4 tps, lat 48.940 ms stddev 123.477 2024-04-26T01:50:41.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:41 smithi169 ceph-mon[32851]: pgmap v302: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 27 MiB/s wr, 1.90k op/s 2024-04-26T01:50:41.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:41 smithi053 ceph-mon[32593]: pgmap v302: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 27 MiB/s wr, 1.90k op/s 2024-04-26T01:50:41.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:41 smithi028 ceph-mon[28247]: pgmap v302: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 27 MiB/s wr, 1.90k op/s 2024-04-26T01:50:43.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:43 smithi169 ceph-mon[32851]: pgmap v303: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 MiB/s wr, 1.63k op/s 2024-04-26T01:50:43.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:43 smithi053 ceph-mon[32593]: pgmap v303: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 MiB/s wr, 1.63k op/s 2024-04-26T01:50:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:43 smithi028 ceph-mon[28247]: pgmap v303: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 MiB/s wr, 1.63k op/s 2024-04-26T01:50:45.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:45 smithi169 ceph-mon[32851]: pgmap v304: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 MiB/s wr, 1.79k op/s 2024-04-26T01:50:45.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:45 smithi053 ceph-mon[32593]: pgmap v304: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 MiB/s wr, 1.79k op/s 2024-04-26T01:50:45.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:45 smithi028 ceph-mon[28247]: pgmap v304: 97 pgs: 97 active+clean; 3.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 27 MiB/s wr, 1.79k op/s 2024-04-26T01:50:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 530.0 s, 2439.2 tps, lat 41.097 ms stddev 50.482 2024-04-26T01:50:47.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:47 smithi169 ceph-mon[32851]: pgmap v305: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 27 MiB/s wr, 1.81k op/s 2024-04-26T01:50:47.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:47 smithi053 ceph-mon[32593]: pgmap v305: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 27 MiB/s wr, 1.81k op/s 2024-04-26T01:50:47.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:47 smithi028 ceph-mon[28247]: pgmap v305: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 27 MiB/s wr, 1.81k op/s 2024-04-26T01:50:49.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:49 smithi169 ceph-mon[32851]: pgmap v306: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 26 MiB/s wr, 1.71k op/s 2024-04-26T01:50:49.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:49 smithi053 ceph-mon[32593]: pgmap v306: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 26 MiB/s wr, 1.71k op/s 2024-04-26T01:50:49.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:49 smithi028 ceph-mon[28247]: pgmap v306: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 26 MiB/s wr, 1.71k op/s 2024-04-26T01:50:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 535.0 s, 2133.6 tps, lat 46.614 ms stddev 60.060 2024-04-26T01:50:51.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:51 smithi169 ceph-mon[32851]: pgmap v307: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 29 MiB/s wr, 1.98k op/s 2024-04-26T01:50:51.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:51 smithi053 ceph-mon[32593]: pgmap v307: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 29 MiB/s wr, 1.98k op/s 2024-04-26T01:50:51.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:51 smithi028 ceph-mon[28247]: pgmap v307: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 29 MiB/s wr, 1.98k op/s 2024-04-26T01:50:53.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:53 smithi169 ceph-mon[32851]: pgmap v308: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 5.8 MiB/s wr, 579 op/s 2024-04-26T01:50:53.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:53 smithi053 ceph-mon[32593]: pgmap v308: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 5.8 MiB/s wr, 579 op/s 2024-04-26T01:50:53.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:53 smithi028 ceph-mon[28247]: pgmap v308: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 5.8 MiB/s wr, 579 op/s 2024-04-26T01:50:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.174+0000 7fc69c400640 1 -- 172.21.15.28:0/1594802287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc694074b70 msgr2=0x7fc69410e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.174+0000 7fc69c400640 1 --2- 172.21.15.28:0/1594802287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc694074b70 0x7fc69410e300 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc680009860 tx=0x7fc68002a3d0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.174+0000 7fc69c400640 1 -- 172.21.15.28:0/1594802287 shutdown_connections 2024-04-26T01:50:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.174+0000 7fc69c400640 1 --2- 172.21.15.28:0/1594802287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc69410e840 0x7fc694110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.175+0000 7fc69c400640 1 --2- 172.21.15.28:0/1594802287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc694074b70 0x7fc69410e300 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.175+0000 7fc69c400640 1 --2- 172.21.15.28:0/1594802287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc694074250 0x7fc694074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.175+0000 7fc69c400640 1 -- 172.21.15.28:0/1594802287 >> 172.21.15.28:0/1594802287 conn(0x7fc6940793e0 msgr2=0x7fc6940797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.175+0000 7fc69c400640 1 -- 172.21.15.28:0/1594802287 shutdown_connections 2024-04-26T01:50:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.175+0000 7fc69c400640 1 -- 172.21.15.28:0/1594802287 wait complete. 2024-04-26T01:50:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.177+0000 7fc69c400640 1 Processor -- start 2024-04-26T01:50:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.177+0000 7fc69c400640 1 -- start start 2024-04-26T01:50:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.179+0000 7fc69c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc694074250 0x7fc694109060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.179+0000 7fc69c400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc694074b70 0x7fc6941095a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.179+0000 7fc69c400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc69410e840 0x7fc6941041f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.179+0000 7fc699000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc694074b70 0x7fc6941095a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.179+0000 7fc699000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc694074b70 0x7fc6941095a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:56322/0 (socket says 172.21.15.28:56322) 2024-04-26T01:50:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.179+0000 7fc699a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc694074250 0x7fc694109060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc699000640 1 -- 172.21.15.28:0/751528296 learned_addr learned my addr 172.21.15.28:0/751528296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc694102f20 con 0x7fc694074250 2024-04-26T01:50:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc694102da0 con 0x7fc69410e840 2024-04-26T01:50:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc6941030a0 con 0x7fc694074b70 2024-04-26T01:50:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69a400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc69410e840 0x7fc6941041f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69a400640 1 -- 172.21.15.28:0/751528296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc694074b70 msgr2=0x7fc6941095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69a400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc694074b70 0x7fc6941095a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69a400640 1 -- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc694074250 msgr2=0x7fc694109060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69a400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc694074250 0x7fc694109060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.180+0000 7fc69a400640 1 -- 172.21.15.28:0/751528296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc694104ab0 con 0x7fc69410e840 2024-04-26T01:50:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.181+0000 7fc69a400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc69410e840 0x7fc6941041f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d590 tx=0x7fc68800da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.182+0000 7fc692000640 1 -- 172.21.15.28:0/751528296 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688017070 con 0x7fc69410e840 2024-04-26T01:50:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.182+0000 7fc692000640 1 -- 172.21.15.28:0/751528296 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6880047a0 con 0x7fc69410e840 2024-04-26T01:50:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.182+0000 7fc692000640 1 -- 172.21.15.28:0/751528296 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688009b30 con 0x7fc69410e840 2024-04-26T01:50:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.182+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc694104da0 con 0x7fc69410e840 2024-04-26T01:50:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.182+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69410cf40 con 0x7fc69410e840 2024-04-26T01:50:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.188+0000 7fc692000640 1 -- 172.21.15.28:0/751528296 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc688009cd0 con 0x7fc69410e840 2024-04-26T01:50:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.189+0000 7fc69f20c740 1 Processor -- start 2024-04-26T01:50:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.190+0000 7fc69f20c740 1 -- start start 2024-04-26T01:50:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.191+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf8820 0x559f80cf8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.191+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d034c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.192+0000 7fc692000640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc67c0a3e40 0x7fc67c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.192+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d05df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.192+0000 7fc69f20c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559f80c5e900 con 0x559f80d03a00 2024-04-26T01:50:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.192+0000 7fc69f20c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559f80c04ef0 con 0x559f80cf9140 2024-04-26T01:50:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.192+0000 7fc69f20c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x559f80c5a8c0 con 0x559f80cf8820 2024-04-26T01:50:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.192+0000 7fc692000640 1 -- 172.21.15.28:0/751528296 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fc6880d0560 con 0x7fc69410e840 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf8820 0x559f80cf8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf8820 0x559f80cf8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:56334/0 (socket says 172.21.15.28:56334) 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699000640 1 -- 172.21.15.28:0/1195076656 learned_addr learned my addr 172.21.15.28:0/1195076656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699a00640 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc69a400640 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d05df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699a00640 1 -- 172.21.15.28:0/1195076656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf8820 msgr2=0x559f80cf8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699a00640 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf8820 0x559f80cf8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699a00640 1 -- 172.21.15.28:0/1195076656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 msgr2=0x559f80d05df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699a00640 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d05df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.193+0000 7fc699a00640 1 -- 172.21.15.28:0/1195076656 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f80971210 con 0x559f80cf9140 2024-04-26T01:50:54.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.194+0000 7fc699a00640 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d034c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00e4d0 tx=0x7fc68c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=698af4101d107121 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:54.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.194+0000 7fc699000640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc67c0a3e40 0x7fc67c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.194+0000 7fc686a00640 1 -- 172.21.15.28:0/1195076656 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c01d070 con 0x559f80cf9140 2024-04-26T01:50:54.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.194+0000 7fc686a00640 1 -- 172.21.15.28:0/1195076656 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc68c003e40 con 0x559f80cf9140 2024-04-26T01:50:54.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.194+0000 7fc686a00640 1 -- 172.21.15.28:0/1195076656 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c00dad0 con 0x559f80cf9140 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.195+0000 7fc69f20c740 1 -- 172.21.15.28:0/1195076656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 msgr2=0x559f80d034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.195+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d034c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00e4d0 tx=0x7fc68c0095c0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.196+0000 7fc699000640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc67c0a3e40 0x7fc67c0a6300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc680032ff0 tx=0x7fc68002a3d0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.196+0000 7fc69f20c740 1 -- 172.21.15.28:0/1195076656 shutdown_connections 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.196+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d05df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.196+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d034c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.196+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1195076656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf8820 0x559f80cf8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.196+0000 7fc69f20c740 1 -- 172.21.15.28:0/1195076656 >> 172.21.15.28:0/1195076656 conn(0x559f80cf4d30 msgr2=0x559f80cf5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.196+0000 7fc69f20c740 1 -- 172.21.15.28:0/1195076656 shutdown_connections 2024-04-26T01:50:54.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.196+0000 7fc69f20c740 1 -- 172.21.15.28:0/1195076656 wait complete. 2024-04-26T01:50:54.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.197+0000 7fc69f20c740 1 -- start start 2024-04-26T01:50:54.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.197+0000 7fc69f20c740 1 Processor -- start 2024-04-26T01:50:54.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.198+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 0x559f80d00950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.198+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d00e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.198+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80d03a00 0x559f80d013d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.198+0000 7fc69f20c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559f80d06630 con 0x559f80cf8820 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.198+0000 7fc69f20c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559f80c18310 con 0x559f80cf9140 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.198+0000 7fc69f20c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x559f80d067b0 con 0x559f80d03a00 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.198+0000 7fc69f20c740 10 client.0 _ll_register_callbacks cb 0x559f80be6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.199+0000 7fc699000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d00e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.199+0000 7fc699000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d00e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:56612/0 (socket says 172.21.15.28:56612) 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.199+0000 7fc699000640 1 -- 172.21.15.28:0/2562876232 learned_addr learned my addr 172.21.15.28:0/2562876232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.199+0000 7fc699a00640 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 0x559f80d00950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.199+0000 7fc69a400640 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80d03a00 0x559f80d013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.200+0000 7fc699000640 1 -- 172.21.15.28:0/2562876232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80d03a00 msgr2=0x559f80d013d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.200+0000 7fc699000640 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80d03a00 0x559f80d013d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.200+0000 7fc699000640 1 -- 172.21.15.28:0/2562876232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 msgr2=0x559f80d00950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.200+0000 7fc699000640 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 0x559f80d00950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.200+0000 7fc699000640 1 -- 172.21.15.28:0/2562876232 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f80a3d620 con 0x559f80cf9140 2024-04-26T01:50:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.200+0000 7fc699000640 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d00e90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc680009530 tx=0x7fc680005cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=380abfe5b34f6aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.200+0000 7fc67b400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:50:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.201+0000 7fc67b400640 1 -- 172.21.15.28:0/2562876232 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680047040 con 0x559f80cf9140 2024-04-26T01:50:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.201+0000 7fc67b400640 1 -- 172.21.15.28:0/2562876232 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fc6800048c0 con 0x559f80cf9140 2024-04-26T01:50:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.201+0000 7fc69f20c740 1 -- 172.21.15.28:0/2562876232 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559f80c728b0 con 0x559f80cf9140 2024-04-26T01:50:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.202+0000 7fc67b400640 1 -- 172.21.15.28:0/2562876232 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc680004ac0 con 0x559f80cf9140 2024-04-26T01:50:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.202+0000 7fc69f20c740 10 client.26069 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:54.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.202+0000 7fc69f20c740 1 -- 172.21.15.28:0/2562876232 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x559f80d12840 con 0x559f80cf9140 2024-04-26T01:50:54.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.203+0000 7fc67b400640 1 -- 172.21.15.28:0/2562876232 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x559f80c728b0 con 0x559f80cf9140 2024-04-26T01:50:54.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.203+0000 7fc67b400640 1 -- 172.21.15.28:0/2562876232 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fc680040b70 con 0x559f80cf9140 2024-04-26T01:50:54.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.203+0000 7fc69f20c740 10 client.26069 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.203+0000 7fc69f20c740 10 client.26069 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.203+0000 7fc69f20c740 20 client.26069 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.204+0000 7fc69f20c740 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559f80d13e10 0x559f80d16220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.204+0000 7fc69f20c740 4 client.26069 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.204+0000 7fc69f20c740 1 -- 172.21.15.28:0/2562876232 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f80c6c320 con 0x559f80d13e10 2024-04-26T01:50:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.204+0000 7fc699a00640 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559f80d13e10 0x559f80d16220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.205+0000 7fc699a00640 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559f80d13e10 0x559f80d16220 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1df17bc03a21a783 server_cookie=99405ada3b1a7b67 in_seq=0 out_seq=0 2024-04-26T01:50:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.205+0000 7fc67b400640 10 client.26069 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.206+0000 7fc67b400640 1 -- 172.21.15.28:0/2562876232 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f80c6c320 con 0x559f80d13e10 2024-04-26T01:50:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.206+0000 7fc67b400640 10 client.26069 handle_command_reply: tid=0 2024-04-26T01:50:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.206+0000 7fc69f20c740 1 client.26069 shutdown 2024-04-26T01:50:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.206+0000 7fc69f20c740 1 -- 172.21.15.28:0/2562876232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 msgr2=0x559f80d00e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.206+0000 7fc69f20c740 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d00e90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc680009530 tx=0x7fc680005cb0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.207+0000 7fc69f20c740 1 -- 172.21.15.28:0/2562876232 shutdown_connections 2024-04-26T01:50:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.207+0000 7fc69f20c740 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559f80d13e10 0x559f80d16220 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.207+0000 7fc69f20c740 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80d03a00 0x559f80d013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.207+0000 7fc69f20c740 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf9140 0x559f80d00e90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.207+0000 7fc69f20c740 1 --2- 172.21.15.28:0/2562876232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 0x559f80d00950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.207+0000 7fc69f20c740 1 -- 172.21.15.28:0/2562876232 >> 172.21.15.28:0/2562876232 conn(0x559f80cf4d30 msgr2=0x559f80cfaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.207+0000 7fc67b400640 0 client.26069 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:54.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.209+0000 7fc69f20c740 1 -- 172.21.15.28:0/2562876232 shutdown_connections 2024-04-26T01:50:54.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.209+0000 7fc69f20c740 1 -- 172.21.15.28:0/2562876232 wait complete. 2024-04-26T01:50:54.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.209+0000 7fc69f20c740 20 client.26069 trim_cache size 0 max 16384 2024-04-26T01:50:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.247+0000 7fc69f20c740 1 Processor -- start 2024-04-26T01:50:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.248+0000 7fc69f20c740 1 -- start start 2024-04-26T01:50:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.248+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 0x559f80d0f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.248+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d1bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.248+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80d03a00 0x559f80d1c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.248+0000 7fc69f20c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6941bf1d0 con 0x559f80cf8820 2024-04-26T01:50:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.248+0000 7fc69f20c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc6941bca40 con 0x559f80d03a00 2024-04-26T01:50:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.248+0000 7fc69f20c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc6941ba8b0 con 0x559f80cf9140 2024-04-26T01:50:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.249+0000 7fc69a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80d03a00 0x559f80d1c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.249+0000 7fc69a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80d03a00 0x559f80d1c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:56628/0 (socket says 172.21.15.28:56628) 2024-04-26T01:50:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.249+0000 7fc69a400640 1 -- 172.21.15.28:0/759453239 learned_addr learned my addr 172.21.15.28:0/759453239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.249+0000 7fc699a00640 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 0x559f80d0f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.250+0000 7fc69a400640 1 -- 172.21.15.28:0/759453239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 msgr2=0x559f80d1bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.249+0000 7fc699000640 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d1bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.250+0000 7fc69a400640 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d1bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.250+0000 7fc69a400640 1 -- 172.21.15.28:0/759453239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 msgr2=0x559f80d0f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.250+0000 7fc69a400640 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 0x559f80d0f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.250+0000 7fc69a400640 1 -- 172.21.15.28:0/759453239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f80cf0690 con 0x559f80d03a00 2024-04-26T01:50:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.250+0000 7fc699000640 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d1bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:50:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.251+0000 7fc69a400640 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80d03a00 0x559f80d1c070 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc68809cdf0 tx=0x7fc6880bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0985442bf2a0ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.254+0000 7fc686000640 1 -- 172.21.15.28:0/759453239 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688020040 con 0x559f80d03a00 2024-04-26T01:50:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.254+0000 7fc686000640 1 -- 172.21.15.28:0/759453239 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6880b6070 con 0x559f80d03a00 2024-04-26T01:50:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.254+0000 7fc686000640 1 -- 172.21.15.28:0/759453239 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688005270 con 0x559f80d03a00 2024-04-26T01:50:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.255+0000 7fc69f20c740 1 -- 172.21.15.28:0/759453239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80d03a00 msgr2=0x559f80d1c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.255+0000 7fc69f20c740 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80d03a00 0x559f80d1c070 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc68809cdf0 tx=0x7fc6880bd040 comp rx=0 tx=0).stop 2024-04-26T01:50:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.257+0000 7fc69f20c740 1 -- 172.21.15.28:0/759453239 shutdown_connections 2024-04-26T01:50:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.257+0000 7fc69f20c740 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80d03a00 0x559f80d1c070 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.257+0000 7fc69f20c740 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d1bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.257+0000 7fc69f20c740 1 --2- 172.21.15.28:0/759453239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80cf8820 0x559f80d0f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.257+0000 7fc69f20c740 1 -- 172.21.15.28:0/759453239 >> 172.21.15.28:0/759453239 conn(0x559f80cf4d30 msgr2=0x559f80cf5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.258+0000 7fc69f20c740 1 -- 172.21.15.28:0/759453239 shutdown_connections 2024-04-26T01:50:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.258+0000 7fc69f20c740 1 -- 172.21.15.28:0/759453239 wait complete. 2024-04-26T01:50:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.259+0000 7fc69f20c740 1 -- start start 2024-04-26T01:50:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.259+0000 7fc69f20c740 1 Processor -- start 2024-04-26T01:50:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.260+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf8820 0x559f80d0b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.260+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d0bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.260+0000 7fc69f20c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d09bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.260+0000 7fc699a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf8820 0x559f80d0b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.260+0000 7fc699a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf8820 0x559f80d0b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:56630/0 (socket says 172.21.15.28:56630) 2024-04-26T01:50:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.260+0000 7fc69a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d09bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc699000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d0bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc699000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d0bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:56352/0 (socket says 172.21.15.28:56352) 2024-04-26T01:50:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc69a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d09bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57678/0 (socket says 172.21.15.28:57678) 2024-04-26T01:50:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc699a00640 1 -- 172.21.15.28:0/1221123562 learned_addr learned my addr 172.21.15.28:0/1221123562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:50:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6941baf00 con 0x559f80d03a00 2024-04-26T01:50:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc6941bc370 con 0x559f80cf8820 2024-04-26T01:50:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc6941bb5d0 con 0x559f80cf9140 2024-04-26T01:50:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc69f20c740 10 client.0 _ll_register_callbacks cb 0x7fc680030d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc699a00640 1 -- 172.21.15.28:0/1221123562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 msgr2=0x559f80d0bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc699a00640 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d0bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc699a00640 1 -- 172.21.15.28:0/1221123562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 msgr2=0x559f80d09bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc699a00640 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d09bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.261+0000 7fc699a00640 1 -- 172.21.15.28:0/1221123562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f80d0a340 con 0x559f80cf8820 2024-04-26T01:50:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.262+0000 7fc67b400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:50:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.262+0000 7fc699a00640 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf8820 0x559f80d0b710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc68c000f80 tx=0x7fc68c0093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511f76578ca8559a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.263+0000 7fc67b400640 1 -- 172.21.15.28:0/1221123562 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c02f060 con 0x559f80cf8820 2024-04-26T01:50:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.263+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6880b6070 con 0x559f80cf8820 2024-04-26T01:50:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.263+0000 7fc67b400640 1 -- 172.21.15.28:0/1221123562 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fc68c004cd0 con 0x559f80cf8820 2024-04-26T01:50:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.264+0000 7fc67b400640 1 -- 172.21.15.28:0/1221123562 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc68c02f060 con 0x559f80cf8820 2024-04-26T01:50:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.264+0000 7fc69f20c740 10 client.26075 fetch_fsmap learned FSMap version 40 2024-04-26T01:50:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.264+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x559f80d7ad80 con 0x559f80cf8820 2024-04-26T01:50:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.264+0000 7fc67b400640 1 -- 172.21.15.28:0/1221123562 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6880b6070 con 0x559f80cf8820 2024-04-26T01:50:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.265+0000 7fc67b400640 1 -- 172.21.15.28:0/1221123562 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fc68c00b040 con 0x559f80cf8820 2024-04-26T01:50:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.265+0000 7fc69f20c740 10 client.26075 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:50:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.265+0000 7fc69f20c740 10 client.26075 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.265+0000 7fc69f20c740 20 client.26075 populate_metadata read hostname 'smithi028' 2024-04-26T01:50:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.265+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559f80d7b2c0 0x559f80d7d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.265+0000 7fc69f20c740 4 client.26075 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.265+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f80c66190 con 0x559f80d7b2c0 2024-04-26T01:50:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.266+0000 7fc699000640 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559f80d7b2c0 0x559f80d7d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.267+0000 7fc67b400640 10 client.26075 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.267+0000 7fc699000640 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559f80d7b2c0 0x559f80d7d6e0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2db86d7717543b98 server_cookie=d185b493fc10d2f in_seq=0 out_seq=0 2024-04-26T01:50:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.310+0000 7fc67b400640 1 -- 172.21.15.28:0/1221123562 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559f80c66190 con 0x559f80d7b2c0 2024-04-26T01:50:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.310+0000 7fc67b400640 10 client.26075 handle_command_reply: tid=0 2024-04-26T01:50:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.310+0000 7fc69f20c740 1 client.26075 shutdown 2024-04-26T01:50:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.310+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf8820 msgr2=0x559f80d0b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.310+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf8820 0x559f80d0b710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc68c000f80 tx=0x7fc68c0093d0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 shutdown_connections 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x559f80d7b2c0 0x559f80d7d6e0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f80d03a00 0x559f80d09bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f80cf9140 0x559f80d0bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 1 --2- 172.21.15.28:0/1221123562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559f80cf8820 0x559f80d0b710 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 >> 172.21.15.28:0/1221123562 conn(0x559f80cf4d30 msgr2=0x559f80cfa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc67b400640 0 client.26075 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 shutdown_connections 2024-04-26T01:50:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 1 -- 172.21.15.28:0/1221123562 wait complete. 2024-04-26T01:50:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.311+0000 7fc69f20c740 20 client.26075 trim_cache size 0 max 16384 2024-04-26T01:50:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:50:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:50:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:50:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc67c0a3e40 msgr2=0x7fc67c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc67c0a3e40 0x7fc67c0a6300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc680032ff0 tx=0x7fc68002a3d0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc69410e840 msgr2=0x7fc6941041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc69410e840 0x7fc6941041f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d590 tx=0x7fc68800da50 comp rx=0 tx=0).stop 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 shutdown_connections 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc67c0a3e40 0x7fc67c0a6300 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc69410e840 0x7fc6941041f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc694074b70 0x7fc6941095a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 --2- 172.21.15.28:0/751528296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc694074250 0x7fc694109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.313+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 >> 172.21.15.28:0/751528296 conn(0x7fc6940793e0 msgr2=0x7fc6940fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.314+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 shutdown_connections 2024-04-26T01:50:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:50:54.314+0000 7fc69c400640 1 -- 172.21.15.28:0/751528296 wait complete. 2024-04-26T01:50:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:55 smithi169 ceph-mon[32851]: pgmap v309: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.4 MiB/s wr, 713 op/s 2024-04-26T01:50:55.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:55 smithi053 ceph-mon[32593]: pgmap v309: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.4 MiB/s wr, 713 op/s 2024-04-26T01:50:55.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:55 smithi028 ceph-mon[28247]: pgmap v309: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.4 MiB/s wr, 713 op/s 2024-04-26T01:50:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 540.0 s, 2359.0 tps, lat 42.458 ms stddev 48.889 2024-04-26T01:50:57.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:57 smithi169 ceph-mon[32851]: pgmap v310: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.4 MiB/s wr, 793 op/s 2024-04-26T01:50:57.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:57 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:57.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:57 smithi053 ceph-mon[32593]: pgmap v310: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.4 MiB/s wr, 793 op/s 2024-04-26T01:50:57.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:57 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:57.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:57 smithi028 ceph-mon[28247]: pgmap v310: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.4 MiB/s wr, 793 op/s 2024-04-26T01:50:57.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:57 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:59.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:50:59 smithi169 ceph-mon[32851]: pgmap v311: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 7.0 MiB/s wr, 647 op/s 2024-04-26T01:50:59.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:50:59 smithi053 ceph-mon[32593]: pgmap v311: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 7.0 MiB/s wr, 647 op/s 2024-04-26T01:50:59.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:50:59 smithi028 ceph-mon[28247]: pgmap v311: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 7.0 MiB/s wr, 647 op/s 2024-04-26T01:51:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 545.0 s, 1888.9 tps, lat 49.464 ms stddev 78.160 2024-04-26T01:51:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:01 smithi169 ceph-mon[32851]: pgmap v312: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 9.6 MiB/s wr, 918 op/s 2024-04-26T01:51:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:01 smithi053 ceph-mon[32593]: pgmap v312: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 9.6 MiB/s wr, 918 op/s 2024-04-26T01:51:01.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:01 smithi028 ceph-mon[28247]: pgmap v312: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 9.6 MiB/s wr, 918 op/s 2024-04-26T01:51:02.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:02 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:02.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:02 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:02.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:02 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:03 smithi169 ceph-mon[32851]: pgmap v313: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 642 op/s 2024-04-26T01:51:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:03 smithi169 ceph-mon[32851]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:51:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:03 smithi053 ceph-mon[32593]: pgmap v313: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 642 op/s 2024-04-26T01:51:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:03 smithi053 ceph-mon[32593]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:51:03.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:03 smithi028 ceph-mon[28247]: pgmap v313: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 642 op/s 2024-04-26T01:51:03.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:03 smithi028 ceph-mon[28247]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:51:05.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:05 smithi053 ceph-mon[32593]: pgmap v315: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 7.1 MiB/s wr, 703 op/s 2024-04-26T01:51:05.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:05 smithi028 ceph-mon[28247]: pgmap v315: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 7.1 MiB/s wr, 703 op/s 2024-04-26T01:51:05.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:05 smithi169 ceph-mon[32851]: pgmap v315: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 7.1 MiB/s wr, 703 op/s 2024-04-26T01:51:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 550.0 s, 131.0 tps, lat 739.841 ms stddev 470.636 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 -- 172.21.15.28:0/148466997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a4103aa0 msgr2=0x7f95a410bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 --2- 172.21.15.28:0/148466997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a4103aa0 0x7f95a410bf80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9590009900 tx=0x7f959002a490 comp rx=0 tx=0).stop 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 -- 172.21.15.28:0/148466997 shutdown_connections 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 --2- 172.21.15.28:0/148466997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95a410c550 0x7f95a410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 --2- 172.21.15.28:0/148466997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a4103aa0 0x7f95a410bf80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 --2- 172.21.15.28:0/148466997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a4103180 0x7f95a4103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 -- 172.21.15.28:0/148466997 >> 172.21.15.28:0/148466997 conn(0x7f95a4077fd0 msgr2=0x7f95a40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 -- 172.21.15.28:0/148466997 shutdown_connections 2024-04-26T01:51:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.487+0000 7f95ab800640 1 -- 172.21.15.28:0/148466997 wait complete. 2024-04-26T01:51:07.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.490+0000 7f95ab800640 1 Processor -- start 2024-04-26T01:51:07.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.490+0000 7f95ab800640 1 -- start start 2024-04-26T01:51:07.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.491+0000 7f95ab800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a4103180 0x7f95a4108030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.491+0000 7f95ab800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95a4103aa0 0x7f95a4108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.491+0000 7f95ab800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a410c550 0x7f95a419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.491+0000 7f95ab800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95a4112200 con 0x7f95a410c550 2024-04-26T01:51:07.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.491+0000 7f95ab800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f95a4112080 con 0x7f95a4103aa0 2024-04-26T01:51:07.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.491+0000 7f95ab800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f95a4112380 con 0x7f95a4103180 2024-04-26T01:51:07.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.491+0000 7f95a9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a410c550 0x7f95a419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a410c550 0x7f95a419ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55376/0 (socket says 172.21.15.28:55376) 2024-04-26T01:51:07.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a9800640 1 -- 172.21.15.28:0/602680216 learned_addr learned my addr 172.21.15.28:0/602680216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:07.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a3e00640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95a4103aa0 0x7f95a4108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a8e00640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a4103180 0x7f95a4108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a9800640 1 -- 172.21.15.28:0/602680216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a4103180 msgr2=0x7f95a4108030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:07.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a9800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a4103180 0x7f95a4108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a9800640 1 -- 172.21.15.28:0/602680216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95a4103aa0 msgr2=0x7f95a4108570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:07.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a9800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95a4103aa0 0x7f95a4108570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.492+0000 7f95a9800640 1 -- 172.21.15.28:0/602680216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a419b390 con 0x7f95a410c550 2024-04-26T01:51:07.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.493+0000 7f95a9800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a410c550 0x7f95a419ae50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f959c00ed70 tx=0x7f959c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:07.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.496+0000 7f95a1600640 1 -- 172.21.15.28:0/602680216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c004b20 con 0x7f95a410c550 2024-04-26T01:51:07.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.497+0000 7f95ab800640 1 -- 172.21.15.28:0/602680216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a419b620 con 0x7f95a410c550 2024-04-26T01:51:07.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.497+0000 7f95ab800640 1 -- 172.21.15.28:0/602680216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a419bb40 con 0x7f95a410c550 2024-04-26T01:51:07.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.497+0000 7f95a1600640 1 -- 172.21.15.28:0/602680216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f959c004cc0 con 0x7f95a410c550 2024-04-26T01:51:07.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.497+0000 7f95a1600640 1 -- 172.21.15.28:0/602680216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c010760 con 0x7f95a410c550 2024-04-26T01:51:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.502+0000 7f95ae767740 1 Processor -- start 2024-04-26T01:51:07.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.503+0000 7f95a1600640 1 -- 172.21.15.28:0/602680216 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f959c010900 con 0x7f95a410c550 2024-04-26T01:51:07.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.504+0000 7f95ae767740 1 -- start start 2024-04-26T01:51:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.504+0000 7f95ae767740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53fab4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.504+0000 7f95ae767740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab5140 0x55d53fabf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.504+0000 7f95ae767740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.504+0000 7f95ae767740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d53fa1a900 con 0x55d53fab5140 2024-04-26T01:51:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.504+0000 7f95ae767740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d53f9c0ef0 con 0x55d53fab4820 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.504+0000 7f95ae767740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d53fa168c0 con 0x55d53fabfa00 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.504+0000 7f95a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53fab4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.505+0000 7f95a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53fab4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:33970/0 (socket says 172.21.15.28:33970) 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.505+0000 7f95a3e00640 1 -- 172.21.15.28:0/1370910261 learned_addr learned my addr 172.21.15.28:0/1370910261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.506+0000 7f95a8e00640 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab5140 0x55d53fabf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.506+0000 7f95a1600640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f958c0a3dc0 0x7f958c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.506+0000 7f95a1600640 1 -- 172.21.15.28:0/602680216 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8557+0+0 (secure 0 0 0) 0x7f959c0d0e50 con 0x7f95a410c550 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.507+0000 7f95a3e00640 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.508+0000 7f95a9800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f958c0a3dc0 0x7f958c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.508+0000 7f95a8e00640 1 -- 172.21.15.28:0/1370910261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 msgr2=0x55d53fac1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.508+0000 7f95a8e00640 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.508+0000 7f95a8e00640 1 -- 172.21.15.28:0/1370910261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 msgr2=0x55d53fab4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.508+0000 7f95a8e00640 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53fab4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.508+0000 7f95a8e00640 1 -- 172.21.15.28:0/1370910261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d53f72d210 con 0x55d53fab5140 2024-04-26T01:51:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.509+0000 7f95a3e00640 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:51:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.509+0000 7f95a8e00640 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab5140 0x55d53fabf4c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9598009b90 tx=0x7f9598010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c59564c4934d0a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.509+0000 7f95a9800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f958c0a3dc0 0x7f958c0a6280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f959c09cad0 tx=0x7f959c0bbf90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:07.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.511+0000 7f9596000640 1 -- 172.21.15.28:0/1370910261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9598004020 con 0x55d53fab5140 2024-04-26T01:51:07.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.513+0000 7f9596000640 1 -- 172.21.15.28:0/1370910261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95980041c0 con 0x55d53fab5140 2024-04-26T01:51:07.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.513+0000 7f9596000640 1 -- 172.21.15.28:0/1370910261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95980126b0 con 0x55d53fab5140 2024-04-26T01:51:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.516+0000 7f95ae767740 1 -- 172.21.15.28:0/1370910261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab5140 msgr2=0x55d53fabf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.517+0000 7f95ae767740 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab5140 0x55d53fabf4c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9598009b90 tx=0x7f9598010740 comp rx=0 tx=0).stop 2024-04-26T01:51:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.520+0000 7f95ae767740 1 -- 172.21.15.28:0/1370910261 shutdown_connections 2024-04-26T01:51:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.520+0000 7f95ae767740 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.520+0000 7f95ae767740 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab5140 0x55d53fabf4c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.520+0000 7f95ae767740 1 --2- 172.21.15.28:0/1370910261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53fab4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.520+0000 7f95ae767740 1 -- 172.21.15.28:0/1370910261 >> 172.21.15.28:0/1370910261 conn(0x55d53fab0d30 msgr2=0x55d53fab1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.520+0000 7f95ae767740 1 -- 172.21.15.28:0/1370910261 shutdown_connections 2024-04-26T01:51:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.520+0000 7f95ae767740 1 -- 172.21.15.28:0/1370910261 wait complete. 2024-04-26T01:51:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.521+0000 7f95ae767740 1 -- start start 2024-04-26T01:51:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.521+0000 7f95ae767740 1 Processor -- start 2024-04-26T01:51:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.526+0000 7f95ae767740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab4820 0x55d53fabc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.526+0000 7f95ae767740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fabcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95a3e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab4820 0x55d53fabc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95ae767740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fabfa00 0x55d53fabd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95a3e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab4820 0x55d53fabc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46006/0 (socket says 172.21.15.28:46006) 2024-04-26T01:51:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95a8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fabcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95a8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fabcea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:33980/0 (socket says 172.21.15.28:33980) 2024-04-26T01:51:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95a3e00640 1 -- 172.21.15.28:0/4095957566 learned_addr learned my addr 172.21.15.28:0/4095957566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d53fac2630 con 0x55d53fabfa00 2024-04-26T01:51:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d53f9d4310 con 0x55d53fab5140 2024-04-26T01:51:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.527+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55d53fac27b0 con 0x55d53fab4820 2024-04-26T01:51:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.528+0000 7f95ae767740 10 client.0 _ll_register_callbacks cb 0x55d53f9a2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.529+0000 7f95a3e00640 1 -- 172.21.15.28:0/4095957566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 msgr2=0x55d53fabcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.529+0000 7f95a3e00640 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fabcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.529+0000 7f95a3e00640 1 -- 172.21.15.28:0/4095957566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fabfa00 msgr2=0x55d53fabd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.529+0000 7f95a3e00640 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fabfa00 0x55d53fabd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.529+0000 7f95a3e00640 1 -- 172.21.15.28:0/4095957566 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d53f7f9620 con 0x55d53fab4820 2024-04-26T01:51:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.531+0000 7f95a3e00640 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab4820 0x55d53fabc960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f959002aa80 tx=0x7f9590004540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a25ed5357686b5b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.532+0000 7f958aa00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:51:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.532+0000 7f958aa00640 1 -- 172.21.15.28:0/4095957566 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590035030 con 0x55d53fab4820 2024-04-26T01:51:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.532+0000 7f958aa00640 1 -- 172.21.15.28:0/4095957566 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8557+0+0 (secure 0 0 0) 0x7f95900046c0 con 0x55d53fab4820 2024-04-26T01:51:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.532+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95980041c0 con 0x55d53fab4820 2024-04-26T01:51:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.533+0000 7f958aa00640 1 -- 172.21.15.28:0/4095957566 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f95900048c0 con 0x55d53fab4820 2024-04-26T01:51:07.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.538+0000 7f95ae767740 10 client.26104 fetch_fsmap learned FSMap version 42 2024-04-26T01:51:07.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.538+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d53face860 con 0x55d53fab4820 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.540+0000 7f958aa00640 1 -- 172.21.15.28:0/4095957566 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95980041c0 con 0x55d53fab4820 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.540+0000 7f958aa00640 1 -- 172.21.15.28:0/4095957566 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f95900314a0 con 0x55d53fab4820 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.540+0000 7f95ae767740 10 client.26104 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.540+0000 7f95ae767740 10 client.26104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.540+0000 7f95ae767740 20 client.26104 populate_metadata read hostname 'smithi028' 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.540+0000 7f95ae767740 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d53facfd70 0x55d53fad21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.540+0000 7f95ae767740 4 client.26104 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.540+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d53fa2e8b0 con 0x55d53facfd70 2024-04-26T01:51:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.541+0000 7f95a8e00640 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d53facfd70 0x55d53fad21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.542+0000 7f958aa00640 10 client.26104 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.544+0000 7f95a8e00640 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d53facfd70 0x55d53fad21f0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd459bf294efe0b4 server_cookie=126a52453ad5ad22 in_seq=0 out_seq=0 2024-04-26T01:51:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.546+0000 7f958aa00640 1 -- 172.21.15.28:0/4095957566 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d53fa2e8b0 con 0x55d53facfd70 2024-04-26T01:51:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.546+0000 7f958aa00640 10 client.26104 handle_command_reply: tid=0 2024-04-26T01:51:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.546+0000 7f95ae767740 1 client.26104 shutdown 2024-04-26T01:51:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.547+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab4820 msgr2=0x55d53fabc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.547+0000 7f95ae767740 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab4820 0x55d53fabc960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f959002aa80 tx=0x7f9590004540 comp rx=0 tx=0).stop 2024-04-26T01:51:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.548+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 shutdown_connections 2024-04-26T01:51:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.548+0000 7f95ae767740 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d53facfd70 0x55d53fad21f0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.548+0000 7f95ae767740 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fabfa00 0x55d53fabd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.548+0000 7f95ae767740 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fabcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.548+0000 7f95ae767740 1 --2- 172.21.15.28:0/4095957566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab4820 0x55d53fabc960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.548+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 >> 172.21.15.28:0/4095957566 conn(0x55d53fab0d30 msgr2=0x55d53fab6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.548+0000 7f958aa00640 0 client.26104 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.549+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 shutdown_connections 2024-04-26T01:51:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.549+0000 7f95ae767740 1 -- 172.21.15.28:0/4095957566 wait complete. 2024-04-26T01:51:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.549+0000 7f95ae767740 20 client.26104 trim_cache size 0 max 16384 2024-04-26T01:51:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.591+0000 7f95ae767740 1 Processor -- start 2024-04-26T01:51:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.591+0000 7f95ae767740 1 -- start start 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.591+0000 7f95ae767740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53facb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.591+0000 7f95ae767740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab5140 0x55d53fad7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.591+0000 7f95ae767740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fabfa00 0x55d53fad8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.591+0000 7f95ae767740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9584011230 con 0x55d53fabfa00 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.591+0000 7f95ae767740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f958400ea40 con 0x55d53fab4820 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.591+0000 7f95ae767740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f958400c8b0 con 0x55d53fab5140 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.592+0000 7f95a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53facb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.592+0000 7f95a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53facb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:33992/0 (socket says 172.21.15.28:33992) 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.592+0000 7f95a3e00640 1 -- 172.21.15.28:0/2336028655 learned_addr learned my addr 172.21.15.28:0/2336028655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.592+0000 7f95a3e00640 1 -- 172.21.15.28:0/2336028655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab5140 msgr2=0x55d53fad7ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.592+0000 7f95a3e00640 1 --2- 172.21.15.28:0/2336028655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab5140 0x55d53fad7ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.592+0000 7f95a3e00640 1 -- 172.21.15.28:0/2336028655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fabfa00 msgr2=0x55d53fad8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.592+0000 7f95a3e00640 1 --2- 172.21.15.28:0/2336028655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fabfa00 0x55d53fad8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.592+0000 7f95a3e00640 1 -- 172.21.15.28:0/2336028655 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d53fab4f70 con 0x55d53fab4820 2024-04-26T01:51:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.593+0000 7f95a3e00640 1 --2- 172.21.15.28:0/2336028655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53facb550 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9590031440 tx=0x7f9590005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c88ffc83a7d9a6ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.600+0000 7f9595600640 1 -- 172.21.15.28:0/2336028655 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590035070 con 0x55d53fab4820 2024-04-26T01:51:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.600+0000 7f9595600640 1 -- 172.21.15.28:0/2336028655 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9590041030 con 0x55d53fab4820 2024-04-26T01:51:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.600+0000 7f9595600640 1 -- 172.21.15.28:0/2336028655 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95900035f0 con 0x55d53fab4820 2024-04-26T01:51:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.603+0000 7f95ae767740 1 -- 172.21.15.28:0/2336028655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 msgr2=0x55d53facb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.603+0000 7f95ae767740 1 --2- 172.21.15.28:0/2336028655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53facb550 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9590031440 tx=0x7f9590005fd0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.604+0000 7f95ae767740 1 -- 172.21.15.28:0/2336028655 shutdown_connections 2024-04-26T01:51:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.604+0000 7f95ae767740 1 --2- 172.21.15.28:0/2336028655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fabfa00 0x55d53fad8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.604+0000 7f95ae767740 1 --2- 172.21.15.28:0/2336028655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fab5140 0x55d53fad7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.604+0000 7f95ae767740 1 --2- 172.21.15.28:0/2336028655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab4820 0x55d53facb550 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.604+0000 7f95ae767740 1 -- 172.21.15.28:0/2336028655 >> 172.21.15.28:0/2336028655 conn(0x55d53fab0d30 msgr2=0x55d53fabb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:07.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.604+0000 7f95ae767740 1 -- 172.21.15.28:0/2336028655 shutdown_connections 2024-04-26T01:51:07.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.604+0000 7f95ae767740 1 -- 172.21.15.28:0/2336028655 wait complete. 2024-04-26T01:51:07.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.605+0000 7f95ae767740 1 -- start start 2024-04-26T01:51:07.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.605+0000 7f95ae767740 1 Processor -- start 2024-04-26T01:51:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.607+0000 7f95ae767740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab4820 0x55d53fac7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.607+0000 7f95ae767740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fac5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.607+0000 7f95ae767740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.607+0000 7f95ae767740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f958400cf00 con 0x55d53fab4820 2024-04-26T01:51:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.607+0000 7f95ae767740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f958400e370 con 0x55d53fab5140 2024-04-26T01:51:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.607+0000 7f95ae767740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f958400d5d0 con 0x55d53fabfa00 2024-04-26T01:51:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.607+0000 7f95ae767740 10 client.0 _ll_register_callbacks cb 0x7f9590009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:07.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.608+0000 7f95a8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fac5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.608+0000 7f95a8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fac5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:34006/0 (socket says 172.21.15.28:34006) 2024-04-26T01:51:07.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.608+0000 7f95a8e00640 1 -- 172.21.15.28:0/491490005 learned_addr learned my addr 172.21.15.28:0/491490005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.609+0000 7f95a3e00640 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.610+0000 7f95a3e00640 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab4820 0x55d53fac7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.610+0000 7f95a8e00640 1 -- 172.21.15.28:0/491490005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 msgr2=0x55d53fac5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.610+0000 7f95a8e00640 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.610+0000 7f95a8e00640 1 -- 172.21.15.28:0/491490005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab4820 msgr2=0x55d53fac7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:07.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.611+0000 7f95a8e00640 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab4820 0x55d53fac7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:07.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.611+0000 7f95a8e00640 1 -- 172.21.15.28:0/491490005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d53fac5a90 con 0x55d53fab5140 2024-04-26T01:51:07.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.611+0000 7f95a3e00640 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab4820 0x55d53fac7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:51:07.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.613+0000 7f95a8e00640 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fac5010 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9598005fd0 tx=0x7f95980097c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d761d5375279b1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:07.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.613+0000 7f958aa00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:51:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.614+0000 7f958aa00640 1 -- 172.21.15.28:0/491490005 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9598009e90 con 0x55d53fab5140 2024-04-26T01:51:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.614+0000 7f958aa00640 1 -- 172.21.15.28:0/491490005 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8557+0+0 (secure 0 0 0) 0x7f9598022040 con 0x55d53fab5140 2024-04-26T01:51:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.614+0000 7f95ae767740 1 -- 172.21.15.28:0/491490005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9590041030 con 0x55d53fab5140 2024-04-26T01:51:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.615+0000 7f958aa00640 1 -- 172.21.15.28:0/491490005 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f9598009530 con 0x55d53fab5140 2024-04-26T01:51:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.615+0000 7f95ae767740 10 client.26093 fetch_fsmap learned FSMap version 42 2024-04-26T01:51:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.615+0000 7f95ae767740 1 -- 172.21.15.28:0/491490005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d53fb360c0 con 0x55d53fab5140 2024-04-26T01:51:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.617+0000 7f958aa00640 1 -- 172.21.15.28:0/491490005 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9590041030 con 0x55d53fab5140 2024-04-26T01:51:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.617+0000 7f958aa00640 1 -- 172.21.15.28:0/491490005 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f959800f040 con 0x55d53fab5140 2024-04-26T01:51:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.617+0000 7f95ae767740 10 client.26093 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:51:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.617+0000 7f95ae767740 10 client.26093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.617+0000 7f95ae767740 20 client.26093 populate_metadata read hostname 'smithi028' 2024-04-26T01:51:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.617+0000 7f95ae767740 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d53fb366f0 0x55d53fb38b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.617+0000 7f95ae767740 4 client.26093 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.617+0000 7f95ae767740 1 -- 172.21.15.28:0/491490005 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d53fa28320 con 0x55d53fb366f0 2024-04-26T01:51:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.618+0000 7f95a3e00640 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d53fb366f0 0x55d53fb38b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:07.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.619+0000 7f95a3e00640 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d53fb366f0 0x55d53fb38b10 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccd80fc6468a43d7 server_cookie=695aefe6fd0e4216 in_seq=0 out_seq=0 2024-04-26T01:51:07.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:07.619+0000 7f958aa00640 10 client.26093 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:07.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:07 smithi053 ceph-mon[32593]: pgmap v316: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.0 MiB/s wr, 506 op/s 2024-04-26T01:51:07.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:07 smithi028 ceph-mon[28247]: pgmap v316: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.0 MiB/s wr, 506 op/s 2024-04-26T01:51:07.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:07 smithi169 ceph-mon[32851]: pgmap v316: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.0 MiB/s wr, 506 op/s 2024-04-26T01:51:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:09 smithi169 ceph-mon[32851]: pgmap v317: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.0 MiB/s wr, 506 op/s 2024-04-26T01:51:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:09 smithi169 ceph-mon[32851]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:51:09.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:09 smithi053 ceph-mon[32593]: pgmap v317: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.0 MiB/s wr, 506 op/s 2024-04-26T01:51:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:09 smithi053 ceph-mon[32593]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:51:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:09 smithi028 ceph-mon[28247]: pgmap v317: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.0 MiB/s wr, 506 op/s 2024-04-26T01:51:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:09 smithi028 ceph-mon[28247]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:51:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 555.0 s, 149.6 tps, lat 690.509 ms stddev 416.073 2024-04-26T01:51:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:11 smithi169 ceph-mon[32851]: pgmap v319: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 9.8 MiB/s wr, 680 op/s 2024-04-26T01:51:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:11 smithi053 ceph-mon[32593]: pgmap v319: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 9.8 MiB/s wr, 680 op/s 2024-04-26T01:51:11.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:11 smithi028 ceph-mon[28247]: pgmap v319: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 9.8 MiB/s wr, 680 op/s 2024-04-26T01:51:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.156+0000 7f958aa00640 1 -- 172.21.15.28:0/491490005 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d53fa28320 con 0x55d53fb366f0 2024-04-26T01:51:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.156+0000 7f958aa00640 10 client.26093 handle_command_reply: tid=0 2024-04-26T01:51:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.156+0000 7f95ae767740 1 client.26093 shutdown 2024-04-26T01:51:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 -- 172.21.15.28:0/491490005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 msgr2=0x55d53fac5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fac5010 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9598005fd0 tx=0x7f95980097c0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 -- 172.21.15.28:0/491490005 shutdown_connections 2024-04-26T01:51:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55d53fb366f0 0x55d53fb38b10 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d53fabfa00 0x55d53fac5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d53fab5140 0x55d53fac5010 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f958aa00640 0 client.26093 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 --2- 172.21.15.28:0/491490005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d53fab4820 0x55d53fac7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 -- 172.21.15.28:0/491490005 >> 172.21.15.28:0/491490005 conn(0x55d53fab0d30 msgr2=0x55d53fab61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 -- 172.21.15.28:0/491490005 shutdown_connections 2024-04-26T01:51:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 1 -- 172.21.15.28:0/491490005 wait complete. 2024-04-26T01:51:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.157+0000 7f95ae767740 20 client.26093 trim_cache size 0 max 16384 2024-04-26T01:51:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:51:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:51:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:51:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:51:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.158+0000 7f95ab800640 1 -- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f958c0a3dc0 msgr2=0x7f958c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.158+0000 7f95ab800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f958c0a3dc0 0x7f958c0a6280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f959c09cad0 tx=0x7f959c0bbf90 comp rx=0 tx=0).stop 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.158+0000 7f95ab800640 1 -- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a410c550 msgr2=0x7f95a419ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.158+0000 7f95ab800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a410c550 0x7f95a419ae50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f959c00ed70 tx=0x7f959c00c6a0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.159+0000 7f95ab800640 1 -- 172.21.15.28:0/602680216 shutdown_connections 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.159+0000 7f95ab800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f958c0a3dc0 0x7f958c0a6280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.159+0000 7f95ab800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a410c550 0x7f95a419ae50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.159+0000 7f95ab800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95a4103aa0 0x7f95a4108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.159+0000 7f95ab800640 1 --2- 172.21.15.28:0/602680216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a4103180 0x7f95a4108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.159+0000 7f95ab800640 1 -- 172.21.15.28:0/602680216 >> 172.21.15.28:0/602680216 conn(0x7f95a4077fd0 msgr2=0x7f95a4109640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.159+0000 7f95ab800640 1 -- 172.21.15.28:0/602680216 shutdown_connections 2024-04-26T01:51:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:12.159+0000 7f95ab800640 1 -- 172.21.15.28:0/602680216 wait complete. 2024-04-26T01:51:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:13 smithi169 ceph-mon[32851]: pgmap v320: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.0 MiB/s wr, 555 op/s 2024-04-26T01:51:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:13 smithi053 ceph-mon[32593]: pgmap v320: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.0 MiB/s wr, 555 op/s 2024-04-26T01:51:13.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:13 smithi028 ceph-mon[28247]: pgmap v320: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.0 MiB/s wr, 555 op/s 2024-04-26T01:51:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:15 smithi169 ceph-mon[32851]: pgmap v321: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 13 MiB/s wr, 740 op/s 2024-04-26T01:51:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:15 smithi053 ceph-mon[32593]: pgmap v321: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 13 MiB/s wr, 740 op/s 2024-04-26T01:51:15.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:15 smithi028 ceph-mon[28247]: pgmap v321: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 13 MiB/s wr, 740 op/s 2024-04-26T01:51:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 560.0 s, 1033.2 tps, lat 102.160 ms stddev 263.930 2024-04-26T01:51:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:17 smithi169 ceph-mon[32851]: pgmap v322: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 1.30k op/s 2024-04-26T01:51:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:17 smithi053 ceph-mon[32593]: pgmap v322: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 1.30k op/s 2024-04-26T01:51:17.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:17 smithi028 ceph-mon[28247]: pgmap v322: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 1.30k op/s 2024-04-26T01:51:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:19 smithi169 ceph-mon[32851]: pgmap v323: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 1.30k op/s 2024-04-26T01:51:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:19 smithi053 ceph-mon[32593]: pgmap v323: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 1.30k op/s 2024-04-26T01:51:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:19 smithi028 ceph-mon[28247]: pgmap v323: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 1.30k op/s 2024-04-26T01:51:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 565.0 s, 3025.4 tps, lat 33.236 ms stddev 36.754 2024-04-26T01:51:21.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:21 smithi169 ceph-mon[32851]: pgmap v324: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 29 MiB/s wr, 1.46k op/s 2024-04-26T01:51:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:21 smithi053 ceph-mon[32593]: pgmap v324: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 29 MiB/s wr, 1.46k op/s 2024-04-26T01:51:21.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:21 smithi028 ceph-mon[28247]: pgmap v324: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 29 MiB/s wr, 1.46k op/s 2024-04-26T01:51:23.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:23 smithi169 ceph-mon[32851]: pgmap v325: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 1.15k op/s 2024-04-26T01:51:23.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:23 smithi053 ceph-mon[32593]: pgmap v325: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 1.15k op/s 2024-04-26T01:51:23.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:23 smithi028 ceph-mon[28247]: pgmap v325: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 1.15k op/s 2024-04-26T01:51:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.350+0000 7f821b600640 1 -- 172.21.15.28:0/3123039916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 msgr2=0x7f8214100550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.350+0000 7f821b600640 1 --2- 172.21.15.28:0/3123039916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 0x7f8214100550 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8208009900 tx=0x7f820802a420 comp rx=0 tx=0).stop 2024-04-26T01:51:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.350+0000 7f821b600640 1 -- 172.21.15.28:0/3123039916 shutdown_connections 2024-04-26T01:51:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.350+0000 7f821b600640 1 --2- 172.21.15.28:0/3123039916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8214105310 0x7f82141057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.350+0000 7f821b600640 1 --2- 172.21.15.28:0/3123039916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8214100a90 0x7f8214104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.350+0000 7f821b600640 1 --2- 172.21.15.28:0/3123039916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 0x7f8214100550 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.350+0000 7f821b600640 1 -- 172.21.15.28:0/3123039916 >> 172.21.15.28:0/3123039916 conn(0x7f8214076f50 msgr2=0x7f82140fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.351+0000 7f821b600640 1 -- 172.21.15.28:0/3123039916 shutdown_connections 2024-04-26T01:51:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.351+0000 7f821b600640 1 -- 172.21.15.28:0/3123039916 wait complete. 2024-04-26T01:51:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.352+0000 7f821b600640 1 Processor -- start 2024-04-26T01:51:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.352+0000 7f821b600640 1 -- start start 2024-04-26T01:51:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f821b600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 0x7f821419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f821b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8214100a90 0x7f821419ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f821b600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8214105310 0x7f821419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f821b600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8214109c90 con 0x7f8214100a90 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f821b600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8214109b10 con 0x7f8214105310 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f821b600640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8214109e10 con 0x7f8214100170 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f8213e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8214100a90 0x7f821419ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f8218c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 0x7f821419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f8218c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 0x7f821419a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52640/0 (socket says 172.21.15.28:52640) 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f8213e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8214100a90 0x7f821419ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56552/0 (socket says 172.21.15.28:56552) 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f8218c00640 1 -- 172.21.15.28:0/2108928095 learned_addr learned my addr 172.21.15.28:0/2108928095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.353+0000 7f8219600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8214105310 0x7f821419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.354+0000 7f8219600640 1 -- 172.21.15.28:0/2108928095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 msgr2=0x7f821419a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.354+0000 7f8219600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 0x7f821419a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.354+0000 7f8219600640 1 -- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8214100a90 msgr2=0x7f821419ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.355+0000 7f8219600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8214100a90 0x7f821419ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.355+0000 7f8219600640 1 -- 172.21.15.28:0/2108928095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821419f6b0 con 0x7f8214105310 2024-04-26T01:51:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.355+0000 7f8213e00640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8214100a90 0x7f821419ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.356+0000 7f8219600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8214105310 0x7f821419ef80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f820c00cab0 tx=0x7f820c00cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.358+0000 7f8211600640 1 -- 172.21.15.28:0/2108928095 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c00ae20 con 0x7f8214105310 2024-04-26T01:51:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.358+0000 7f8211600640 1 -- 172.21.15.28:0/2108928095 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f820c00f900 con 0x7f8214105310 2024-04-26T01:51:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.358+0000 7f8211600640 1 -- 172.21.15.28:0/2108928095 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c0189f0 con 0x7f8214105310 2024-04-26T01:51:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.358+0000 7f821b600640 1 -- 172.21.15.28:0/2108928095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821419f9a0 con 0x7f8214105310 2024-04-26T01:51:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.359+0000 7f821b600640 1 -- 172.21.15.28:0/2108928095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8214110740 con 0x7f8214105310 2024-04-26T01:51:25.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.370+0000 7f8211600640 1 -- 172.21.15.28:0/2108928095 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f820c00f450 con 0x7f8214105310 2024-04-26T01:51:25.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.372+0000 7f8211600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f81fc0a3e40 0x7f81fc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.373+0000 7f8211600640 1 -- 172.21.15.28:0/2108928095 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f820c0d0df0 con 0x7f8214105310 2024-04-26T01:51:25.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.376+0000 7f821e480740 1 Processor -- start 2024-04-26T01:51:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.377+0000 7f8218c00640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f81fc0a3e40 0x7f81fc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.377+0000 7f821e480740 1 -- start start 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.378+0000 7f821e480740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 0x555e3475fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.378+0000 7f821e480740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e34760140 0x555e3476a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.378+0000 7f821e480740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 0x555e3476cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.378+0000 7f821e480740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e346c5900 con 0x555e3475f820 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.378+0000 7f821e480740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555e3466bef0 con 0x555e34760140 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.378+0000 7f821e480740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x555e346c18c0 con 0x555e3476aa00 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.379+0000 7f8213e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e34760140 0x555e3476a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.379+0000 7f8213e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e34760140 0x555e3476a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54156/0 (socket says 172.21.15.28:54156) 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.379+0000 7f8213e00640 1 -- 172.21.15.28:0/483386379 learned_addr learned my addr 172.21.15.28:0/483386379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.379+0000 7f8219600640 1 --2- 172.21.15.28:0/483386379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 0x555e3476cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.381+0000 7f8213e00640 1 -- 172.21.15.28:0/483386379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 msgr2=0x555e3476cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.381+0000 7f8213e00640 1 --2- 172.21.15.28:0/483386379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 0x555e3476cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.381+0000 7f8213e00640 1 -- 172.21.15.28:0/483386379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 msgr2=0x555e3475fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.381+0000 7f8213e00640 1 --2- 172.21.15.28:0/483386379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 0x555e3475fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.381+0000 7f8213e00640 1 -- 172.21.15.28:0/483386379 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e343d8210 con 0x555e34760140 2024-04-26T01:51:25.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.382+0000 7f8218c00640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f81fc0a3e40 0x7f81fc0a6300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82080096c0 tx=0x7f8208005d50 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.383+0000 7f8213e00640 1 --2- 172.21.15.28:0/483386379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e34760140 0x555e3476a4c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8200009de0 tx=0x7f820000e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4511aa02811bb9ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.383+0000 7f8205c00640 1 -- 172.21.15.28:0/483386379 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200017070 con 0x555e34760140 2024-04-26T01:51:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.384+0000 7f8205c00640 1 -- 172.21.15.28:0/483386379 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8200013920 con 0x555e34760140 2024-04-26T01:51:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.384+0000 7f8205c00640 1 -- 172.21.15.28:0/483386379 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200012930 con 0x555e34760140 2024-04-26T01:51:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.385+0000 7f821e480740 1 -- 172.21.15.28:0/483386379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e34760140 msgr2=0x555e3476a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.385+0000 7f821e480740 1 --2- 172.21.15.28:0/483386379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e34760140 0x555e3476a4c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8200009de0 tx=0x7f820000e5a0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.385+0000 7f821e480740 1 -- 172.21.15.28:0/483386379 shutdown_connections 2024-04-26T01:51:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.385+0000 7f821e480740 1 --2- 172.21.15.28:0/483386379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 0x555e3476cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.385+0000 7f821e480740 1 --2- 172.21.15.28:0/483386379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e34760140 0x555e3476a4c0 secure :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8200009de0 tx=0x7f820000e5a0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.385+0000 7f821e480740 1 --2- 172.21.15.28:0/483386379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 0x555e3475fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.385+0000 7f821e480740 1 -- 172.21.15.28:0/483386379 >> 172.21.15.28:0/483386379 conn(0x555e3475bd30 msgr2=0x555e3475c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.387+0000 7f821e480740 1 -- 172.21.15.28:0/483386379 shutdown_connections 2024-04-26T01:51:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.387+0000 7f821e480740 1 -- 172.21.15.28:0/483386379 wait complete. 2024-04-26T01:51:25.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.388+0000 7f821e480740 1 -- start start 2024-04-26T01:51:25.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.388+0000 7f821e480740 1 Processor -- start 2024-04-26T01:51:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.391+0000 7f821e480740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e3475f820 0x555e34767b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.391+0000 7f821e480740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 0x555e34768080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.391+0000 7f821e480740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e347685c0 0x555e34776aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.391+0000 7f8218c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e3475f820 0x555e34767b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.391+0000 7f8218c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e3475f820 0x555e34767b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54168/0 (socket says 172.21.15.28:54168) 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.391+0000 7f8218c00640 1 -- 172.21.15.28:0/2347054087 learned_addr learned my addr 172.21.15.28:0/2347054087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.391+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e3476d630 con 0x555e3476aa00 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555e3467f310 con 0x555e3475f820 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x555e3476d7b0 con 0x555e347685c0 2024-04-26T01:51:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f8219600640 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e347685c0 0x555e34776aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f821e480740 10 client.0 _ll_register_callbacks cb 0x555e3464dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f8219600640 1 -- 172.21.15.28:0/2347054087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e3475f820 msgr2=0x555e34767b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f8219600640 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e3475f820 0x555e34767b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f8219600640 1 -- 172.21.15.28:0/2347054087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 msgr2=0x555e34768080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f8219600640 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 0x555e34768080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f8219600640 1 -- 172.21.15.28:0/2347054087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e344a4620 con 0x555e347685c0 2024-04-26T01:51:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.392+0000 7f8218c00640 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e3475f820 0x555e34767b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.394+0000 7f81fa000640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:51:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.394+0000 7f8219600640 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e347685c0 0x555e34776aa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f820c0ce510 tx=0x7f820c00f800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa2f4f928768ca9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.394+0000 7f81fa000640 1 -- 172.21.15.28:0/2347054087 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c0bb060 con 0x555e347685c0 2024-04-26T01:51:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.395+0000 7f81fa000640 1 -- 172.21.15.28:0/2347054087 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f820c0a5570 con 0x555e347685c0 2024-04-26T01:51:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.395+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8200013920 con 0x555e347685c0 2024-04-26T01:51:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.395+0000 7f81fa000640 1 -- 172.21.15.28:0/2347054087 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f820c0a1140 con 0x555e347685c0 2024-04-26T01:51:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.396+0000 7f821e480740 10 client.26119 fetch_fsmap learned FSMap version 42 2024-04-26T01:51:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.396+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x555e34779e60 con 0x555e347685c0 2024-04-26T01:51:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.396+0000 7f81fa000640 1 -- 172.21.15.28:0/2347054087 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8200013920 con 0x555e347685c0 2024-04-26T01:51:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.397+0000 7f81fa000640 1 -- 172.21.15.28:0/2347054087 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f820c0bd0b0 con 0x555e347685c0 2024-04-26T01:51:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.397+0000 7f821e480740 10 client.26119 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:51:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.398+0000 7f821e480740 10 client.26119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.398+0000 7f821e480740 20 client.26119 populate_metadata read hostname 'smithi028' 2024-04-26T01:51:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.398+0000 7f821e480740 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555e3477b390 0x555e3477d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.398+0000 7f821e480740 4 client.26119 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.398+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e346d98b0 con 0x555e3477b390 2024-04-26T01:51:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.398+0000 7f8218c00640 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555e3477b390 0x555e3477d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.401+0000 7f81fa000640 10 client.26119 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.401+0000 7f8218c00640 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555e3477b390 0x555e3477d7e0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11e39ebc26066a58 server_cookie=2b807a6151c3b2f5 in_seq=0 out_seq=0 2024-04-26T01:51:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.402+0000 7f81fa000640 1 -- 172.21.15.28:0/2347054087 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e346d98b0 con 0x555e3477b390 2024-04-26T01:51:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.402+0000 7f81fa000640 10 client.26119 handle_command_reply: tid=0 2024-04-26T01:51:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.402+0000 7f821e480740 1 client.26119 shutdown 2024-04-26T01:51:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.403+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e347685c0 msgr2=0x555e34776aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.403+0000 7f821e480740 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e347685c0 0x555e34776aa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f820c0ce510 tx=0x7f820c00f800 comp rx=0 tx=0).stop 2024-04-26T01:51:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.403+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 shutdown_connections 2024-04-26T01:51:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.403+0000 7f821e480740 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555e3477b390 0x555e3477d7e0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.403+0000 7f821e480740 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e347685c0 0x555e34776aa0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.403+0000 7f821e480740 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 0x555e34768080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.403+0000 7f821e480740 1 --2- 172.21.15.28:0/2347054087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e3475f820 0x555e34767b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.403+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 >> 172.21.15.28:0/2347054087 conn(0x555e3475bd30 msgr2=0x555e34761ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:25.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.404+0000 7f81fa000640 0 client.26119 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:25.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.404+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 shutdown_connections 2024-04-26T01:51:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.406+0000 7f821e480740 1 -- 172.21.15.28:0/2347054087 wait complete. 2024-04-26T01:51:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.406+0000 7f821e480740 20 client.26119 trim_cache size 0 max 16384 2024-04-26T01:51:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.442+0000 7f821e480740 1 Processor -- start 2024-04-26T01:51:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.442+0000 7f821e480740 1 -- start start 2024-04-26T01:51:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.443+0000 7f821e480740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 0x555e3475b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.443+0000 7f821e480740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e347830f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.444+0000 7f821e480740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 0x555e34783630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.444+0000 7f821e480740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81f4011090 con 0x555e3475f820 2024-04-26T01:51:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.444+0000 7f821e480740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f81f400e8a0 con 0x555e347685c0 2024-04-26T01:51:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.444+0000 7f821e480740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f81f400c710 con 0x555e3476aa00 2024-04-26T01:51:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.443+0000 7f8218c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 0x555e3475b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.444+0000 7f8213e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e347830f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.444+0000 7f8213e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e347830f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54182/0 (socket says 172.21.15.28:54182) 2024-04-26T01:51:25.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.444+0000 7f8213e00640 1 -- 172.21.15.28:0/3283909994 learned_addr learned my addr 172.21.15.28:0/3283909994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:25.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.445+0000 7f8219600640 1 --2- 172.21.15.28:0/3283909994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 0x555e34783630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.445+0000 7f8213e00640 1 -- 172.21.15.28:0/3283909994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 msgr2=0x555e34783630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:25.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.445+0000 7f8213e00640 1 --2- 172.21.15.28:0/3283909994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 0x555e34783630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.445+0000 7f8213e00640 1 -- 172.21.15.28:0/3283909994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 msgr2=0x555e3475b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:25.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.445+0000 7f8213e00640 1 --2- 172.21.15.28:0/3283909994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 0x555e3475b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.445+0000 7f8213e00640 1 -- 172.21.15.28:0/3283909994 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e3477de60 con 0x555e347685c0 2024-04-26T01:51:25.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.445+0000 7f8213e00640 1 --2- 172.21.15.28:0/3283909994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e347830f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f820000eaa0 tx=0x7f8200006d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e6ace2f92629b24 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:25.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.446+0000 7f8205200640 1 -- 172.21.15.28:0/3283909994 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200017040 con 0x555e347685c0 2024-04-26T01:51:25.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.446+0000 7f8205200640 1 -- 172.21.15.28:0/3283909994 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8200020070 con 0x555e347685c0 2024-04-26T01:51:25.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.446+0000 7f8205200640 1 -- 172.21.15.28:0/3283909994 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200013470 con 0x555e347685c0 2024-04-26T01:51:25.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 -- 172.21.15.28:0/3283909994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 msgr2=0x555e347830f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 --2- 172.21.15.28:0/3283909994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e347830f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f820000eaa0 tx=0x7f8200006d10 comp rx=0 tx=0).stop 2024-04-26T01:51:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 -- 172.21.15.28:0/3283909994 shutdown_connections 2024-04-26T01:51:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 --2- 172.21.15.28:0/3283909994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3476aa00 0x555e34783630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 --2- 172.21.15.28:0/3283909994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e347830f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 --2- 172.21.15.28:0/3283909994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3475f820 0x555e3475b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 -- 172.21.15.28:0/3283909994 >> 172.21.15.28:0/3283909994 conn(0x555e3475bd30 msgr2=0x555e34762d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 -- 172.21.15.28:0/3283909994 shutdown_connections 2024-04-26T01:51:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.447+0000 7f821e480740 1 -- 172.21.15.28:0/3283909994 wait complete. 2024-04-26T01:51:25.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.448+0000 7f821e480740 1 -- start start 2024-04-26T01:51:25.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.448+0000 7f821e480740 1 Processor -- start 2024-04-26T01:51:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.449+0000 7f821e480740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3475f820 0x555e3476e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f821e480740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e3476e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f821e480740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 0x555e3476ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f821e480740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81f400cd60 con 0x555e3476aa00 2024-04-26T01:51:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f821e480740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f81f400e1d0 con 0x555e347685c0 2024-04-26T01:51:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f821e480740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f81f400d430 con 0x555e3475f820 2024-04-26T01:51:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f821e480740 10 client.0 _ll_register_callbacks cb 0x7f820c09c6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f8213e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e3476e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f8218c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3475f820 0x555e3476e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f8213e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e3476e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:54198/0 (socket says 172.21.15.28:54198) 2024-04-26T01:51:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f8218c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3475f820 0x555e3476e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52666/0 (socket says 172.21.15.28:52666) 2024-04-26T01:51:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.450+0000 7f8218c00640 1 -- 172.21.15.28:0/3740443643 learned_addr learned my addr 172.21.15.28:0/3740443643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:25.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.451+0000 7f8219600640 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 0x555e3476ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.451+0000 7f8218c00640 1 -- 172.21.15.28:0/3740443643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 msgr2=0x555e3476e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:25.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.451+0000 7f8218c00640 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e3476e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.451+0000 7f8218c00640 1 -- 172.21.15.28:0/3740443643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 msgr2=0x555e3476ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.451+0000 7f8218c00640 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 0x555e3476ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.451+0000 7f8218c00640 1 -- 172.21.15.28:0/3740443643 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e347e0990 con 0x555e3475f820 2024-04-26T01:51:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.452+0000 7f8218c00640 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3475f820 0x555e3476e480 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82080099f0 tx=0x7f8208005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84cb92320eaa2c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.452+0000 7f81fa000640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:51:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.453+0000 7f81fa000640 1 -- 172.21.15.28:0/3740443643 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208035040 con 0x555e3475f820 2024-04-26T01:51:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.453+0000 7f81fa000640 1 -- 172.21.15.28:0/3740443643 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f82080513b0 con 0x555e3475f820 2024-04-26T01:51:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.453+0000 7f821e480740 1 -- 172.21.15.28:0/3740443643 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8200020070 con 0x555e3475f820 2024-04-26T01:51:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.454+0000 7f81fa000640 1 -- 172.21.15.28:0/3740443643 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f82080515b0 con 0x555e3475f820 2024-04-26T01:51:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.454+0000 7f821e480740 10 client.26125 fetch_fsmap learned FSMap version 42 2024-04-26T01:51:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.454+0000 7f821e480740 1 -- 172.21.15.28:0/3740443643 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x555e347e1cc0 con 0x555e3475f820 2024-04-26T01:51:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.455+0000 7f81fa000640 1 -- 172.21.15.28:0/3740443643 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8200020070 con 0x555e3475f820 2024-04-26T01:51:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.455+0000 7f81fa000640 1 -- 172.21.15.28:0/3740443643 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f820803c440 con 0x555e3475f820 2024-04-26T01:51:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.455+0000 7f821e480740 10 client.26125 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:51:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.455+0000 7f821e480740 10 client.26125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.455+0000 7f821e480740 20 client.26125 populate_metadata read hostname 'smithi028' 2024-04-26T01:51:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.455+0000 7f821e480740 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555e347e2420 0x555e347e4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.455+0000 7f821e480740 4 client.26125 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.455+0000 7f821e480740 1 -- 172.21.15.28:0/3740443643 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555e346d3320 con 0x555e347e2420 2024-04-26T01:51:25.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.457+0000 7f8213e00640 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555e347e2420 0x555e347e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.457+0000 7f81fa000640 10 client.26125 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.458+0000 7f8213e00640 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555e347e2420 0x555e347e4840 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4533c5197efd9ce server_cookie=f0e5d48b3e3618d in_seq=0 out_seq=0 2024-04-26T01:51:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.479+0000 7f81fa000640 1 -- 172.21.15.28:0/3740443643 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555e346d3320 con 0x555e347e2420 2024-04-26T01:51:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.479+0000 7f81fa000640 10 client.26125 handle_command_reply: tid=0 2024-04-26T01:51:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.479+0000 7f821e480740 1 client.26125 shutdown 2024-04-26T01:51:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.480+0000 7f821e480740 1 -- 172.21.15.28:0/3740443643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3475f820 msgr2=0x555e3476e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.480+0000 7f821e480740 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3475f820 0x555e3476e480 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82080099f0 tx=0x7f8208005fd0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.480+0000 7f821e480740 1 -- 172.21.15.28:0/3740443643 shutdown_connections 2024-04-26T01:51:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.481+0000 7f821e480740 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555e347e2420 0x555e347e4840 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.481+0000 7f821e480740 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3476aa00 0x555e3476ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.481+0000 7f821e480740 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555e347685c0 0x555e3476e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.481+0000 7f821e480740 1 --2- 172.21.15.28:0/3740443643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555e3475f820 0x555e3476e480 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.481+0000 7f81fa000640 0 client.26125 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.481+0000 7f821e480740 1 -- 172.21.15.28:0/3740443643 >> 172.21.15.28:0/3740443643 conn(0x555e3475bd30 msgr2=0x555e34761220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.481+0000 7f821e480740 1 -- 172.21.15.28:0/3740443643 shutdown_connections 2024-04-26T01:51:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.482+0000 7f821e480740 1 -- 172.21.15.28:0/3740443643 wait complete. 2024-04-26T01:51:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.482+0000 7f821e480740 20 client.26125 trim_cache size 0 max 16384 2024-04-26T01:51:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:51:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:51:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:51:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 -- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f81fc0a3e40 msgr2=0x7f81fc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f81fc0a3e40 0x7f81fc0a6300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82080096c0 tx=0x7f8208005d50 comp rx=0 tx=0).stop 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 -- 172.21.15.28:0/2108928095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8214105310 msgr2=0x7f821419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8214105310 0x7f821419ef80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f820c00cab0 tx=0x7f820c00cf70 comp rx=0 tx=0).stop 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 -- 172.21.15.28:0/2108928095 shutdown_connections 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f81fc0a3e40 0x7f81fc0a6300 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8214105310 0x7f821419ef80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8214100a90 0x7f821419ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 --2- 172.21.15.28:0/2108928095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8214100170 0x7f821419a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.483+0000 7f821b600640 1 -- 172.21.15.28:0/2108928095 >> 172.21.15.28:0/2108928095 conn(0x7f8214076f50 msgr2=0x7f8214102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.484+0000 7f821b600640 1 -- 172.21.15.28:0/2108928095 shutdown_connections 2024-04-26T01:51:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:25.484+0000 7f821b600640 1 -- 172.21.15.28:0/2108928095 wait complete. 2024-04-26T01:51:25.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:25 smithi169 ceph-mon[32851]: pgmap v326: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 MiB/s wr, 1.36k op/s 2024-04-26T01:51:25.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:25 smithi053 ceph-mon[32593]: pgmap v326: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 MiB/s wr, 1.36k op/s 2024-04-26T01:51:25.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:25 smithi028 ceph-mon[28247]: pgmap v326: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 MiB/s wr, 1.36k op/s 2024-04-26T01:51:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 570.0 s, 2378.9 tps, lat 41.671 ms stddev 47.165 2024-04-26T01:51:26.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:26 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:26.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:26 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:26.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:26 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:27.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:27 smithi169 ceph-mon[32851]: pgmap v327: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T01:51:27.709 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:27 smithi053 ceph-mon[32593]: pgmap v327: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T01:51:27.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:27 smithi028 ceph-mon[28247]: pgmap v327: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 MiB/s wr, 1.34k op/s 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: pgmap v328: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.4 MiB/s wr, 802 op/s 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:29 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: pgmap v328: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.4 MiB/s wr, 802 op/s 2024-04-26T01:51:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:29 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: pgmap v328: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.4 MiB/s wr, 802 op/s 2024-04-26T01:51:29.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:29.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:29.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:29 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: Unable to set osd_memory_target on smithi169 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:30.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:30 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: Unable to set osd_memory_target on smithi169 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T01:51:30.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:30.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:30 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:30.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-04-26T01:51:30.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: Unable to set osd_memory_target on smithi169 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-26T01:51:30.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:30.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-26T01:51:30.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-26T01:51:30.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:30.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:30 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 575.0 s, 1587.4 tps, lat 62.915 ms stddev 73.785 2024-04-26T01:51:31.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:31 smithi053 ceph-mon[32593]: pgmap v329: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-26T01:51:31.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:31 smithi028 ceph-mon[28247]: pgmap v329: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-26T01:51:31.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:31 smithi169 ceph-mon[32851]: pgmap v329: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:32.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:32 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:32.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-26T01:51:32.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:51:32.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:32.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:32.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:32 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:32.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:51:32.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-26T01:51:32.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:51:32.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:32.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:32.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:32 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:51:33.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:33 smithi028 ceph-mon[28247]: pgmap v330: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.7 MiB/s wr, 701 op/s 2024-04-26T01:51:33.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:33 smithi169 ceph-mon[32851]: pgmap v330: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.7 MiB/s wr, 701 op/s 2024-04-26T01:51:33.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:33 smithi053 ceph-mon[32593]: pgmap v330: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.7 MiB/s wr, 701 op/s 2024-04-26T01:51:35.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:35 smithi169 ceph-mon[32851]: pgmap v331: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.8 MiB/s wr, 815 op/s 2024-04-26T01:51:35.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:35 smithi053 ceph-mon[32593]: pgmap v331: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.8 MiB/s wr, 815 op/s 2024-04-26T01:51:35.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:35 smithi028 ceph-mon[28247]: pgmap v331: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.8 MiB/s wr, 815 op/s 2024-04-26T01:51:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 580.0 s, 1955.0 tps, lat 51.278 ms stddev 62.010 2024-04-26T01:51:37.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:37 smithi169 ceph-mon[32851]: pgmap v332: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 749 op/s 2024-04-26T01:51:37.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:37 smithi053 ceph-mon[32593]: pgmap v332: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 749 op/s 2024-04-26T01:51:37.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:37 smithi028 ceph-mon[28247]: pgmap v332: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 749 op/s 2024-04-26T01:51:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.678+0000 7f8b6f600640 1 -- 172.21.15.28:0/1480987867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68103b00 msgr2=0x7f8b6810bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.678+0000 7f8b6f600640 1 --2- 172.21.15.28:0/1480987867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68103b00 0x7f8b6810bfe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009900 tx=0x7f8b5802a510 comp rx=0 tx=0).stop 2024-04-26T01:51:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.680+0000 7f8b6f600640 1 -- 172.21.15.28:0/1480987867 shutdown_connections 2024-04-26T01:51:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.680+0000 7f8b6f600640 1 --2- 172.21.15.28:0/1480987867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b6810c520 0x7f8b6810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.680+0000 7f8b6f600640 1 --2- 172.21.15.28:0/1480987867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68103b00 0x7f8b6810bfe0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.680+0000 7f8b6f600640 1 --2- 172.21.15.28:0/1480987867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b681031e0 0x7f8b681035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.680+0000 7f8b6f600640 1 -- 172.21.15.28:0/1480987867 >> 172.21.15.28:0/1480987867 conn(0x7f8b68077fd0 msgr2=0x7f8b680783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.680+0000 7f8b6f600640 1 -- 172.21.15.28:0/1480987867 shutdown_connections 2024-04-26T01:51:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.680+0000 7f8b6f600640 1 -- 172.21.15.28:0/1480987867 wait complete. 2024-04-26T01:51:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.682+0000 7f8b6f600640 1 Processor -- start 2024-04-26T01:51:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.682+0000 7f8b6f600640 1 -- start start 2024-04-26T01:51:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.683+0000 7f8b6f600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b681031e0 0x7f8b681a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.683+0000 7f8b6f600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b68103b00 0x7f8b681a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.683+0000 7f8b6f600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b6810c520 0x7f8b6819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.683+0000 7f8b67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b68103b00 0x7f8b681a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.683+0000 7f8b6cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b681031e0 0x7f8b681a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.683+0000 7f8b67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b68103b00 0x7f8b681a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54940/0 (socket says 172.21.15.28:54940) 2024-04-26T01:51:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.683+0000 7f8b6cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b681031e0 0x7f8b681a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59592/0 (socket says 172.21.15.28:59592) 2024-04-26T01:51:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.684+0000 7f8b6f600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b681122e0 con 0x7f8b68103b00 2024-04-26T01:51:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.684+0000 7f8b6f600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b68112160 con 0x7f8b681031e0 2024-04-26T01:51:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.684+0000 7f8b6f600640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8b68112460 con 0x7f8b6810c520 2024-04-26T01:51:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.684+0000 7f8b67e00640 1 -- 172.21.15.28:0/461405596 learned_addr learned my addr 172.21.15.28:0/461405596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.685+0000 7f8b6d600640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b6810c520 0x7f8b6819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.685+0000 7f8b67e00640 1 -- 172.21.15.28:0/461405596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b6810c520 msgr2=0x7f8b6819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.685+0000 7f8b67e00640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b6810c520 0x7f8b6819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.685+0000 7f8b67e00640 1 -- 172.21.15.28:0/461405596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b681031e0 msgr2=0x7f8b681a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.685+0000 7f8b67e00640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b681031e0 0x7f8b681a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.685+0000 7f8b67e00640 1 -- 172.21.15.28:0/461405596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6819b0a0 con 0x7f8b68103b00 2024-04-26T01:51:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.686+0000 7f8b67e00640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b68103b00 0x7f8b681a0c60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8b58005ab0 tx=0x7f8b58003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.687+0000 7f8b65600640 1 -- 172.21.15.28:0/461405596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58035030 con 0x7f8b68103b00 2024-04-26T01:51:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.687+0000 7f8b65600640 1 -- 172.21.15.28:0/461405596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b58045070 con 0x7f8b68103b00 2024-04-26T01:51:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.687+0000 7f8b65600640 1 -- 172.21.15.28:0/461405596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b580307c0 con 0x7f8b68103b00 2024-04-26T01:51:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.687+0000 7f8b6f600640 1 -- 172.21.15.28:0/461405596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6819b330 con 0x7f8b68103b00 2024-04-26T01:51:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.687+0000 7f8b6f600640 1 -- 172.21.15.28:0/461405596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b68103f80 con 0x7f8b68103b00 2024-04-26T01:51:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.694+0000 7f8b65600640 1 -- 172.21.15.28:0/461405596 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b58030960 con 0x7f8b68103b00 2024-04-26T01:51:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.696+0000 7f8b72447740 1 Processor -- start 2024-04-26T01:51:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.696+0000 7f8b72447740 1 -- start start 2024-04-26T01:51:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.696+0000 7f8b72447740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2cd820 0x5620eb2cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.696+0000 7f8b72447740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2ce140 0x5620eb2d84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.697+0000 7f8b72447740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2dadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.697+0000 7f8b72447740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620eb233900 con 0x5620eb2ce140 2024-04-26T01:51:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.697+0000 7f8b72447740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5620eb1d9ef0 con 0x5620eb2cd820 2024-04-26T01:51:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.697+0000 7f8b72447740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5620eb22f8c0 con 0x5620eb2d8a00 2024-04-26T01:51:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.697+0000 7f8b6d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2cd820 0x5620eb2cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.697+0000 7f8b6d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2cd820 0x5620eb2cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59606/0 (socket says 172.21.15.28:59606) 2024-04-26T01:51:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.697+0000 7f8b6d600640 1 -- 172.21.15.28:0/1159940498 learned_addr learned my addr 172.21.15.28:0/1159940498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.698+0000 7f8b67e00640 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2dadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.698+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2ce140 0x5620eb2d84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.698+0000 7f8b6d600640 1 -- 172.21.15.28:0/1159940498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 msgr2=0x5620eb2dadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.698+0000 7f8b6d600640 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2dadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.698+0000 7f8b6d600640 1 -- 172.21.15.28:0/1159940498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2ce140 msgr2=0x5620eb2d84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.698+0000 7f8b6d600640 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2ce140 0x5620eb2d84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.698+0000 7f8b6d600640 1 -- 172.21.15.28:0/1159940498 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620eaf46210 con 0x5620eb2cd820 2024-04-26T01:51:38.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.698+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2ce140 0x5620eb2d84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:51:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.700+0000 7f8b65600640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8b4c0a3e40 0x7f8b4c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.700+0000 7f8b65600640 1 -- 172.21.15.28:0/461405596 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f8b580f95c0 con 0x7f8b68103b00 2024-04-26T01:51:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.701+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8b4c0a3e40 0x7f8b4c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.702+0000 7f8b6d600640 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2cd820 0x5620eb2cdc00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8b500099e0 tx=0x7f8b50009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3d79d2931c3e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.703+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8b4c0a3e40 0x7f8b4c0a6300 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x5620eb2cedc0 tx=0x7f8b600096d0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.703+0000 7f8b5dc00640 1 -- 172.21.15.28:0/1159940498 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50017070 con 0x5620eb2cd820 2024-04-26T01:51:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.703+0000 7f8b5dc00640 1 -- 172.21.15.28:0/1159940498 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b50013470 con 0x5620eb2cd820 2024-04-26T01:51:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.703+0000 7f8b5dc00640 1 -- 172.21.15.28:0/1159940498 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50012610 con 0x5620eb2cd820 2024-04-26T01:51:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.704+0000 7f8b72447740 1 -- 172.21.15.28:0/1159940498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2cd820 msgr2=0x5620eb2cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.704+0000 7f8b72447740 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2cd820 0x5620eb2cdc00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8b500099e0 tx=0x7f8b50009ea0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.704+0000 7f8b72447740 1 -- 172.21.15.28:0/1159940498 shutdown_connections 2024-04-26T01:51:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.705+0000 7f8b72447740 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2dadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.705+0000 7f8b72447740 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2ce140 0x5620eb2d84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.705+0000 7f8b72447740 1 --2- 172.21.15.28:0/1159940498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2cd820 0x5620eb2cdc00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.705+0000 7f8b72447740 1 -- 172.21.15.28:0/1159940498 >> 172.21.15.28:0/1159940498 conn(0x5620eb2c9d30 msgr2=0x5620eb2ca110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.705+0000 7f8b72447740 1 -- 172.21.15.28:0/1159940498 shutdown_connections 2024-04-26T01:51:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.705+0000 7f8b72447740 1 -- 172.21.15.28:0/1159940498 wait complete. 2024-04-26T01:51:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.706+0000 7f8b72447740 1 -- start start 2024-04-26T01:51:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.706+0000 7f8b72447740 1 Processor -- start 2024-04-26T01:51:38.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.707+0000 7f8b72447740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2d5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.707+0000 7f8b72447740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2ce140 0x5620eb2d5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.707+0000 7f8b72447740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2d8a00 0x5620eb2d6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.708+0000 7f8b72447740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620eb2db630 con 0x5620eb2cd820 2024-04-26T01:51:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.708+0000 7f8b72447740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5620eb1ed310 con 0x5620eb2d8a00 2024-04-26T01:51:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.708+0000 7f8b72447740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5620eb2db7b0 con 0x5620eb2ce140 2024-04-26T01:51:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.708+0000 7f8b72447740 10 client.0 _ll_register_callbacks cb 0x5620eb1bbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.708+0000 7f8b6d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2d8a00 0x5620eb2d6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.708+0000 7f8b67e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2ce140 0x5620eb2d5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.708+0000 7f8b6d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2d8a00 0x5620eb2d6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59622/0 (socket says 172.21.15.28:59622) 2024-04-26T01:51:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.708+0000 7f8b6d600640 1 -- 172.21.15.28:0/2447879644 learned_addr learned my addr 172.21.15.28:0/2447879644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.709+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2d5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.709+0000 7f8b6d600640 1 -- 172.21.15.28:0/2447879644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2ce140 msgr2=0x5620eb2d5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.709+0000 7f8b6d600640 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2ce140 0x5620eb2d5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.709+0000 7f8b6d600640 1 -- 172.21.15.28:0/2447879644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 msgr2=0x5620eb2d5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.709+0000 7f8b6d600640 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2d5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.709+0000 7f8b6d600640 1 -- 172.21.15.28:0/2447879644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620eb012620 con 0x5620eb2d8a00 2024-04-26T01:51:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.709+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2d5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:51:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.710+0000 7f8b6d600640 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2d8a00 0x5620eb2d6380 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000ed40 tx=0x7f8b5000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bd510b3b324653d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.710+0000 7f8b56000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:51:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.711+0000 7f8b56000640 1 -- 172.21.15.28:0/2447879644 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50017040 con 0x5620eb2d8a00 2024-04-26T01:51:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.711+0000 7f8b56000640 1 -- 172.21.15.28:0/2447879644 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f8b5001c450 con 0x5620eb2d8a00 2024-04-26T01:51:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.711+0000 7f8b72447740 1 -- 172.21.15.28:0/2447879644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b50013470 con 0x5620eb2d8a00 2024-04-26T01:51:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.711+0000 7f8b56000640 1 -- 172.21.15.28:0/2447879644 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f8b5001c670 con 0x5620eb2d8a00 2024-04-26T01:51:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.712+0000 7f8b72447740 10 client.26117 fetch_fsmap learned FSMap version 42 2024-04-26T01:51:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.712+0000 7f8b72447740 1 -- 172.21.15.28:0/2447879644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5620eb2e77a0 con 0x5620eb2d8a00 2024-04-26T01:51:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.712+0000 7f8b56000640 1 -- 172.21.15.28:0/2447879644 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b50013470 con 0x5620eb2d8a00 2024-04-26T01:51:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.715+0000 7f8b56000640 1 -- 172.21.15.28:0/2447879644 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f8b50023020 con 0x5620eb2d8a00 2024-04-26T01:51:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.715+0000 7f8b72447740 10 client.26117 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:51:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.715+0000 7f8b72447740 10 client.26117 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.715+0000 7f8b72447740 20 client.26117 populate_metadata read hostname 'smithi028' 2024-04-26T01:51:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.716+0000 7f8b72447740 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620eb2e8af0 0x5620eb2eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.716+0000 7f8b72447740 4 client.26117 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.716+0000 7f8b72447740 1 -- 172.21.15.28:0/2447879644 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620eb2478b0 con 0x5620eb2e8af0 2024-04-26T01:51:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.716+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620eb2e8af0 0x5620eb2eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.718+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620eb2e8af0 0x5620eb2eaee0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b6ed9ec479078a9 server_cookie=f0f908af1771d8e6 in_seq=0 out_seq=0 2024-04-26T01:51:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.718+0000 7f8b56000640 10 client.26117 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.719+0000 7f8b56000640 1 -- 172.21.15.28:0/2447879644 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620eb2478b0 con 0x5620eb2e8af0 2024-04-26T01:51:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.719+0000 7f8b56000640 10 client.26117 handle_command_reply: tid=0 2024-04-26T01:51:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.719+0000 7f8b72447740 1 client.26117 shutdown 2024-04-26T01:51:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b72447740 1 -- 172.21.15.28:0/2447879644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2d8a00 msgr2=0x5620eb2d6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b72447740 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2d8a00 0x5620eb2d6380 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000ed40 tx=0x7f8b5000ed70 comp rx=0 tx=0).stop 2024-04-26T01:51:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b72447740 1 -- 172.21.15.28:0/2447879644 shutdown_connections 2024-04-26T01:51:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b72447740 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620eb2e8af0 0x5620eb2eaee0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b72447740 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2d8a00 0x5620eb2d6380 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b72447740 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2ce140 0x5620eb2d5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b56000640 0 client.26117 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b72447740 1 --2- 172.21.15.28:0/2447879644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2d5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.720+0000 7f8b72447740 1 -- 172.21.15.28:0/2447879644 >> 172.21.15.28:0/2447879644 conn(0x5620eb2c9d30 msgr2=0x5620eb2cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.721+0000 7f8b72447740 1 -- 172.21.15.28:0/2447879644 shutdown_connections 2024-04-26T01:51:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.721+0000 7f8b72447740 1 -- 172.21.15.28:0/2447879644 wait complete. 2024-04-26T01:51:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.722+0000 7f8b72447740 20 client.26117 trim_cache size 0 max 16384 2024-04-26T01:51:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.755+0000 7f8b72447740 1 Processor -- start 2024-04-26T01:51:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b72447740 1 -- start start 2024-04-26T01:51:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b72447740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b72447740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb2d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b72447740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b72447740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b44011020 con 0x5620eb2cd820 2024-04-26T01:51:38.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b72447740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4400e830 con 0x5620eb2ce140 2024-04-26T01:51:38.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b72447740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4400c730 con 0x5620eb2d8a00 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb2d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb2d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59628/0 (socket says 172.21.15.28:59628) 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.756+0000 7f8b67e00640 1 -- 172.21.15.28:0/1214726934 learned_addr learned my addr 172.21.15.28:0/1214726934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.757+0000 7f8b6d600640 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.757+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.757+0000 7f8b6d600640 1 -- 172.21.15.28:0/1214726934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 msgr2=0x5620eb2d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.757+0000 7f8b6d600640 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb2d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.757+0000 7f8b6d600640 1 -- 172.21.15.28:0/1214726934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 msgr2=0x5620eb2e44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.757+0000 7f8b6d600640 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.757+0000 7f8b6d600640 1 -- 172.21.15.28:0/1214726934 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620eb2d48b0 con 0x5620eb2d8a00 2024-04-26T01:51:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.757+0000 7f8b6d600640 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2ca110 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b50025980 tx=0x7f8b50009ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52906c214515c179 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.758+0000 7f8b5d200640 1 -- 172.21.15.28:0/1214726934 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50017070 con 0x5620eb2d8a00 2024-04-26T01:51:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.758+0000 7f8b5d200640 1 -- 172.21.15.28:0/1214726934 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b50013470 con 0x5620eb2d8a00 2024-04-26T01:51:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.758+0000 7f8b5d200640 1 -- 172.21.15.28:0/1214726934 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5002f420 con 0x5620eb2d8a00 2024-04-26T01:51:38.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.760+0000 7f8b72447740 1 -- 172.21.15.28:0/1214726934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 msgr2=0x5620eb2ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.760+0000 7f8b72447740 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2ca110 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b50025980 tx=0x7f8b50009ea0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.760+0000 7f8b72447740 1 -- 172.21.15.28:0/1214726934 shutdown_connections 2024-04-26T01:51:38.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.760+0000 7f8b72447740 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb2ca110 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.760+0000 7f8b72447740 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb2d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.760+0000 7f8b72447740 1 --2- 172.21.15.28:0/1214726934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.760+0000 7f8b72447740 1 -- 172.21.15.28:0/1214726934 >> 172.21.15.28:0/1214726934 conn(0x5620eb2c9d30 msgr2=0x5620eb2c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.760+0000 7f8b72447740 1 -- 172.21.15.28:0/1214726934 shutdown_connections 2024-04-26T01:51:38.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.761+0000 7f8b72447740 1 -- 172.21.15.28:0/1214726934 wait complete. 2024-04-26T01:51:38.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.762+0000 7f8b72447740 1 -- start start 2024-04-26T01:51:38.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.762+0000 7f8b72447740 1 Processor -- start 2024-04-26T01:51:38.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.763+0000 7f8b72447740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.764+0000 7f8b72447740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb3474f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.765+0000 7f8b72447740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb347a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.765+0000 7f8b72447740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4400ccf0 con 0x5620eb2cd820 2024-04-26T01:51:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.765+0000 7f8b72447740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4400e160 con 0x5620eb2ce140 2024-04-26T01:51:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.765+0000 7f8b72447740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4400d3c0 con 0x5620eb2d8a00 2024-04-26T01:51:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.765+0000 7f8b72447740 10 client.0 _ll_register_callbacks cb 0x7f8b50006d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.766+0000 7f8b6d600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb347a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.766+0000 7f8b6d600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb347a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:52006/0 (socket says 172.21.15.28:52006) 2024-04-26T01:51:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.766+0000 7f8b6d600640 1 -- 172.21.15.28:0/1644071287 learned_addr learned my addr 172.21.15.28:0/1644071287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.766+0000 7f8b67e00640 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb3474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.766+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.767+0000 7f8b67e00640 1 -- 172.21.15.28:0/1644071287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 msgr2=0x5620eb347a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.767+0000 7f8b67e00640 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb347a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.767+0000 7f8b67e00640 1 -- 172.21.15.28:0/1644071287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 msgr2=0x5620eb2e0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.767+0000 7f8b67e00640 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.767+0000 7f8b67e00640 1 -- 172.21.15.28:0/1644071287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620eb348210 con 0x5620eb2ce140 2024-04-26T01:51:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.767+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:51:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.768+0000 7f8b67e00640 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb3474f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8b580c1ad0 tx=0x7f8b580ff240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5a90a1ee30d3f6e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.768+0000 7f8b56000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:51:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.768+0000 7f8b56000640 1 -- 172.21.15.28:0/1644071287 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b580c5d70 con 0x5620eb2ce140 2024-04-26T01:51:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.768+0000 7f8b56000640 1 -- 172.21.15.28:0/1644071287 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f8b580c1ce0 con 0x5620eb2ce140 2024-04-26T01:51:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.768+0000 7f8b72447740 1 -- 172.21.15.28:0/1644071287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b50013470 con 0x5620eb2ce140 2024-04-26T01:51:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.770+0000 7f8b56000640 1 -- 172.21.15.28:0/1644071287 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f8b580c5670 con 0x5620eb2ce140 2024-04-26T01:51:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.770+0000 7f8b72447740 10 client.26123 fetch_fsmap learned FSMap version 42 2024-04-26T01:51:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.770+0000 7f8b72447740 1 -- 172.21.15.28:0/1644071287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5620eb35fe80 con 0x5620eb2ce140 2024-04-26T01:51:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.770+0000 7f8b56000640 1 -- 172.21.15.28:0/1644071287 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b50013470 con 0x5620eb2ce140 2024-04-26T01:51:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.771+0000 7f8b56000640 1 -- 172.21.15.28:0/1644071287 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f8b580cde50 con 0x5620eb2ce140 2024-04-26T01:51:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.771+0000 7f8b72447740 10 client.26123 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:51:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.771+0000 7f8b72447740 10 client.26123 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.771+0000 7f8b72447740 20 client.26123 populate_metadata read hostname 'smithi028' 2024-04-26T01:51:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.771+0000 7f8b72447740 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620eb3605e0 0x5620eb362a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.771+0000 7f8b72447740 4 client.26123 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.771+0000 7f8b72447740 1 -- 172.21.15.28:0/1644071287 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5620eb241320 con 0x5620eb3605e0 2024-04-26T01:51:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.772+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620eb3605e0 0x5620eb362a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.772+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620eb3605e0 0x5620eb362a00 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d4c5bd00b0fa212 server_cookie=c09dd891f376649f in_seq=0 out_seq=0 2024-04-26T01:51:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.772+0000 7f8b56000640 10 client.26123 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.787+0000 7f8b56000640 1 -- 172.21.15.28:0/1644071287 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5620eb241320 con 0x5620eb3605e0 2024-04-26T01:51:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.787+0000 7f8b56000640 10 client.26123 handle_command_reply: tid=0 2024-04-26T01:51:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.787+0000 7f8b72447740 1 client.26123 shutdown 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 -- 172.21.15.28:0/1644071287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 msgr2=0x5620eb3474f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb3474f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8b580c1ad0 tx=0x7f8b580ff240 comp rx=0 tx=0).stop 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 -- 172.21.15.28:0/1644071287 shutdown_connections 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620eb3605e0 0x5620eb362a00 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5620eb2d8a00 0x5620eb347a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5620eb2ce140 0x5620eb3474f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 --2- 172.21.15.28:0/1644071287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620eb2cd820 0x5620eb2e0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 -- 172.21.15.28:0/1644071287 >> 172.21.15.28:0/1644071287 conn(0x5620eb2c9d30 msgr2=0x5620eb2c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b56000640 0 client.26123 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.788+0000 7f8b72447740 1 -- 172.21.15.28:0/1644071287 shutdown_connections 2024-04-26T01:51:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.789+0000 7f8b72447740 1 -- 172.21.15.28:0/1644071287 wait complete. 2024-04-26T01:51:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.789+0000 7f8b72447740 20 client.26123 trim_cache size 0 max 16384 2024-04-26T01:51:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:51:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:51:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:51:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:51:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 -- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8b4c0a3e40 msgr2=0x7f8b4c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8b4c0a3e40 0x7f8b4c0a6300 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x5620eb2cedc0 tx=0x7f8b600096d0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 -- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b68103b00 msgr2=0x7f8b681a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b68103b00 0x7f8b681a0c60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8b58005ab0 tx=0x7f8b58003d50 comp rx=0 tx=0).stop 2024-04-26T01:51:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 -- 172.21.15.28:0/461405596 shutdown_connections 2024-04-26T01:51:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8b4c0a3e40 0x7f8b4c0a6300 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b6810c520 0x7f8b6819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b68103b00 0x7f8b681a0c60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 --2- 172.21.15.28:0/461405596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b681031e0 0x7f8b681a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 -- 172.21.15.28:0/461405596 >> 172.21.15.28:0/461405596 conn(0x7f8b68077fd0 msgr2=0x7f8b68108510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 -- 172.21.15.28:0/461405596 shutdown_connections 2024-04-26T01:51:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:38.790+0000 7f8b6f600640 1 -- 172.21.15.28:0/461405596 wait complete. 2024-04-26T01:51:39.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:39 smithi169 ceph-mon[32851]: pgmap v333: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.8 MiB/s wr, 526 op/s 2024-04-26T01:51:39.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:39 smithi053 ceph-mon[32593]: pgmap v333: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.8 MiB/s wr, 526 op/s 2024-04-26T01:51:39.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:39 smithi028 ceph-mon[28247]: pgmap v333: 97 pgs: 97 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.8 MiB/s wr, 526 op/s 2024-04-26T01:51:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 585.0 s, 2193.4 tps, lat 45.842 ms stddev 56.828 2024-04-26T01:51:41.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:41 smithi169 ceph-mon[32851]: pgmap v334: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 7.4 MiB/s wr, 809 op/s 2024-04-26T01:51:41.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:41 smithi053 ceph-mon[32593]: pgmap v334: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 7.4 MiB/s wr, 809 op/s 2024-04-26T01:51:41.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:41 smithi028 ceph-mon[28247]: pgmap v334: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 7.4 MiB/s wr, 809 op/s 2024-04-26T01:51:43.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:43 smithi169 ceph-mon[32851]: pgmap v335: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-04-26T01:51:43.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:43 smithi053 ceph-mon[32593]: pgmap v335: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-04-26T01:51:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:43 smithi028 ceph-mon[28247]: pgmap v335: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-04-26T01:51:45.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:45 smithi169 ceph-mon[32851]: pgmap v336: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.7 MiB/s wr, 653 op/s 2024-04-26T01:51:45.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:45 smithi053 ceph-mon[32593]: pgmap v336: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.7 MiB/s wr, 653 op/s 2024-04-26T01:51:45.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:45 smithi028 ceph-mon[28247]: pgmap v336: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.7 MiB/s wr, 653 op/s 2024-04-26T01:51:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 590.0 s, 1700.6 tps, lat 58.110 ms stddev 67.577 2024-04-26T01:51:47.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:47 smithi169 ceph-mon[32851]: pgmap v337: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.6 MiB/s wr, 782 op/s 2024-04-26T01:51:47.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:47 smithi053 ceph-mon[32593]: pgmap v337: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.6 MiB/s wr, 782 op/s 2024-04-26T01:51:47.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:47 smithi028 ceph-mon[28247]: pgmap v337: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.6 MiB/s wr, 782 op/s 2024-04-26T01:51:49.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:49 smithi169 ceph-mon[32851]: pgmap v338: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.4 MiB/s wr, 639 op/s 2024-04-26T01:51:49.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:49 smithi053 ceph-mon[32593]: pgmap v338: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.4 MiB/s wr, 639 op/s 2024-04-26T01:51:49.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:49 smithi028 ceph-mon[28247]: pgmap v338: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.4 MiB/s wr, 639 op/s 2024-04-26T01:51:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 595.0 s, 1921.6 tps, lat 52.789 ms stddev 62.523 2024-04-26T01:51:51.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:51 smithi169 ceph-mon[32851]: pgmap v339: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 16 MiB/s wr, 960 op/s 2024-04-26T01:51:51.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:51 smithi053 ceph-mon[32593]: pgmap v339: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 16 MiB/s wr, 960 op/s 2024-04-26T01:51:51.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:51 smithi028 ceph-mon[28247]: pgmap v339: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 16 MiB/s wr, 960 op/s 2024-04-26T01:51:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.982+0000 7f2b89200640 1 -- 172.21.15.28:0/315388208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b8410e840 msgr2=0x7f2b84110c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.982+0000 7f2b89200640 1 --2- 172.21.15.28:0/315388208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b8410e840 0x7f2b84110c30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009860 tx=0x7f2b7002a3d0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.982+0000 7f2b89200640 1 -- 172.21.15.28:0/315388208 shutdown_connections 2024-04-26T01:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.982+0000 7f2b89200640 1 --2- 172.21.15.28:0/315388208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b8410e840 0x7f2b84110c30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.982+0000 7f2b89200640 1 --2- 172.21.15.28:0/315388208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b84074b70 0x7f2b8410e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.982+0000 7f2b89200640 1 --2- 172.21.15.28:0/315388208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b84074250 0x7f2b84074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.982+0000 7f2b89200640 1 -- 172.21.15.28:0/315388208 >> 172.21.15.28:0/315388208 conn(0x7f2b840793e0 msgr2=0x7f2b840797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.985+0000 7f2b89200640 1 -- 172.21.15.28:0/315388208 shutdown_connections 2024-04-26T01:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.988+0000 7f2b89200640 1 -- 172.21.15.28:0/315388208 wait complete. 2024-04-26T01:51:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.990+0000 7f2b89200640 1 Processor -- start 2024-04-26T01:51:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.990+0000 7f2b89200640 1 -- start start 2024-04-26T01:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.990+0000 7f2b89200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b84074250 0x7f2b841a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.990+0000 7f2b89200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b84074b70 0x7f2b841a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b89200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8410e840 0x7f2b8419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b89200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b84114490 con 0x7f2b8410e840 2024-04-26T01:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b89200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b84114310 con 0x7f2b84074250 2024-04-26T01:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b89200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2b84114610 con 0x7f2b84074b70 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b84074250 0x7f2b841a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b84074250 0x7f2b841a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35628/0 (socket says 172.21.15.28:35628) 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b81e00640 1 -- 172.21.15.28:0/4222599112 learned_addr learned my addr 172.21.15.28:0/4222599112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b82800640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8410e840 0x7f2b8419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.991+0000 7f2b81400640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b84074b70 0x7f2b841a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.992+0000 7f2b81400640 1 -- 172.21.15.28:0/4222599112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b84074250 msgr2=0x7f2b841a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.992+0000 7f2b81400640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b84074250 0x7f2b841a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.992+0000 7f2b81400640 1 -- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8410e840 msgr2=0x7f2b8419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.992+0000 7f2b81400640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8410e840 0x7f2b8419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.992+0000 7f2b81400640 1 -- 172.21.15.28:0/4222599112 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8419ae40 con 0x7f2b84074b70 2024-04-26T01:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.992+0000 7f2b81400640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b84074b70 0x7f2b841a09b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400b3f0 tx=0x7f2b7400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.993+0000 7f2b7aa00640 1 -- 172.21.15.28:0/4222599112 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74015020 con 0x7f2b84074b70 2024-04-26T01:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.993+0000 7f2b7aa00640 1 -- 172.21.15.28:0/4222599112 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b74011ce0 con 0x7f2b84074b70 2024-04-26T01:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.993+0000 7f2b7aa00640 1 -- 172.21.15.28:0/4222599112 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74015020 con 0x7f2b84074b70 2024-04-26T01:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.993+0000 7f2b89200640 1 -- 172.21.15.28:0/4222599112 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8419b130 con 0x7f2b84074b70 2024-04-26T01:51:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.993+0000 7f2b89200640 1 -- 172.21.15.28:0/4222599112 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b841a7330 con 0x7f2b84074b70 2024-04-26T01:51:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:51.998+0000 7f2b7aa00640 1 -- 172.21.15.28:0/4222599112 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2b74018020 con 0x7f2b84074b70 2024-04-26T01:51:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.001+0000 7f2b7aa00640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2b680a4040 0x7f2b680a6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.001+0000 7f2b8bfdc740 1 Processor -- start 2024-04-26T01:51:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.001+0000 7f2b8bfdc740 1 -- start start 2024-04-26T01:51:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b81e00640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2b680a4040 0x7f2b680a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999addf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562499906900 con 0x5624999a0820 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5624998acef0 con 0x5624999aba00 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5624999028c0 con 0x5624999a1140 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b7aa00640 1 -- 172.21.15.28:0/4222599112 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f2b740cf9c0 con 0x7f2b84074b70 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b82800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b82800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35644/0 (socket says 172.21.15.28:35644) 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b82800640 1 -- 172.21.15.28:0/1505829715 learned_addr learned my addr 172.21.15.28:0/1505829715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.002+0000 7f2b81400640 1 --2- 172.21.15.28:0/1505829715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.003+0000 7f2b81400640 1 -- 172.21.15.28:0/1505829715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 msgr2=0x5624999addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.003+0000 7f2b81400640 1 --2- 172.21.15.28:0/1505829715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.003+0000 7f2b81400640 1 -- 172.21.15.28:0/1505829715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 msgr2=0x5624999a0c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:51:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.003+0000 7f2b81400640 1 --2- 172.21.15.28:0/1505829715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999a0c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.003+0000 7f2b81400640 1 -- 172.21.15.28:0/1505829715 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562499619210 con 0x5624999a1140 2024-04-26T01:51:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.003+0000 7f2b81400640 1 --2- 172.21.15.28:0/1505829715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999ab4c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2b740cd220 tx=0x7f2b740a4860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=577aa2040a5c417c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.004+0000 7f2b6f400640 1 -- 172.21.15.28:0/1505829715 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74010380 con 0x5624999a1140 2024-04-26T01:51:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.004+0000 7f2b6f400640 1 -- 172.21.15.28:0/1505829715 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b74010520 con 0x5624999a1140 2024-04-26T01:51:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.004+0000 7f2b81e00640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2b680a4040 0x7f2b680a6500 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009fd0 tx=0x7f2b7c010040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.004+0000 7f2b6f400640 1 -- 172.21.15.28:0/1505829715 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74010380 con 0x5624999a1140 2024-04-26T01:51:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.005+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1505829715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 msgr2=0x5624999ab4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.005+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1505829715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999ab4c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2b740cd220 tx=0x7f2b740a4860 comp rx=0 tx=0).stop 2024-04-26T01:51:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.006+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1505829715 shutdown_connections 2024-04-26T01:51:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.006+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1505829715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.006+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1505829715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999ab4c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.006+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1505829715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.006+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1505829715 >> 172.21.15.28:0/1505829715 conn(0x56249999cd30 msgr2=0x56249999d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:52.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.008+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1505829715 shutdown_connections 2024-04-26T01:51:52.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.008+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1505829715 wait complete. 2024-04-26T01:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.009+0000 7f2b8bfdc740 1 -- start start 2024-04-26T01:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.010+0000 7f2b8bfdc740 1 Processor -- start 2024-04-26T01:51:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.012+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999a89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.012+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a1140 0x5624999a8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.012+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999a9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.012+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624999ae5d0 con 0x5624999a1140 2024-04-26T01:51:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.012+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5624998c0310 con 0x5624999aba00 2024-04-26T01:51:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.012+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5624999ae750 con 0x5624999a0820 2024-04-26T01:51:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.013+0000 7f2b8bfdc740 10 client.0 _ll_register_callbacks cb 0x56249988ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.014+0000 7f2b81400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a1140 0x5624999a8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.014+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999a89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.014+0000 7f2b81400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a1140 0x5624999a8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53684/0 (socket says 172.21.15.28:53684) 2024-04-26T01:51:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.014+0000 7f2b81400640 1 -- 172.21.15.28:0/1750119281 learned_addr learned my addr 172.21.15.28:0/1750119281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.014+0000 7f2b82800640 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.015+0000 7f2b81e00640 1 -- 172.21.15.28:0/1750119281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 msgr2=0x5624999a9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.015+0000 7f2b81e00640 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999a9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.015+0000 7f2b81e00640 1 -- 172.21.15.28:0/1750119281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a1140 msgr2=0x5624999a8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.015+0000 7f2b81e00640 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a1140 0x5624999a8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.015+0000 7f2b81e00640 1 -- 172.21.15.28:0/1750119281 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624996e5620 con 0x5624999a0820 2024-04-26T01:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.016+0000 7f2b82800640 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999a9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.016+0000 7f2b81e00640 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999a89b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0043b0 tx=0x7f2b7c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60f9b3474eecd33d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.016+0000 7f2b63e00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.017+0000 7f2b63e00640 1 -- 172.21.15.28:0/1750119281 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c00dc10 con 0x5624999a0820 2024-04-26T01:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.017+0000 7f2b63e00640 1 -- 172.21.15.28:0/1750119281 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f2b7c00ddb0 con 0x5624999a0820 2024-04-26T01:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.017+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1750119281 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b74010520 con 0x5624999a0820 2024-04-26T01:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.018+0000 7f2b63e00640 1 -- 172.21.15.28:0/1750119281 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2b7c004b20 con 0x5624999a0820 2024-04-26T01:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.018+0000 7f2b8bfdc740 10 client.26155 fetch_fsmap learned FSMap version 42 2024-04-26T01:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.018+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1750119281 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5624999ba890 con 0x5624999a0820 2024-04-26T01:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.019+0000 7f2b63e00640 1 -- 172.21.15.28:0/1750119281 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b74010520 con 0x5624999a0820 2024-04-26T01:51:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.019+0000 7f2b63e00640 1 -- 172.21.15.28:0/1750119281 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f2b7c00b040 con 0x5624999a0820 2024-04-26T01:51:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.020+0000 7f2b8bfdc740 10 client.26155 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:51:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.021+0000 7f2b8bfdc740 10 client.26155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.021+0000 7f2b8bfdc740 20 client.26155 populate_metadata read hostname 'smithi028' 2024-04-26T01:51:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.021+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5624999bbda0 0x5624999be1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.021+0000 7f2b8bfdc740 4 client.26155 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.021+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1750119281 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56249991a8b0 con 0x5624999bbda0 2024-04-26T01:51:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.021+0000 7f2b81400640 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5624999bbda0 0x5624999be1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.022+0000 7f2b81400640 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5624999bbda0 0x5624999be1d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca1fe5043f6c9ef9 server_cookie=e871fdc385d22594 in_seq=0 out_seq=0 2024-04-26T01:51:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.022+0000 7f2b63e00640 10 client.26155 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:52.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.023+0000 7f2b63e00640 1 -- 172.21.15.28:0/1750119281 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56249991a8b0 con 0x5624999bbda0 2024-04-26T01:51:52.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.023+0000 7f2b63e00640 10 client.26155 handle_command_reply: tid=0 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.024+0000 7f2b8bfdc740 1 client.26155 shutdown 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.024+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1750119281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 msgr2=0x5624999a89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.024+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999a89b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0043b0 tx=0x7f2b7c004a40 comp rx=0 tx=0).stop 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1750119281 shutdown_connections 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5624999bbda0 0x5624999be1d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999a9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a1140 0x5624999a8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1750119281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999a89b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1750119281 >> 172.21.15.28:0/1750119281 conn(0x56249999cd30 msgr2=0x5624999a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b63e00640 0 client.26155 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1750119281 shutdown_connections 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.025+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1750119281 wait complete. 2024-04-26T01:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.026+0000 7f2b8bfdc740 20 client.26155 trim_cache size 0 max 16384 2024-04-26T01:51:52.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.054+0000 7f2b8bfdc740 1 Processor -- start 2024-04-26T01:51:52.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.054+0000 7f2b8bfdc740 1 -- start start 2024-04-26T01:51:52.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999b7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999a1140 0x5624999a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999aba00 0x5624999c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b640111f0 con 0x5624999aba00 2024-04-26T01:51:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400ea00 con 0x5624999a1140 2024-04-26T01:51:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400c870 con 0x5624999a0820 2024-04-26T01:51:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999b7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46572/0 (socket says 172.21.15.28:46572) 2024-04-26T01:51:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b81e00640 1 -- 172.21.15.28:0/1334167992 learned_addr learned my addr 172.21.15.28:0/1334167992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.055+0000 7f2b81400640 1 --2- 172.21.15.28:0/1334167992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999a1140 0x5624999a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.056+0000 7f2b81e00640 1 -- 172.21.15.28:0/1334167992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999a1140 msgr2=0x5624999a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.056+0000 7f2b81e00640 1 --2- 172.21.15.28:0/1334167992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999a1140 0x5624999a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.056+0000 7f2b81e00640 1 -- 172.21.15.28:0/1334167992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999aba00 msgr2=0x5624999c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.056+0000 7f2b81e00640 1 --2- 172.21.15.28:0/1334167992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999aba00 0x5624999c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.056+0000 7f2b81e00640 1 -- 172.21.15.28:0/1334167992 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624999a3720 con 0x5624999a0820 2024-04-26T01:51:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.056+0000 7f2b81e00640 1 --2- 172.21.15.28:0/1334167992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999b7490 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00efd0 tx=0x7f2b7c024900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38b57917c16d0668 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.057+0000 7f2b6ea00640 1 -- 172.21.15.28:0/1334167992 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c032070 con 0x5624999a0820 2024-04-26T01:51:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.057+0000 7f2b6ea00640 1 -- 172.21.15.28:0/1334167992 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7c0095c0 con 0x5624999a0820 2024-04-26T01:51:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.057+0000 7f2b6ea00640 1 -- 172.21.15.28:0/1334167992 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c02e690 con 0x5624999a0820 2024-04-26T01:51:52.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1334167992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 msgr2=0x5624999b7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:52.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1334167992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999b7490 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00efd0 tx=0x7f2b7c024900 comp rx=0 tx=0).stop 2024-04-26T01:51:52.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1334167992 shutdown_connections 2024-04-26T01:51:52.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1334167992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999aba00 0x5624999c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1334167992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999a1140 0x5624999a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/1334167992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a0820 0x5624999b7490 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1334167992 >> 172.21.15.28:0/1334167992 conn(0x56249999cd30 msgr2=0x56249999be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1334167992 shutdown_connections 2024-04-26T01:51:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/1334167992 wait complete. 2024-04-26T01:51:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.058+0000 7f2b8bfdc740 1 -- start start 2024-04-26T01:51:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.059+0000 7f2b8bfdc740 1 Processor -- start 2024-04-26T01:51:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.060+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.060+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.060+0000 7f2b8bfdc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.060+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400cec0 con 0x5624999a0820 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.060+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400e330 con 0x5624999aba00 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.060+0000 7f2b8bfdc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400d590 con 0x5624999a1140 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.061+0000 7f2b8bfdc740 10 client.0 _ll_register_callbacks cb 0x7f2b7c004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.061+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.061+0000 7f2b81400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.061+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53692/0 (socket says 172.21.15.28:53692) 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.061+0000 7f2b81e00640 1 -- 172.21.15.28:0/3606556848 learned_addr learned my addr 172.21.15.28:0/3606556848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.061+0000 7f2b81400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46578/0 (socket says 172.21.15.28:46578) 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b82800640 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b81400640 1 -- 172.21.15.28:0/3606556848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 msgr2=0x5624999b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b81400640 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b81400640 1 -- 172.21.15.28:0/3606556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 msgr2=0x5624999b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b81400640 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b81400640 1 -- 172.21.15.28:0/3606556848 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624999b1c60 con 0x5624999a1140 2024-04-26T01:51:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b82800640 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b81e00640 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b81400640 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999b1030 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2b74000c00 tx=0x7f2b7409ba00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=586c661887956de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b63e00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.062+0000 7f2b63e00640 1 -- 172.21.15.28:0/3606556848 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74010b50 con 0x5624999a1140 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b63e00640 1 -- 172.21.15.28:0/3606556848 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f2b74010cf0 con 0x5624999a1140 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/3606556848 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562499914320 con 0x5624999a1140 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b63e00640 1 -- 172.21.15.28:0/3606556848 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2b74011830 con 0x5624999a1140 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 10 client.26161 fetch_fsmap learned FSMap version 42 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/3606556848 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562499a2aa80 con 0x5624999a1140 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b63e00640 1 -- 172.21.15.28:0/3606556848 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562499914320 con 0x5624999a1140 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b63e00640 1 -- 172.21.15.28:0/3606556848 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f2b74097980 con 0x5624999a1140 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 10 client.26161 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 10 client.26161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 20 client.26161 populate_metadata read hostname 'smithi028' 2024-04-26T01:51:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562499a2ad60 0x562499a2d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 4 client.26161 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.063+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/3606556848 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56249990e190 con 0x562499a2ad60 2024-04-26T01:51:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.064+0000 7f2b81e00640 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562499a2ad60 0x562499a2d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.064+0000 7f2b81e00640 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562499a2ad60 0x562499a2d150 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc7f317f331badf6 server_cookie=b99d66c23318074f in_seq=0 out_seq=0 2024-04-26T01:51:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.064+0000 7f2b63e00640 10 client.26161 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:52.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.081+0000 7f2b63e00640 1 -- 172.21.15.28:0/3606556848 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56249990e190 con 0x562499a2ad60 2024-04-26T01:51:52.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.081+0000 7f2b63e00640 10 client.26161 handle_command_reply: tid=0 2024-04-26T01:51:52.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.082+0000 7f2b8bfdc740 1 client.26161 shutdown 2024-04-26T01:51:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.082+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/3606556848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 msgr2=0x5624999b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.082+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999b1030 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2b74000c00 tx=0x7f2b7409ba00 comp rx=0 tx=0).stop 2024-04-26T01:51:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/3606556848 shutdown_connections 2024-04-26T01:51:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562499a2ad60 0x562499a2d150 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624999aba00 0x5624999b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5624999a1140 0x5624999b1030 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2b74000c00 tx=0x7f2b7409ba00 comp rx=0 tx=0).stop 2024-04-26T01:51:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b8bfdc740 1 --2- 172.21.15.28:0/3606556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624999a0820 0x5624999b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/3606556848 >> 172.21.15.28:0/3606556848 conn(0x56249999cd30 msgr2=0x5624999a8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b63e00640 0 client.26161 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:51:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/3606556848 shutdown_connections 2024-04-26T01:51:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.083+0000 7f2b8bfdc740 1 -- 172.21.15.28:0/3606556848 wait complete. 2024-04-26T01:51:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.084+0000 7f2b8bfdc740 20 client.26161 trim_cache size 0 max 16384 2024-04-26T01:51:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:51:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:51:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:51:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:51:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.084+0000 7f2b89200640 1 -- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2b680a4040 msgr2=0x7f2b680a6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2b680a4040 0x7f2b680a6500 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009fd0 tx=0x7f2b7c010040 comp rx=0 tx=0).stop 2024-04-26T01:51:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 -- 172.21.15.28:0/4222599112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b84074b70 msgr2=0x7f2b841a09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b84074b70 0x7f2b841a09b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400b3f0 tx=0x7f2b7400b8b0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 -- 172.21.15.28:0/4222599112 shutdown_connections 2024-04-26T01:51:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2b680a4040 0x7f2b680a6500 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8410e840 0x7f2b8419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b84074b70 0x7f2b841a09b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 --2- 172.21.15.28:0/4222599112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b84074250 0x7f2b841a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.085+0000 7f2b89200640 1 -- 172.21.15.28:0/4222599112 >> 172.21.15.28:0/4222599112 conn(0x7f2b840793e0 msgr2=0x7f2b840fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:52.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.086+0000 7f2b89200640 1 -- 172.21.15.28:0/4222599112 shutdown_connections 2024-04-26T01:51:52.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:51:52.086+0000 7f2b89200640 1 -- 172.21.15.28:0/4222599112 wait complete. 2024-04-26T01:51:53.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:53 smithi169 ceph-mon[32851]: pgmap v340: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 13 MiB/s wr, 677 op/s 2024-04-26T01:51:53.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:53 smithi053 ceph-mon[32593]: pgmap v340: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 13 MiB/s wr, 677 op/s 2024-04-26T01:51:53.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:53 smithi028 ceph-mon[28247]: pgmap v340: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 13 MiB/s wr, 677 op/s 2024-04-26T01:51:55.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:55 smithi169 ceph-mon[32851]: pgmap v341: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 17 MiB/s wr, 796 op/s 2024-04-26T01:51:55.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:55 smithi053 ceph-mon[32593]: pgmap v341: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 17 MiB/s wr, 796 op/s 2024-04-26T01:51:55.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:55 smithi028 ceph-mon[28247]: pgmap v341: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 17 MiB/s wr, 796 op/s 2024-04-26T01:51:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 600.0 s, 1989.6 tps, lat 49.996 ms stddev 60.704 2024-04-26T01:51:57.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:57 smithi169 ceph-mon[32851]: pgmap v342: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 20 MiB/s wr, 896 op/s 2024-04-26T01:51:57.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:57 smithi053 ceph-mon[32593]: pgmap v342: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 20 MiB/s wr, 896 op/s 2024-04-26T01:51:57.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:57 smithi028 ceph-mon[28247]: pgmap v342: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 20 MiB/s wr, 896 op/s 2024-04-26T01:51:59.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:51:59 smithi169 ceph-mon[32851]: pgmap v343: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 16 MiB/s wr, 653 op/s 2024-04-26T01:51:59.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:51:59 smithi053 ceph-mon[32593]: pgmap v343: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 16 MiB/s wr, 653 op/s 2024-04-26T01:51:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:51:59 smithi028 ceph-mon[28247]: pgmap v343: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 16 MiB/s wr, 653 op/s 2024-04-26T01:52:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 605.0 s, 1586.1 tps, lat 56.983 ms stddev 81.950 2024-04-26T01:52:01.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:01 smithi169 ceph-mon[32851]: pgmap v344: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 22 MiB/s wr, 927 op/s 2024-04-26T01:52:01.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:01 smithi053 ceph-mon[32593]: pgmap v344: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 22 MiB/s wr, 927 op/s 2024-04-26T01:52:01.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:01 smithi028 ceph-mon[28247]: pgmap v344: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 22 MiB/s wr, 927 op/s 2024-04-26T01:52:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:03 smithi169 ceph-mon[32851]: pgmap v345: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 606 op/s 2024-04-26T01:52:03.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:03 smithi169 ceph-mon[32851]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:52:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:03 smithi053 ceph-mon[32593]: pgmap v345: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 606 op/s 2024-04-26T01:52:03.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:03 smithi053 ceph-mon[32593]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:52:03.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:03 smithi028 ceph-mon[28247]: pgmap v345: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 606 op/s 2024-04-26T01:52:03.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:03 smithi028 ceph-mon[28247]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 -- 172.21.15.28:0/3362983211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee0100b20 msgr2=0x7f1ee0100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3362983211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee0100b20 0x7f1ee0100f00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009900 tx=0x7f1ed002a410 comp rx=0 tx=0).stop 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 -- 172.21.15.28:0/3362983211 shutdown_connections 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3362983211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ee010e6d0 0x7f1ee0110ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3362983211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee01014d0 0x7f1ee010e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3362983211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee0100b20 0x7f1ee0100f00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 -- 172.21.15.28:0/3362983211 >> 172.21.15.28:0/3362983211 conn(0x7f1ee00fc770 msgr2=0x7f1ee00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 -- 172.21.15.28:0/3362983211 shutdown_connections 2024-04-26T01:52:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.249+0000 7f1edfe00640 1 -- 172.21.15.28:0/3362983211 wait complete. 2024-04-26T01:52:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.250+0000 7f1edfe00640 1 Processor -- start 2024-04-26T01:52:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edfe00640 1 -- start start 2024-04-26T01:52:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ee0100b20 0x7f1ee01a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee01014d0 0x7f1ee01a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee010e6d0 0x7f1ee019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0114250 con 0x7f1ee0100b20 2024-04-26T01:52:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edfe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee01140d0 con 0x7f1ee010e6d0 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edfe00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee01143d0 con 0x7f1ee01014d0 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1ede000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee01014d0 0x7f1ee01a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee010e6d0 0x7f1ee019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1edea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ee0100b20 0x7f1ee01a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1ede000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee01014d0 0x7f1ee01a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58852/0 (socket says 172.21.15.28:58852) 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.251+0000 7f1ede000640 1 -- 172.21.15.28:0/3424070884 learned_addr learned my addr 172.21.15.28:0/3424070884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1edf400640 1 -- 172.21.15.28:0/3424070884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee01014d0 msgr2=0x7f1ee01a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1edf400640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee01014d0 0x7f1ee01a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1edf400640 1 -- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ee0100b20 msgr2=0x7f1ee01a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1edf400640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ee0100b20 0x7f1ee01a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1edf400640 1 -- 172.21.15.28:0/3424070884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee019ad70 con 0x7f1ee010e6d0 2024-04-26T01:52:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1ede000640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee01014d0 0x7f1ee01a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1edf400640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee010e6d0 0x7f1ee019a5f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0101a70 tx=0x7f1ed800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1ed7400640 1 -- 172.21.15.28:0/3424070884 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800cdf0 con 0x7f1ee010e6d0 2024-04-26T01:52:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1edfe00640 1 -- 172.21.15.28:0/3424070884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee019b060 con 0x7f1ee010e6d0 2024-04-26T01:52:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1ed7400640 1 -- 172.21.15.28:0/3424070884 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ed8005020 con 0x7f1ee010e6d0 2024-04-26T01:52:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.252+0000 7f1ed7400640 1 -- 172.21.15.28:0/3424070884 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8010690 con 0x7f1ee010e6d0 2024-04-26T01:52:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.253+0000 7f1edfe00640 1 -- 172.21.15.28:0/3424070884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee01a7300 con 0x7f1ee010e6d0 2024-04-26T01:52:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.258+0000 7f1ee840d740 1 Processor -- start 2024-04-26T01:52:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.258+0000 7f1ed7400640 1 -- 172.21.15.28:0/3424070884 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1ed8005460 con 0x7f1ee010e6d0 2024-04-26T01:52:05.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.260+0000 7f1ee840d740 1 -- start start 2024-04-26T01:52:05.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.260+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d644885a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.260+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64492e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.260+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64495790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.260+0000 7f1ee840d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d643e5900 con 0x561d644881c0 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.260+0000 7f1ee840d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561d6438bef0 con 0x561d644933a0 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.260+0000 7f1ee840d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x561d643e18c0 con 0x561d64488ae0 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.261+0000 7f1ede000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d644885a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.261+0000 7f1ede000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d644885a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39354/0 (socket says 172.21.15.28:39354) 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.261+0000 7f1ede000640 1 -- 172.21.15.28:0/4052947811 learned_addr learned my addr 172.21.15.28:0/4052947811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.261+0000 7f1ed7400640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ec40a3d90 0x7f1ec40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.261+0000 7f1ede000640 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64495790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.261+0000 7f1edea00640 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64492e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.262+0000 7f1ed7400640 1 -- 172.21.15.28:0/3424070884 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8557+0+0 (secure 0 0 0) 0x7f1ed80d0ed0 con 0x7f1ee010e6d0 2024-04-26T01:52:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.262+0000 7f1ede000640 1 -- 172.21.15.28:0/4052947811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 msgr2=0x561d64492e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.262+0000 7f1ede000640 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64492e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.262+0000 7f1ede000640 1 -- 172.21.15.28:0/4052947811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 msgr2=0x561d64495790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.262+0000 7f1ede000640 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64495790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.262+0000 7f1ede000640 1 -- 172.21.15.28:0/4052947811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d640f8210 con 0x561d644881c0 2024-04-26T01:52:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.262+0000 7f1edf400640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ec40a3d90 0x7f1ec40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.263+0000 7f1ede000640 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d644885a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80097f0 tx=0x7f1ec8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aeee1ab868961f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:05.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.263+0000 7f1edf400640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ec40a3d90 0x7f1ec40a6250 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1ed801a040 tx=0x7f1ed8005750 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:05.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.264+0000 7f1ecf400640 1 -- 172.21.15.28:0/4052947811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8013130 con 0x561d644881c0 2024-04-26T01:52:05.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.264+0000 7f1ecf400640 1 -- 172.21.15.28:0/4052947811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ec80132d0 con 0x561d644881c0 2024-04-26T01:52:05.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.264+0000 7f1ecf400640 1 -- 172.21.15.28:0/4052947811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80054f0 con 0x561d644881c0 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.265+0000 7f1ee840d740 1 -- 172.21.15.28:0/4052947811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 msgr2=0x561d644885a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.265+0000 7f1ee840d740 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d644885a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80097f0 tx=0x7f1ec8005f90 comp rx=0 tx=0).stop 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.265+0000 7f1ee840d740 1 -- 172.21.15.28:0/4052947811 shutdown_connections 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.265+0000 7f1ee840d740 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64495790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.265+0000 7f1ee840d740 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64492e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.265+0000 7f1ee840d740 1 --2- 172.21.15.28:0/4052947811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d644885a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.265+0000 7f1ee840d740 1 -- 172.21.15.28:0/4052947811 >> 172.21.15.28:0/4052947811 conn(0x561d644846d0 msgr2=0x561d64484ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.266+0000 7f1ee840d740 1 -- 172.21.15.28:0/4052947811 shutdown_connections 2024-04-26T01:52:05.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.266+0000 7f1ee840d740 1 -- 172.21.15.28:0/4052947811 wait complete. 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.266+0000 7f1ee840d740 1 -- start start 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.266+0000 7f1ee840d740 1 Processor -- start 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d64490320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64490860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64490da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1ee840d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d64495f70 con 0x561d644881c0 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1ee840d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561d6439f310 con 0x561d644933a0 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1ee840d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x561d644960f0 con 0x561d64488ae0 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1ee840d740 10 client.0 _ll_register_callbacks cb 0x561d6436dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1edea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64490860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1edea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64490860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58868/0 (socket says 172.21.15.28:58868) 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1edea00640 1 -- 172.21.15.28:0/2799231335 learned_addr learned my addr 172.21.15.28:0/2799231335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:05.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.267+0000 7f1ede000640 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d64490320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1edea00640 1 -- 172.21.15.28:0/2799231335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 msgr2=0x561d64490da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1ede000640 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64490da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1edea00640 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64490da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1edea00640 1 -- 172.21.15.28:0/2799231335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 msgr2=0x561d64490320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1edea00640 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d64490320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1edea00640 1 -- 172.21.15.28:0/2799231335 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d641c4620 con 0x561d64488ae0 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1ede000640 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64490da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1edea00640 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64490860 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009230 tx=0x7f1ed0004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22dd304ec3a8c940 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.268+0000 7f1ec3e00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.269+0000 7f1ec3e00640 1 -- 172.21.15.28:0/2799231335 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0035040 con 0x561d64488ae0 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.269+0000 7f1ec3e00640 1 -- 172.21.15.28:0/2799231335 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8557+0+0 (secure 0 0 0) 0x7f1ed0005450 con 0x561d64488ae0 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.269+0000 7f1ee840d740 1 -- 172.21.15.28:0/2799231335 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ec80132d0 con 0x561d64488ae0 2024-04-26T01:52:05.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.269+0000 7f1ec3e00640 1 -- 172.21.15.28:0/2799231335 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1ed0005650 con 0x561d64488ae0 2024-04-26T01:52:05.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.269+0000 7f1ee840d740 10 client.26173 fetch_fsmap learned FSMap version 42 2024-04-26T01:52:05.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.269+0000 7f1ee840d740 1 -- 172.21.15.28:0/2799231335 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561d644a21f0 con 0x561d64488ae0 2024-04-26T01:52:05.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.270+0000 7f1ec3e00640 1 -- 172.21.15.28:0/2799231335 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ec80132d0 con 0x561d64488ae0 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.270+0000 7f1ec3e00640 1 -- 172.21.15.28:0/2799231335 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f1ed00039f0 con 0x561d64488ae0 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.270+0000 7f1ee840d740 10 client.26173 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.270+0000 7f1ee840d740 10 client.26173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.270+0000 7f1ee840d740 20 client.26173 populate_metadata read hostname 'smithi028' 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.270+0000 7f1ee840d740 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x561d644a36b0 0x561d644a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.270+0000 7f1ee840d740 4 client.26173 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.270+0000 7f1ee840d740 1 -- 172.21.15.28:0/2799231335 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d643f98b0 con 0x561d644a36b0 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.271+0000 7f1ede000640 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x561d644a36b0 0x561d644a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.271+0000 7f1ede000640 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x561d644a36b0 0x561d644a5b10 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cec82e8335addf76 server_cookie=4b9f6304ebbf829d in_seq=0 out_seq=0 2024-04-26T01:52:05.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.271+0000 7f1ec3e00640 10 client.26173 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:05.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.272+0000 7f1ec3e00640 1 -- 172.21.15.28:0/2799231335 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d643f98b0 con 0x561d644a36b0 2024-04-26T01:52:05.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.272+0000 7f1ec3e00640 10 client.26173 handle_command_reply: tid=0 2024-04-26T01:52:05.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.272+0000 7f1ee840d740 1 client.26173 shutdown 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.272+0000 7f1ee840d740 1 -- 172.21.15.28:0/2799231335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 msgr2=0x561d64490860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.272+0000 7f1ee840d740 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64490860 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009230 tx=0x7f1ed0004340 comp rx=0 tx=0).stop 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.272+0000 7f1ee840d740 1 -- 172.21.15.28:0/2799231335 shutdown_connections 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.272+0000 7f1ee840d740 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x561d644a36b0 0x561d644a5b10 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.273+0000 7f1ee840d740 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d64490da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.273+0000 7f1ee840d740 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d64488ae0 0x561d64490860 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.273+0000 7f1ee840d740 1 --2- 172.21.15.28:0/2799231335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d644881c0 0x561d64490320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.273+0000 7f1ec3e00640 0 client.26173 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.273+0000 7f1ee840d740 1 -- 172.21.15.28:0/2799231335 >> 172.21.15.28:0/2799231335 conn(0x561d644846d0 msgr2=0x561d6448a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.273+0000 7f1ee840d740 1 -- 172.21.15.28:0/2799231335 shutdown_connections 2024-04-26T01:52:05.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.273+0000 7f1ee840d740 1 -- 172.21.15.28:0/2799231335 wait complete. 2024-04-26T01:52:05.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.273+0000 7f1ee840d740 20 client.26173 trim_cache size 0 max 16384 2024-04-26T01:52:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.298+0000 7f1ee840d740 1 Processor -- start 2024-04-26T01:52:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.298+0000 7f1ee840d740 1 -- start start 2024-04-26T01:52:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644881c0 0x561d6449ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d6447aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d644ab420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ee840d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0072660 con 0x561d64488ae0 2024-04-26T01:52:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ee840d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee006fe70 con 0x561d644933a0 2024-04-26T01:52:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ee840d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee006dce0 con 0x561d644881c0 2024-04-26T01:52:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1edea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d6447aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ede000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644881c0 0x561d6449ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ede000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644881c0 0x561d6449ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:58870/0 (socket says 172.21.15.28:58870) 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ede000640 1 -- 172.21.15.28:0/579600139 learned_addr learned my addr 172.21.15.28:0/579600139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1edea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d6447aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39370/0 (socket says 172.21.15.28:39370) 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ede000640 1 --2- 172.21.15.28:0/579600139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d644ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.299+0000 7f1ede000640 1 -- 172.21.15.28:0/579600139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 msgr2=0x561d644ab420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.300+0000 7f1ede000640 1 --2- 172.21.15.28:0/579600139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d644ab420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.300+0000 7f1ede000640 1 -- 172.21.15.28:0/579600139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 msgr2=0x561d6447aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.300+0000 7f1ede000640 1 --2- 172.21.15.28:0/579600139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d6447aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.300+0000 7f1ede000640 1 -- 172.21.15.28:0/579600139 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d64477a00 con 0x561d644881c0 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.300+0000 7f1ede000640 1 --2- 172.21.15.28:0/579600139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644881c0 0x561d6449ee30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80095a0 tx=0x7f1ec8009c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=467e813c1cb45e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.300+0000 7f1ecea00640 1 -- 172.21.15.28:0/579600139 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8041070 con 0x561d644881c0 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.300+0000 7f1ecea00640 1 -- 172.21.15.28:0/579600139 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ec803a750 con 0x561d644881c0 2024-04-26T01:52:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.300+0000 7f1ecea00640 1 -- 172.21.15.28:0/579600139 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8018040 con 0x561d644881c0 2024-04-26T01:52:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 -- 172.21.15.28:0/579600139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644881c0 msgr2=0x561d6449ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 --2- 172.21.15.28:0/579600139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644881c0 0x561d6449ee30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80095a0 tx=0x7f1ec8009c10 comp rx=0 tx=0).stop 2024-04-26T01:52:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 -- 172.21.15.28:0/579600139 shutdown_connections 2024-04-26T01:52:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 --2- 172.21.15.28:0/579600139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644933a0 0x561d644ab420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 --2- 172.21.15.28:0/579600139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d6447aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 --2- 172.21.15.28:0/579600139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644881c0 0x561d6449ee30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 -- 172.21.15.28:0/579600139 >> 172.21.15.28:0/579600139 conn(0x561d644846d0 msgr2=0x561d6447a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 -- 172.21.15.28:0/579600139 shutdown_connections 2024-04-26T01:52:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.301+0000 7f1ee840d740 1 -- 172.21.15.28:0/579600139 wait complete. 2024-04-26T01:52:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.302+0000 7f1ee840d740 1 -- start start 2024-04-26T01:52:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.302+0000 7f1ee840d740 1 Processor -- start 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644881c0 0x561d64498ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d64499410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ee840d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644933a0 0x561d64499950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ee840d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee006e330 con 0x561d64488ae0 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ee840d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee006f7a0 con 0x561d644881c0 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ee840d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee006ea00 con 0x561d644933a0 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ee840d740 10 client.0 _ll_register_callbacks cb 0x7f1ed0004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644881c0 0x561d64498ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644881c0 0x561d64498ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48606/0 (socket says 172.21.15.28:48606) 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 -- 172.21.15.28:0/38173073 learned_addr learned my addr 172.21.15.28:0/38173073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644933a0 0x561d64499950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1edea00640 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d64499410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 -- 172.21.15.28:0/38173073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644933a0 msgr2=0x561d64499950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644933a0 0x561d64499950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 -- 172.21.15.28:0/38173073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 msgr2=0x561d64499410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d64499410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1ede000640 1 -- 172.21.15.28:0/38173073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d64508e00 con 0x561d644881c0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.303+0000 7f1edea00640 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d64499410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.304+0000 7f1ede000640 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644881c0 0x561d64498ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800fd40 tx=0x7f1ec8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c366c0f61087fc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.304+0000 7f1ec3e00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.304+0000 7f1ec3e00640 1 -- 172.21.15.28:0/38173073 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8041040 con 0x561d644881c0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.304+0000 7f1ec3e00640 1 -- 172.21.15.28:0/38173073 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8557+0+0 (secure 0 0 0) 0x7f1ec803a2a0 con 0x561d644881c0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.304+0000 7f1ee840d740 1 -- 172.21.15.28:0/38173073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ec803a750 con 0x561d644881c0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.304+0000 7f1ec3e00640 1 -- 172.21.15.28:0/38173073 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1ec803a920 con 0x561d644881c0 2024-04-26T01:52:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.304+0000 7f1ee840d740 10 client.26156 fetch_fsmap learned FSMap version 42 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.304+0000 7f1ee840d740 1 -- 172.21.15.28:0/38173073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561d6450a0f0 con 0x561d644881c0 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1ec3e00640 1 -- 172.21.15.28:0/38173073 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ec803a750 con 0x561d644881c0 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1ec3e00640 1 -- 172.21.15.28:0/38173073 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f1ec80039c0 con 0x561d644881c0 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1ee840d740 10 client.26156 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1ee840d740 10 client.26156 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1ee840d740 20 client.26156 populate_metadata read hostname 'smithi028' 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1ee840d740 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x561d6450a540 0x561d6450c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1ee840d740 4 client.26156 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1ee840d740 1 -- 172.21.15.28:0/38173073 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d643f3320 con 0x561d6450a540 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.305+0000 7f1edea00640 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x561d6450a540 0x561d6450c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.306+0000 7f1edea00640 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x561d6450a540 0x561d6450c960 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3370b1a18979b53f server_cookie=53c8b32febb6b86a in_seq=0 out_seq=0 2024-04-26T01:52:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.306+0000 7f1ec3e00640 10 client.26156 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ec3e00640 1 -- 172.21.15.28:0/38173073 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d643f3320 con 0x561d6450a540 2024-04-26T01:52:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ec3e00640 10 client.26156 handle_command_reply: tid=0 2024-04-26T01:52:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ee840d740 1 client.26156 shutdown 2024-04-26T01:52:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ee840d740 1 -- 172.21.15.28:0/38173073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644881c0 msgr2=0x561d64498ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ee840d740 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644881c0 0x561d64498ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800fd40 tx=0x7f1ec8009260 comp rx=0 tx=0).stop 2024-04-26T01:52:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ee840d740 1 -- 172.21.15.28:0/38173073 shutdown_connections 2024-04-26T01:52:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ee840d740 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x561d6450a540 0x561d6450c960 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ee840d740 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561d644933a0 0x561d64499950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ee840d740 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d64488ae0 0x561d64499410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.318+0000 7f1ee840d740 1 --2- 172.21.15.28:0/38173073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561d644881c0 0x561d64498ed0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.319+0000 7f1ee840d740 1 -- 172.21.15.28:0/38173073 >> 172.21.15.28:0/38173073 conn(0x561d644846d0 msgr2=0x561d6448a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.319+0000 7f1ec3e00640 0 client.26156 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.319+0000 7f1ee840d740 1 -- 172.21.15.28:0/38173073 shutdown_connections 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.319+0000 7f1ee840d740 1 -- 172.21.15.28:0/38173073 wait complete. 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.319+0000 7f1ee840d740 20 client.26156 trim_cache size 0 max 16384 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:52:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:52:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 -- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ec40a3d90 msgr2=0x7f1ec40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ec40a3d90 0x7f1ec40a6250 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1ed801a040 tx=0x7f1ed8005750 comp rx=0 tx=0).stop 2024-04-26T01:52:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 -- 172.21.15.28:0/3424070884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee010e6d0 msgr2=0x7f1ee019a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee010e6d0 0x7f1ee019a5f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0101a70 tx=0x7f1ed800ef10 comp rx=0 tx=0).stop 2024-04-26T01:52:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 -- 172.21.15.28:0/3424070884 shutdown_connections 2024-04-26T01:52:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f1ec40a3d90 0x7f1ec40a6250 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee010e6d0 0x7f1ee019a5f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ee01014d0 0x7f1ee01a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 --2- 172.21.15.28:0/3424070884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ee0100b20 0x7f1ee01a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 -- 172.21.15.28:0/3424070884 >> 172.21.15.28:0/3424070884 conn(0x7f1ee00fc770 msgr2=0x7f1ee00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 -- 172.21.15.28:0/3424070884 shutdown_connections 2024-04-26T01:52:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:05.320+0000 7f1edfe00640 1 -- 172.21.15.28:0/3424070884 wait complete. 2024-04-26T01:52:05.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:05 smithi169 ceph-mon[32851]: pgmap v347: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 14 MiB/s wr, 636 op/s 2024-04-26T01:52:05.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:05 smithi053 ceph-mon[32593]: pgmap v347: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 14 MiB/s wr, 636 op/s 2024-04-26T01:52:05.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:05 smithi028 ceph-mon[28247]: pgmap v347: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 14 MiB/s wr, 636 op/s 2024-04-26T01:52:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 610.0 s, 182.4 tps, lat 560.045 ms stddev 378.642 2024-04-26T01:52:07.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:07 smithi169 ceph-mon[32851]: pgmap v348: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.3 MiB/s wr, 502 op/s 2024-04-26T01:52:07.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:07 smithi053 ceph-mon[32593]: pgmap v348: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.3 MiB/s wr, 502 op/s 2024-04-26T01:52:07.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:07 smithi028 ceph-mon[28247]: pgmap v348: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.3 MiB/s wr, 502 op/s 2024-04-26T01:52:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:09 smithi169 ceph-mon[32851]: pgmap v349: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.3 MiB/s wr, 502 op/s 2024-04-26T01:52:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:09 smithi169 ceph-mon[32851]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:52:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:09 smithi053 ceph-mon[32593]: pgmap v349: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.3 MiB/s wr, 502 op/s 2024-04-26T01:52:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:09 smithi053 ceph-mon[32593]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:52:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:09 smithi028 ceph-mon[28247]: pgmap v349: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.3 MiB/s wr, 502 op/s 2024-04-26T01:52:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:09 smithi028 ceph-mon[28247]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:52:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 615.0 s, 1117.8 tps, lat 95.503 ms stddev 228.056 2024-04-26T01:52:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:11 smithi169 ceph-mon[32851]: pgmap v351: 97 pgs: 97 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 2.13k op/s 2024-04-26T01:52:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:11 smithi053 ceph-mon[32593]: pgmap v351: 97 pgs: 97 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 2.13k op/s 2024-04-26T01:52:11.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:11 smithi028 ceph-mon[28247]: pgmap v351: 97 pgs: 97 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 2.13k op/s 2024-04-26T01:52:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:13 smithi169 ceph-mon[32851]: pgmap v352: 97 pgs: 97 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 30 MiB/s wr, 1.73k op/s 2024-04-26T01:52:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:13 smithi053 ceph-mon[32593]: pgmap v352: 97 pgs: 97 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 30 MiB/s wr, 1.73k op/s 2024-04-26T01:52:13.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:13 smithi028 ceph-mon[28247]: pgmap v352: 97 pgs: 97 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 30 MiB/s wr, 1.73k op/s 2024-04-26T01:52:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:15 smithi169 ceph-mon[32851]: pgmap v353: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 MiB/s wr, 1.76k op/s 2024-04-26T01:52:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:15 smithi053 ceph-mon[32593]: pgmap v353: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 MiB/s wr, 1.76k op/s 2024-04-26T01:52:15.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:15 smithi028 ceph-mon[28247]: pgmap v353: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 31 MiB/s wr, 1.76k op/s 2024-04-26T01:52:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 620.0 s, 2018.2 tps, lat 50.018 ms stddev 78.403 2024-04-26T01:52:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:17 smithi169 ceph-mon[32851]: pgmap v354: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 MiB/s wr, 1.86k op/s 2024-04-26T01:52:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:17 smithi053 ceph-mon[32593]: pgmap v354: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 MiB/s wr, 1.86k op/s 2024-04-26T01:52:17.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:17 smithi028 ceph-mon[28247]: pgmap v354: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 MiB/s wr, 1.86k op/s 2024-04-26T01:52:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 -- 172.21.15.28:0/4234112610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8440100fd0 msgr2=0x7f84401013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 --2- 172.21.15.28:0/4234112610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8440100fd0 0x7f84401013b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8430009900 tx=0x7f843002a540 comp rx=0 tx=0).stop 2024-04-26T01:52:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 -- 172.21.15.28:0/4234112610 shutdown_connections 2024-04-26T01:52:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 --2- 172.21.15.28:0/4234112610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84401062f0 0x7f8440106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 --2- 172.21.15.28:0/4234112610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84401018f0 0x7f8440105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 --2- 172.21.15.28:0/4234112610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8440100fd0 0x7f84401013b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 -- 172.21.15.28:0/4234112610 >> 172.21.15.28:0/4234112610 conn(0x7f8440077fd0 msgr2=0x7f84400783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 -- 172.21.15.28:0/4234112610 shutdown_connections 2024-04-26T01:52:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.481+0000 7f8445200640 1 -- 172.21.15.28:0/4234112610 wait complete. 2024-04-26T01:52:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.482+0000 7f8445200640 1 Processor -- start 2024-04-26T01:52:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.482+0000 7f8445200640 1 -- start start 2024-04-26T01:52:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.483+0000 7f8445200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8440100fd0 0x7f844019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f8445200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84401018f0 0x7f844019f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f8445200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84401062f0 0x7f84401a3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f8445200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8440118880 con 0x7f8440100fd0 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f8445200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8440118700 con 0x7f84401018f0 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f8445200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8440118a00 con 0x7f84401062f0 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f843e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84401062f0 0x7f84401a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f843d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84401018f0 0x7f844019f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f843e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84401062f0 0x7f84401a3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:34632/0 (socket says 172.21.15.28:34632) 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.484+0000 7f843e800640 1 -- 172.21.15.28:0/1288202030 learned_addr learned my addr 172.21.15.28:0/1288202030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843de00640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8440100fd0 0x7f844019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843d400640 1 -- 172.21.15.28:0/1288202030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84401062f0 msgr2=0x7f84401a3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843d400640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84401062f0 0x7f84401a3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843d400640 1 -- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8440100fd0 msgr2=0x7f844019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843d400640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8440100fd0 0x7f844019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843d400640 1 -- 172.21.15.28:0/1288202030 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84401a3d70 con 0x7f84401018f0 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843de00640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8440100fd0 0x7f844019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843e800640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84401062f0 0x7f84401a3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.485+0000 7f843d400640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84401018f0 0x7f844019f2c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f842c009fd0 tx=0x7f842c00b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.486+0000 7f8436a00640 1 -- 172.21.15.28:0/1288202030 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c015020 con 0x7f84401018f0 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.486+0000 7f8445200640 1 -- 172.21.15.28:0/1288202030 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84401a4060 con 0x7f84401018f0 2024-04-26T01:52:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.486+0000 7f8436a00640 1 -- 172.21.15.28:0/1288202030 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f842c011ce0 con 0x7f84401018f0 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.486+0000 7f8436a00640 1 -- 172.21.15.28:0/1288202030 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c00f2e0 con 0x7f84401018f0 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.486+0000 7f8445200640 1 -- 172.21.15.28:0/1288202030 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84401ab8f0 con 0x7f84401018f0 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.490+0000 7f8447ff1740 1 Processor -- start 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.491+0000 7f8447ff1740 1 -- start start 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.491+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b695c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.491+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.491+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b6a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.491+0000 7f8447ff1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55df6b5fb900 con 0x55df6b695820 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.491+0000 7f8447ff1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55df6b5a1ef0 con 0x55df6b696140 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.491+0000 7f8447ff1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55df6b5f78c0 con 0x55df6b6a0a00 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48222/0 (socket says 172.21.15.28:48222) 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843de00640 1 -- 172.21.15.28:0/1929160021 learned_addr learned my addr 172.21.15.28:0/1929160021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843e800640 1 --2- 172.21.15.28:0/1929160021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b695c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843de00640 1 -- 172.21.15.28:0/1929160021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 msgr2=0x55df6b6a2df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843de00640 1 --2- 172.21.15.28:0/1929160021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b6a2df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843de00640 1 -- 172.21.15.28:0/1929160021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 msgr2=0x55df6b695c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843de00640 1 --2- 172.21.15.28:0/1929160021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b695c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.492+0000 7f843de00640 1 -- 172.21.15.28:0/1929160021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df6b30e210 con 0x55df6b696140 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.493+0000 7f8436a00640 1 -- 172.21.15.28:0/1288202030 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f842c018020 con 0x7f84401018f0 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.493+0000 7f843de00640 1 --2- 172.21.15.28:0/1929160021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a04c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8430006010 tx=0x7f8430004380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c96a3d271604015d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.493+0000 7f842b400640 1 -- 172.21.15.28:0/1929160021 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843003f070 con 0x55df6b696140 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.494+0000 7f842b400640 1 -- 172.21.15.28:0/1929160021 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8430005d10 con 0x55df6b696140 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.494+0000 7f842b400640 1 -- 172.21.15.28:0/1929160021 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843003f070 con 0x55df6b696140 2024-04-26T01:52:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.494+0000 7f8447ff1740 1 -- 172.21.15.28:0/1929160021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 msgr2=0x55df6b6a04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.494+0000 7f8447ff1740 1 --2- 172.21.15.28:0/1929160021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a04c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8430006010 tx=0x7f8430004380 comp rx=0 tx=0).stop 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.495+0000 7f8447ff1740 1 -- 172.21.15.28:0/1929160021 shutdown_connections 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.495+0000 7f8447ff1740 1 --2- 172.21.15.28:0/1929160021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b6a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.495+0000 7f8447ff1740 1 --2- 172.21.15.28:0/1929160021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a04c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.495+0000 7f8447ff1740 1 --2- 172.21.15.28:0/1929160021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b695c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.495+0000 7f8447ff1740 1 -- 172.21.15.28:0/1929160021 >> 172.21.15.28:0/1929160021 conn(0x55df6b691d30 msgr2=0x55df6b692110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.495+0000 7f8447ff1740 1 -- 172.21.15.28:0/1929160021 shutdown_connections 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.495+0000 7f8447ff1740 1 -- 172.21.15.28:0/1929160021 wait complete. 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.495+0000 7f8447ff1740 1 -- start start 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.496+0000 7f8436a00640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f84240a3d50 0x7f84240a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.496+0000 7f8447ff1740 1 Processor -- start 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.496+0000 7f8436a00640 1 -- 172.21.15.28:0/1288202030 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f842c0cffa0 con 0x7f84401018f0 2024-04-26T01:52:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.496+0000 7f843e800640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f84240a3d50 0x7f84240a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.496+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b69d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 0x55df6b69df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b69e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f8447ff1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55df6b6a35d0 con 0x55df6b696140 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f8447ff1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55df6b5b5310 con 0x55df6b695820 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f8447ff1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55df6b6a3750 con 0x55df6b6a0a00 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f8447ff1740 10 client.0 _ll_register_callbacks cb 0x55df6b583cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f843de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b69d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f843d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 0x55df6b69df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f843de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b69d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48236/0 (socket says 172.21.15.28:48236) 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.497+0000 7f843de00640 1 -- 172.21.15.28:0/2027750298 learned_addr learned my addr 172.21.15.28:0/2027750298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.498+0000 7f843e800640 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b69e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.498+0000 7f843e800640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f84240a3d50 0x7f84240a6210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8438009fd0 tx=0x7f8438009680 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.498+0000 7f843e800640 1 -- 172.21.15.28:0/2027750298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 msgr2=0x55df6b69d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.498+0000 7f843e800640 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b69d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.498+0000 7f843e800640 1 -- 172.21.15.28:0/2027750298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 msgr2=0x55df6b69df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.498+0000 7f843e800640 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 0x55df6b69df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.498+0000 7f843e800640 1 -- 172.21.15.28:0/2027750298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df6b3da620 con 0x55df6b6a0a00 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.499+0000 7f843e800640 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b69e460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8438011a90 tx=0x7f8438004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae96f17535336b18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.499+0000 7f841fe00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.499+0000 7f841fe00640 1 -- 172.21.15.28:0/2027750298 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438004d20 con 0x55df6b6a0a00 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.499+0000 7f8447ff1740 1 -- 172.21.15.28:0/2027750298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8430005d10 con 0x55df6b6a0a00 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.499+0000 7f841fe00640 1 -- 172.21.15.28:0/2027750298 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f843801f7b0 con 0x55df6b6a0a00 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.499+0000 7f841fe00640 1 -- 172.21.15.28:0/2027750298 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f843801f9b0 con 0x55df6b6a0a00 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.499+0000 7f8447ff1740 10 client.26188 fetch_fsmap learned FSMap version 42 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.499+0000 7f8447ff1740 1 -- 172.21.15.28:0/2027750298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55df6b6af910 con 0x55df6b6a0a00 2024-04-26T01:52:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f841fe00640 1 -- 172.21.15.28:0/2027750298 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8430005d10 con 0x55df6b6a0a00 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f841fe00640 1 -- 172.21.15.28:0/2027750298 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f8438026030 con 0x55df6b6a0a00 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f8447ff1740 10 client.26188 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f8447ff1740 10 client.26188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f8447ff1740 20 client.26188 populate_metadata read hostname 'smithi028' 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f8447ff1740 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55df6b6b0e90 0x55df6b6b3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f8447ff1740 4 client.26188 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f8447ff1740 1 -- 172.21.15.28:0/2027750298 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df6b60f8b0 con 0x55df6b6b0e90 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.500+0000 7f843de00640 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55df6b6b0e90 0x55df6b6b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.501+0000 7f843de00640 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55df6b6b0e90 0x55df6b6b3280 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eecc1e067a61c876 server_cookie=75923dd851ed6aed in_seq=0 out_seq=0 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.501+0000 7f841fe00640 10 client.26188 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f841fe00640 1 -- 172.21.15.28:0/2027750298 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df6b60f8b0 con 0x55df6b6b0e90 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f841fe00640 10 client.26188 handle_command_reply: tid=0 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 client.26188 shutdown 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 -- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 msgr2=0x55df6b69e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b69e460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8438011a90 tx=0x7f8438004a40 comp rx=0 tx=0).stop 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 -- 172.21.15.28:0/2027750298 shutdown_connections 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55df6b6b0e90 0x55df6b6b3280 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b69e460 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 0x55df6b69df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 --2- 172.21.15.28:0/2027750298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b69d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 -- 172.21.15.28:0/2027750298 >> 172.21.15.28:0/2027750298 conn(0x55df6b691d30 msgr2=0x55df6b697ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f841fe00640 0 client.26188 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 -- 172.21.15.28:0/2027750298 shutdown_connections 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.502+0000 7f8447ff1740 1 -- 172.21.15.28:0/2027750298 wait complete. 2024-04-26T01:52:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.504+0000 7f8447ff1740 20 client.26188 trim_cache size 0 max 16384 2024-04-26T01:52:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.534+0000 7f8447ff1740 1 Processor -- start 2024-04-26T01:52:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.534+0000 7f8447ff1740 1 -- start start 2024-04-26T01:52:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b6ac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 0x55df6b69e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b690d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f8447ff1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84401b6ae0 con 0x55df6b696140 2024-04-26T01:52:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f8447ff1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f84401b42f0 con 0x55df6b695820 2024-04-26T01:52:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f8447ff1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f84401b2160 con 0x55df6b6a0a00 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b6ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b6ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:48250/0 (socket says 172.21.15.28:48250) 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 0x55df6b69e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843de00640 1 -- 172.21.15.28:0/4213636476 learned_addr learned my addr 172.21.15.28:0/4213636476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843e800640 1 --2- 172.21.15.28:0/4213636476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b690d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843de00640 1 -- 172.21.15.28:0/4213636476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 msgr2=0x55df6b690d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843de00640 1 --2- 172.21.15.28:0/4213636476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b690d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843de00640 1 -- 172.21.15.28:0/4213636476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 msgr2=0x55df6b69e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843de00640 1 --2- 172.21.15.28:0/4213636476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 0x55df6b69e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.535+0000 7f843de00640 1 -- 172.21.15.28:0/4213636476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df6b6987d0 con 0x55df6b695820 2024-04-26T01:52:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.536+0000 7f843de00640 1 --2- 172.21.15.28:0/4213636476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b6ac490 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f84300096c0 tx=0x7f8430005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=999f21a8563b5d54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.536+0000 7f842aa00640 1 -- 172.21.15.28:0/4213636476 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843003f040 con 0x55df6b695820 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.536+0000 7f842aa00640 1 -- 172.21.15.28:0/4213636476 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8430050780 con 0x55df6b695820 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.536+0000 7f842aa00640 1 -- 172.21.15.28:0/4213636476 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8430035030 con 0x55df6b695820 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 -- 172.21.15.28:0/4213636476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 msgr2=0x55df6b6ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 --2- 172.21.15.28:0/4213636476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b6ac490 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f84300096c0 tx=0x7f8430005bd0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 -- 172.21.15.28:0/4213636476 shutdown_connections 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 --2- 172.21.15.28:0/4213636476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b690d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 --2- 172.21.15.28:0/4213636476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b696140 0x55df6b69e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 --2- 172.21.15.28:0/4213636476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b695820 0x55df6b6ac490 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 -- 172.21.15.28:0/4213636476 >> 172.21.15.28:0/4213636476 conn(0x55df6b691d30 msgr2=0x55df6b690600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 -- 172.21.15.28:0/4213636476 shutdown_connections 2024-04-26T01:52:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.537+0000 7f8447ff1740 1 -- 172.21.15.28:0/4213636476 wait complete. 2024-04-26T01:52:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.538+0000 7f8447ff1740 1 -- start start 2024-04-26T01:52:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.539+0000 7f8447ff1740 1 Processor -- start 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b6a5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f8447ff1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b7153c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f8447ff1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84401b27b0 con 0x55df6b695820 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f8447ff1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f84401b3c20 con 0x55df6b696140 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f8447ff1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f84401b2e80 con 0x55df6b6a0a00 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f8447ff1740 10 client.0 _ll_register_callbacks cb 0x7f8438011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f843e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b7153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f843e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b7153c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:34662/0 (socket says 172.21.15.28:34662) 2024-04-26T01:52:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f843e800640 1 -- 172.21.15.28:0/3852389012 learned_addr learned my addr 172.21.15.28:0/3852389012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.541+0000 7f843de00640 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b6a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.542+0000 7f843d400640 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.542+0000 7f843e800640 1 -- 172.21.15.28:0/3852389012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 msgr2=0x55df6b6a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.542+0000 7f843e800640 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.542+0000 7f843e800640 1 -- 172.21.15.28:0/3852389012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 msgr2=0x55df6b6a5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.542+0000 7f843e800640 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b6a5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.542+0000 7f843e800640 1 -- 172.21.15.28:0/3852389012 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df6b715900 con 0x55df6b6a0a00 2024-04-26T01:52:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.542+0000 7f843de00640 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b6a5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.543+0000 7f843e800640 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b7153c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8438018040 tx=0x7f84380276a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba4a86e8fc9706a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.543+0000 7f841fe00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:52:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.543+0000 7f841fe00640 1 -- 172.21.15.28:0/3852389012 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438027aa0 con 0x55df6b6a0a00 2024-04-26T01:52:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.543+0000 7f841fe00640 1 -- 172.21.15.28:0/3852389012 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f843801fb50 con 0x55df6b6a0a00 2024-04-26T01:52:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.543+0000 7f8447ff1740 1 -- 172.21.15.28:0/3852389012 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8430050780 con 0x55df6b6a0a00 2024-04-26T01:52:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.544+0000 7f841fe00640 1 -- 172.21.15.28:0/3852389012 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f843801fd50 con 0x55df6b6a0a00 2024-04-26T01:52:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.544+0000 7f8447ff1740 10 client.26194 fetch_fsmap learned FSMap version 42 2024-04-26T01:52:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.544+0000 7f8447ff1740 1 -- 172.21.15.28:0/3852389012 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55df6b716a90 con 0x55df6b6a0a00 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.545+0000 7f841fe00640 1 -- 172.21.15.28:0/3852389012 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8430050780 con 0x55df6b6a0a00 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.545+0000 7f841fe00640 1 -- 172.21.15.28:0/3852389012 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f8438038030 con 0x55df6b6a0a00 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.545+0000 7f8447ff1740 10 client.26194 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.545+0000 7f8447ff1740 10 client.26194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.545+0000 7f8447ff1740 20 client.26194 populate_metadata read hostname 'smithi028' 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.545+0000 7f8447ff1740 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55df6b717220 0x55df6b719640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.545+0000 7f8447ff1740 4 client.26194 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.545+0000 7f8447ff1740 1 -- 172.21.15.28:0/3852389012 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df6b609320 con 0x55df6b717220 2024-04-26T01:52:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.546+0000 7f843de00640 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55df6b717220 0x55df6b719640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.547+0000 7f843de00640 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55df6b717220 0x55df6b719640 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e88ae4fb4b566238 server_cookie=1ae8f6df7dac185c in_seq=0 out_seq=0 2024-04-26T01:52:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.547+0000 7f841fe00640 10 client.26194 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f841fe00640 1 -- 172.21.15.28:0/3852389012 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55df6b609320 con 0x55df6b717220 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f841fe00640 10 client.26194 handle_command_reply: tid=0 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 client.26194 shutdown 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 -- 172.21.15.28:0/3852389012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 msgr2=0x55df6b7153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b7153c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8438018040 tx=0x7f84380276a0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 -- 172.21.15.28:0/3852389012 shutdown_connections 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55df6b717220 0x55df6b719640 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55df6b6a0a00 0x55df6b7153c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55df6b696140 0x55df6b6a5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 --2- 172.21.15.28:0/3852389012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df6b695820 0x55df6b6a5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f8447ff1740 1 -- 172.21.15.28:0/3852389012 >> 172.21.15.28:0/3852389012 conn(0x55df6b691d30 msgr2=0x55df6b697220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.643+0000 7f841fe00640 0 client.26194 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.644+0000 7f8447ff1740 1 -- 172.21.15.28:0/3852389012 shutdown_connections 2024-04-26T01:52:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.644+0000 7f8447ff1740 1 -- 172.21.15.28:0/3852389012 wait complete. 2024-04-26T01:52:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.644+0000 7f8447ff1740 20 client.26194 trim_cache size 0 max 16384 2024-04-26T01:52:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:52:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:52:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:52:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:52:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.644+0000 7f8445200640 1 -- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f84240a3d50 msgr2=0x7f84240a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.644+0000 7f8445200640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f84240a3d50 0x7f84240a6210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8438009fd0 tx=0x7f8438009680 comp rx=0 tx=0).stop 2024-04-26T01:52:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.644+0000 7f8445200640 1 -- 172.21.15.28:0/1288202030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84401018f0 msgr2=0x7f844019f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.644+0000 7f8445200640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84401018f0 0x7f844019f2c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f842c009fd0 tx=0x7f842c00b870 comp rx=0 tx=0).stop 2024-04-26T01:52:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.645+0000 7f8445200640 1 -- 172.21.15.28:0/1288202030 shutdown_connections 2024-04-26T01:52:18.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.645+0000 7f8445200640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f84240a3d50 0x7f84240a6210 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.645+0000 7f8445200640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84401062f0 0x7f84401a3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.645+0000 7f8445200640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84401018f0 0x7f844019f2c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.645+0000 7f8445200640 1 --2- 172.21.15.28:0/1288202030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8440100fd0 0x7f844019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:18.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.645+0000 7f8445200640 1 -- 172.21.15.28:0/1288202030 >> 172.21.15.28:0/1288202030 conn(0x7f8440077fd0 msgr2=0x7f8440103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:18.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.645+0000 7f8445200640 1 -- 172.21.15.28:0/1288202030 shutdown_connections 2024-04-26T01:52:18.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:18.645+0000 7f8445200640 1 -- 172.21.15.28:0/1288202030 wait complete. 2024-04-26T01:52:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:19 smithi169 ceph-mon[32851]: pgmap v355: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 MiB/s wr, 1.86k op/s 2024-04-26T01:52:19.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:19 smithi053 ceph-mon[32593]: pgmap v355: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 MiB/s wr, 1.86k op/s 2024-04-26T01:52:19.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:19 smithi028 ceph-mon[28247]: pgmap v355: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 MiB/s wr, 1.86k op/s 2024-04-26T01:52:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 625.0 s, 1853.8 tps, lat 53.580 ms stddev 62.714 2024-04-26T01:52:21.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:21 smithi169 ceph-mon[32851]: pgmap v356: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 34 MiB/s wr, 1.85k op/s 2024-04-26T01:52:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:21 smithi053 ceph-mon[32593]: pgmap v356: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 34 MiB/s wr, 1.85k op/s 2024-04-26T01:52:21.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:21 smithi028 ceph-mon[28247]: pgmap v356: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 34 MiB/s wr, 1.85k op/s 2024-04-26T01:52:23.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:23 smithi169 ceph-mon[32851]: pgmap v357: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-26T01:52:23.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:23 smithi053 ceph-mon[32593]: pgmap v357: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-26T01:52:23.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:23 smithi028 ceph-mon[28247]: pgmap v357: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-26T01:52:25.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:25 smithi169 ceph-mon[32851]: pgmap v358: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 675 op/s 2024-04-26T01:52:25.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:25 smithi053 ceph-mon[32593]: pgmap v358: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 675 op/s 2024-04-26T01:52:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:25 smithi028 ceph-mon[28247]: pgmap v358: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 675 op/s 2024-04-26T01:52:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 630.0 s, 1944.4 tps, lat 50.999 ms stddev 65.769 2024-04-26T01:52:27.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:27 smithi169 ceph-mon[32851]: pgmap v359: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 763 op/s 2024-04-26T01:52:27.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:27 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:27.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:27 smithi053 ceph-mon[32593]: pgmap v359: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 763 op/s 2024-04-26T01:52:27.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:27 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:27.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:27 smithi028 ceph-mon[28247]: pgmap v359: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 763 op/s 2024-04-26T01:52:27.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:27 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:29.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:29 smithi053 ceph-mon[32593]: pgmap v360: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 578 op/s 2024-04-26T01:52:29.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:29 smithi028 ceph-mon[28247]: pgmap v360: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 578 op/s 2024-04-26T01:52:29.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:29 smithi169 ceph-mon[32851]: pgmap v360: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 578 op/s 2024-04-26T01:52:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 635.0 s, 619.7 tps, lat 74.497 ms stddev 128.413 2024-04-26T01:52:31.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:31 smithi053 ceph-mon[32593]: pgmap v361: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 30 MiB/s wr, 1.85k op/s 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 -- 172.21.15.28:0/3748768245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 msgr2=0x7fbab010ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3748768245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 0x7fbab010ee70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009be0 tx=0x7fbaa802a6c0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 -- 172.21.15.28:0/3748768245 shutdown_connections 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3748768245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 0x7fbab010ee70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3748768245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbab0102b80 0x7fbab0106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3748768245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab01021d0 0x7fbab01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 -- 172.21.15.28:0/3748768245 >> 172.21.15.28:0/3748768245 conn(0x7fbab00793e0 msgr2=0x7fbab00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 -- 172.21.15.28:0/3748768245 shutdown_connections 2024-04-26T01:52:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.780+0000 7fbab4e00640 1 -- 172.21.15.28:0/3748768245 wait complete. 2024-04-26T01:52:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.781+0000 7fbab4e00640 1 Processor -- start 2024-04-26T01:52:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.781+0000 7fbab4e00640 1 -- start start 2024-04-26T01:52:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbab4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab01021d0 0x7fbab01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbab4e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 0x7fbab01a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbab4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbab019a6a0 0x7fbab019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbab4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01121f0 con 0x7fbab01021d0 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbab4e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbab0112070 con 0x7fbab019a6a0 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbab4e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fbab0112370 con 0x7fbab01072a0 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbaade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab01021d0 0x7fbab01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbaade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab01021d0 0x7fbab01a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57064/0 (socket says 172.21.15.28:57064) 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbaade00640 1 -- 172.21.15.28:0/3334809494 learned_addr learned my addr 172.21.15.28:0/3334809494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbaa5400640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 0x7fbab01a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.782+0000 7fbaae800640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbab019a6a0 0x7fbab019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaade00640 1 -- 172.21.15.28:0/3334809494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 msgr2=0x7fbab01a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaade00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 0x7fbab01a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaade00640 1 -- 172.21.15.28:0/3334809494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbab019a6a0 msgr2=0x7fbab019ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:31.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaade00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbab019a6a0 0x7fbab019ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaade00640 1 -- 172.21.15.28:0/3334809494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab019b2a0 con 0x7fbab01021d0 2024-04-26T01:52:31.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaa5400640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 0x7fbab01a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:31.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaade00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab01021d0 0x7fbab01a05e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000e9c0 tx=0x7fbaa000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:31.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaa7400640 1 -- 172.21.15.28:0/3334809494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa000cde0 con 0x7fbab01021d0 2024-04-26T01:52:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbaa7400640 1 -- 172.21.15.28:0/3334809494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaa00035f0 con 0x7fbab01021d0 2024-04-26T01:52:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.783+0000 7fbab4e00640 1 -- 172.21.15.28:0/3334809494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab01a7380 con 0x7fbab01021d0 2024-04-26T01:52:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.784+0000 7fbaa7400640 1 -- 172.21.15.28:0/3334809494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0010690 con 0x7fbab01021d0 2024-04-26T01:52:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.784+0000 7fbab4e00640 1 -- 172.21.15.28:0/3334809494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab01a7780 con 0x7fbab01021d0 2024-04-26T01:52:31.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:31 smithi028 ceph-mon[28247]: pgmap v361: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 30 MiB/s wr, 1.85k op/s 2024-04-26T01:52:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.787+0000 7fbab7c7d740 1 Processor -- start 2024-04-26T01:52:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.787+0000 7fbab7c7d740 1 -- start start 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bf3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfe4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c00df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5598e7b59900 con 0x5598e7bf3820 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5598e7affef0 con 0x5598e7bf4140 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5598e7b558c0 con 0x5598e7bfea00 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbaae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bf3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbaae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bf3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57076/0 (socket says 172.21.15.28:57076) 2024-04-26T01:52:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbaae800640 1 -- 172.21.15.28:0/971920407 learned_addr learned my addr 172.21.15.28:0/971920407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbaa5400640 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c00df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.788+0000 7fbaa7400640 1 -- 172.21.15.28:0/3334809494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbaa0004000 con 0x7fbab01021d0 2024-04-26T01:52:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.789+0000 7fbaae800640 1 -- 172.21.15.28:0/971920407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 msgr2=0x5598e7c00df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.789+0000 7fbaae800640 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c00df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.789+0000 7fbaae800640 1 -- 172.21.15.28:0/971920407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 msgr2=0x5598e7bfe4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.789+0000 7fbaae800640 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfe4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.789+0000 7fbaae800640 1 -- 172.21.15.28:0/971920407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598e786c210 con 0x5598e7bf3820 2024-04-26T01:52:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.789+0000 7fbaa5400640 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c00df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.789+0000 7fbaae800640 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bf3c00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009be0 tx=0x7fbaa8032de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5855737242c857ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.789+0000 7fba9f400640 1 -- 172.21.15.28:0/971920407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8041030 con 0x5598e7bf3820 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.790+0000 7fba9f400640 1 -- 172.21.15.28:0/971920407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaa8003970 con 0x5598e7bf3820 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.790+0000 7fba9f400640 1 -- 172.21.15.28:0/971920407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8036040 con 0x5598e7bf3820 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.790+0000 7fbaa7400640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fba940a3dc0 0x7fba940a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.790+0000 7fbaa7400640 1 -- 172.21.15.28:0/3334809494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fbaa00d0b60 con 0x7fbab01021d0 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.790+0000 7fbaade00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fba940a3dc0 0x7fba940a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.790+0000 7fbab7c7d740 1 -- 172.21.15.28:0/971920407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 msgr2=0x5598e7bf3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.790+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bf3c00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009be0 tx=0x7fbaa8032de0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbab7c7d740 1 -- 172.21.15.28:0/971920407 shutdown_connections 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c00df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/971920407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bf3c00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbab7c7d740 1 -- 172.21.15.28:0/971920407 >> 172.21.15.28:0/971920407 conn(0x5598e7befd30 msgr2=0x5598e7bf0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbab7c7d740 1 -- 172.21.15.28:0/971920407 shutdown_connections 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbab7c7d740 1 -- 172.21.15.28:0/971920407 wait complete. 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbaade00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fba940a3dc0 0x7fba940a6280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5598e7bf4dc0 tx=0x7fbaa000c850 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.791+0000 7fbab7c7d740 1 -- start start 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.792+0000 7fbab7c7d740 1 Processor -- start 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.792+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bfb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.792+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7bfbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.792+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bfea00 0x5598e7bfc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.792+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5598e7c015d0 con 0x5598e7bf3820 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.792+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5598e7b13310 con 0x5598e7bfea00 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.792+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5598e7c01750 con 0x5598e7bf4140 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.792+0000 7fbab7c7d740 10 client.0 _ll_register_callbacks cb 0x5598e7ae1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bfb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaa5400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7bfbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bfb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57078/0 (socket says 172.21.15.28:57078) 2024-04-26T01:52:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaae800640 1 -- 172.21.15.28:0/3700349708 learned_addr learned my addr 172.21.15.28:0/3700349708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaae800640 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bfea00 0x5598e7bfc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaa5400640 1 -- 172.21.15.28:0/3700349708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bfea00 msgr2=0x5598e7bfc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaa5400640 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bfea00 0x5598e7bfc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaa5400640 1 -- 172.21.15.28:0/3700349708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 msgr2=0x5598e7bfb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaa5400640 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bfb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaa5400640 1 -- 172.21.15.28:0/3700349708 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598e7938620 con 0x5598e7bf4140 2024-04-26T01:52:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.793+0000 7fbaae800640 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bfb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.794+0000 7fbaa5400640 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7bfbe10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5598e7bf0ec0 tx=0x7fba98012e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f10f7fd89cea877c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.794+0000 7fba93e00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.794+0000 7fba93e00640 1 -- 172.21.15.28:0/3700349708 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba98004020 con 0x5598e7bf4140 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.794+0000 7fbab7c7d740 1 -- 172.21.15.28:0/3700349708 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaa8003970 con 0x5598e7bf4140 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.794+0000 7fba93e00640 1 -- 172.21.15.28:0/3700349708 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fba980041c0 con 0x5598e7bf4140 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.794+0000 7fba93e00640 1 -- 172.21.15.28:0/3700349708 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fba980043c0 con 0x5598e7bf4140 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.794+0000 7fbab7c7d740 10 client.26206 fetch_fsmap learned FSMap version 43 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.794+0000 7fbab7c7d740 1 -- 172.21.15.28:0/3700349708 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5598e7c0d880 con 0x5598e7bf4140 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fba93e00640 1 -- 172.21.15.28:0/3700349708 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaa8003970 con 0x5598e7bf4140 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fba93e00640 1 -- 172.21.15.28:0/3700349708 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7fba98003c40 con 0x5598e7bf4140 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fbab7c7d740 10 client.26206 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fbab7c7d740 10 client.26206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fbab7c7d740 20 client.26206 populate_metadata read hostname 'smithi028' 2024-04-26T01:52:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5598e7c0ecf0 0x5598e7c11100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fbab7c7d740 4 client.26206 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fbab7c7d740 1 -- 172.21.15.28:0/3700349708 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598e7b6d8b0 con 0x5598e7c0ecf0 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.795+0000 7fbaae800640 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5598e7c0ecf0 0x5598e7c11100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.796+0000 7fbaae800640 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5598e7c0ecf0 0x5598e7c11100 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52657669bcefe10 server_cookie=23cd1a6ecdd0d776 in_seq=0 out_seq=0 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.796+0000 7fba93e00640 10 client.26206 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fba93e00640 1 -- 172.21.15.28:0/3700349708 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598e7b6d8b0 con 0x5598e7c0ecf0 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fba93e00640 10 client.26206 handle_command_reply: tid=0 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 client.26206 shutdown 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 -- 172.21.15.28:0/3700349708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 msgr2=0x5598e7bfbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7bfbe10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5598e7bf0ec0 tx=0x7fba98012e70 comp rx=0 tx=0).stop 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 -- 172.21.15.28:0/3700349708 shutdown_connections 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5598e7c0ecf0 0x5598e7c11100 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bfea00 0x5598e7bfc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7bfbe10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/3700349708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7bfb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fba93e00640 0 client.26206 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 -- 172.21.15.28:0/3700349708 >> 172.21.15.28:0/3700349708 conn(0x5598e7befd30 msgr2=0x5598e7bf5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 -- 172.21.15.28:0/3700349708 shutdown_connections 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 1 -- 172.21.15.28:0/3700349708 wait complete. 2024-04-26T01:52:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.797+0000 7fbab7c7d740 20 client.26206 trim_cache size 0 max 16384 2024-04-26T01:52:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.818+0000 7fbab7c7d740 1 Processor -- start 2024-04-26T01:52:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.818+0000 7fbab7c7d740 1 -- start start 2024-04-26T01:52:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7c0a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c16a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01b2640 con 0x5598e7bf3820 2024-04-26T01:52:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01afe80 con 0x5598e7bf4140 2024-04-26T01:52:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01adcf0 con 0x5598e7bfea00 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbaae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7c0a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbaa5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbaa5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:42654/0 (socket says 172.21.15.28:42654) 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbaae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7c0a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57082/0 (socket says 172.21.15.28:57082) 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbaa5400640 1 -- 172.21.15.28:0/2797729988 learned_addr learned my addr 172.21.15.28:0/2797729988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.819+0000 7fbaae800640 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c16a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fbaa5400640 1 -- 172.21.15.28:0/2797729988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 msgr2=0x5598e7c16a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fbaa5400640 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c16a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fbaa5400640 1 -- 172.21.15.28:0/2797729988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 msgr2=0x5598e7c0a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fbaa5400640 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7c0a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fbaa5400640 1 -- 172.21.15.28:0/2797729988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598e7befb60 con 0x5598e7bf4140 2024-04-26T01:52:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fbaae800640 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7c0a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fbaa5400640 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfbfa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fba98000c00 tx=0x7fba98005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2fb3e23ed6f7940 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fba9ea00640 1 -- 172.21.15.28:0/2797729988 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba980256c0 con 0x5598e7bf4140 2024-04-26T01:52:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fba9ea00640 1 -- 172.21.15.28:0/2797729988 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba9802a070 con 0x5598e7bf4140 2024-04-26T01:52:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.820+0000 7fba9ea00640 1 -- 172.21.15.28:0/2797729988 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9801c6d0 con 0x5598e7bf4140 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 -- 172.21.15.28:0/2797729988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 msgr2=0x5598e7bfbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfbfa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fba98000c00 tx=0x7fba98005780 comp rx=0 tx=0).stop 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 -- 172.21.15.28:0/2797729988 shutdown_connections 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bfea00 0x5598e7c16a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf4140 0x5598e7bfbfa0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/2797729988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bf3820 0x5598e7c0a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 -- 172.21.15.28:0/2797729988 >> 172.21.15.28:0/2797729988 conn(0x5598e7befd30 msgr2=0x5598e7bfb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 -- 172.21.15.28:0/2797729988 shutdown_connections 2024-04-26T01:52:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.821+0000 7fbab7c7d740 1 -- 172.21.15.28:0/2797729988 wait complete. 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.822+0000 7fbab7c7d740 1 -- start start 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.822+0000 7fbab7c7d740 1 Processor -- start 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.822+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf3820 0x5598e7c06d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.822+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7c040f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.822+0000 7fbab7c7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bfea00 0x5598e7c04630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.822+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01ae340 con 0x5598e7bfea00 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.822+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01af7b0 con 0x5598e7bf3820 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.822+0000 7fbab7c7d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fbab01aea10 con 0x5598e7bf4140 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbab7c7d740 10 client.0 _ll_register_callbacks cb 0x7fba98012620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7c040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaae800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf3820 0x5598e7c06d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7c040f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:49134/0 (socket says 172.21.15.28:49134) 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 -- 172.21.15.28:0/868161006 learned_addr learned my addr 172.21.15.28:0/868161006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaae800640 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bfea00 0x5598e7c04630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 -- 172.21.15.28:0/868161006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf3820 msgr2=0x5598e7c06d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf3820 0x5598e7c06d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 -- 172.21.15.28:0/868161006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bfea00 msgr2=0x5598e7c04630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bfea00 0x5598e7c04630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 -- 172.21.15.28:0/868161006 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598e7c04e40 con 0x5598e7bf4140 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaae800640 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bfea00 0x5598e7c04630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fbaa5400640 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7c040f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fba980254d0 tx=0x7fba9800fdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4db7341d36d81e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.823+0000 7fba93e00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.824+0000 7fba93e00640 1 -- 172.21.15.28:0/868161006 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba98025e10 con 0x5598e7bf4140 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.824+0000 7fba93e00640 1 -- 172.21.15.28:0/868161006 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fba98005250 con 0x5598e7bf4140 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.824+0000 7fbab7c7d740 1 -- 172.21.15.28:0/868161006 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba9802a070 con 0x5598e7bf4140 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.824+0000 7fba93e00640 1 -- 172.21.15.28:0/868161006 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fba98005450 con 0x5598e7bf4140 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.824+0000 7fbab7c7d740 10 client.26212 fetch_fsmap learned FSMap version 43 2024-04-26T01:52:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.824+0000 7fbab7c7d740 1 -- 172.21.15.28:0/868161006 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5598e7c75270 con 0x5598e7bf4140 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.824+0000 7fba93e00640 1 -- 172.21.15.28:0/868161006 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba9802a070 con 0x5598e7bf4140 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fba93e00640 1 -- 172.21.15.28:0/868161006 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7fba98004ce0 con 0x5598e7bf4140 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fbab7c7d740 10 client.26212 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fbab7c7d740 10 client.26212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fbab7c7d740 20 client.26212 populate_metadata read hostname 'smithi028' 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5598e7c75820 0x5598e7c77c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fbab7c7d740 4 client.26212 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fbab7c7d740 1 -- 172.21.15.28:0/868161006 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5598e7b67320 con 0x5598e7c75820 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fbaae800640 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5598e7c75820 0x5598e7c77c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fbaae800640 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5598e7c75820 0x5598e7c77c40 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58c7edd3787f5b4d server_cookie=376005e9a4c92fbe in_seq=0 out_seq=0 2024-04-26T01:52:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:31.825+0000 7fba93e00640 10 client.26212 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:31.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:31 smithi169 ceph-mon[32851]: pgmap v361: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 30 MiB/s wr, 1.85k op/s 2024-04-26T01:52:32.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:32 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:32.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:32 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:32.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:32 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:33.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:33 smithi053 ceph-mon[32593]: pgmap v362: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 24 MiB/s wr, 1.56k op/s 2024-04-26T01:52:33.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:33 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:33.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:33 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:33.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:33 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:52:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:33 smithi028 ceph-mon[28247]: pgmap v362: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 24 MiB/s wr, 1.56k op/s 2024-04-26T01:52:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:33 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:33 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:33 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:52:33.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:33 smithi169 ceph-mon[32851]: pgmap v362: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 24 MiB/s wr, 1.56k op/s 2024-04-26T01:52:33.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:33 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:33.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:33 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:33.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:33 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:52:35.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:35 smithi053 ceph-mon[32593]: pgmap v363: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 25 MiB/s wr, 1.63k op/s 2024-04-26T01:52:35.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:35 smithi028 ceph-mon[28247]: pgmap v363: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 25 MiB/s wr, 1.63k op/s 2024-04-26T01:52:35.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:35 smithi169 ceph-mon[32851]: pgmap v363: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 25 MiB/s wr, 1.63k op/s 2024-04-26T01:52:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 640.0 s, 1540.8 tps, lat 100.257 ms stddev 455.534 2024-04-26T01:52:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.187+0000 7fba93e00640 1 -- 172.21.15.28:0/868161006 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5598e7b67320 con 0x5598e7c75820 2024-04-26T01:52:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.187+0000 7fba93e00640 10 client.26212 handle_command_reply: tid=0 2024-04-26T01:52:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.188+0000 7fbab7c7d740 1 client.26212 shutdown 2024-04-26T01:52:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.189+0000 7fbab7c7d740 1 -- 172.21.15.28:0/868161006 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 msgr2=0x5598e7c040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.189+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7c040f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fba980254d0 tx=0x7fba9800fdb0 comp rx=0 tx=0).stop 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.189+0000 7fbab7c7d740 1 -- 172.21.15.28:0/868161006 shutdown_connections 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.189+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5598e7c75820 0x5598e7c77c40 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.190+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598e7bfea00 0x5598e7c04630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.190+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5598e7bf4140 0x5598e7c040f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.190+0000 7fbab7c7d740 1 --2- 172.21.15.28:0/868161006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5598e7bf3820 0x5598e7c06d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.190+0000 7fbab7c7d740 1 -- 172.21.15.28:0/868161006 >> 172.21.15.28:0/868161006 conn(0x5598e7befd30 msgr2=0x5598e7bf5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.190+0000 7fba93e00640 0 client.26212 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.190+0000 7fbab7c7d740 1 -- 172.21.15.28:0/868161006 shutdown_connections 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.190+0000 7fbab7c7d740 1 -- 172.21.15.28:0/868161006 wait complete. 2024-04-26T01:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.190+0000 7fbab7c7d740 20 client.26212 trim_cache size 0 max 16384 2024-04-26T01:52:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:52:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:52:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:52:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 -- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fba940a3dc0 msgr2=0x7fba940a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fba940a3dc0 0x7fba940a6280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5598e7bf4dc0 tx=0x7fbaa000c850 comp rx=0 tx=0).stop 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 -- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab01021d0 msgr2=0x7fbab01a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab01021d0 0x7fbab01a05e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000e9c0 tx=0x7fbaa000ee80 comp rx=0 tx=0).stop 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 -- 172.21.15.28:0/3334809494 shutdown_connections 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fba940a3dc0 0x7fba940a6280 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbab019a6a0 0x7fbab019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbab01072a0 0x7fbab01a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 --2- 172.21.15.28:0/3334809494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab01021d0 0x7fbab01a05e0 secure :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000e9c0 tx=0x7fbaa000ee80 comp rx=0 tx=0).stop 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.191+0000 7fbab4e00640 1 -- 172.21.15.28:0/3334809494 >> 172.21.15.28:0/3334809494 conn(0x7fbab00793e0 msgr2=0x7fbab00ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.192+0000 7fbab4e00640 1 -- 172.21.15.28:0/3334809494 shutdown_connections 2024-04-26T01:52:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:37.192+0000 7fbab4e00640 1 -- 172.21.15.28:0/3334809494 wait complete. 2024-04-26T01:52:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:37 smithi028 ceph-mon[28247]: pgmap v364: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 24 MiB/s wr, 1.61k op/s 2024-04-26T01:52:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:37 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:37.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:37 smithi169 ceph-mon[32851]: pgmap v364: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 24 MiB/s wr, 1.61k op/s 2024-04-26T01:52:37.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:37 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:37 smithi053 ceph-mon[32593]: pgmap v364: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 24 MiB/s wr, 1.61k op/s 2024-04-26T01:52:37.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:37 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:39.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:39 smithi028 ceph-mon[28247]: pgmap v365: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 19 MiB/s wr, 1.44k op/s 2024-04-26T01:52:39.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:39 smithi169 ceph-mon[32851]: pgmap v365: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 19 MiB/s wr, 1.44k op/s 2024-04-26T01:52:39.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:39 smithi053 ceph-mon[32593]: pgmap v365: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 19 MiB/s wr, 1.44k op/s 2024-04-26T01:52:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 645.0 s, 2225.8 tps, lat 45.060 ms stddev 59.741 2024-04-26T01:52:41.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:41 smithi028 ceph-mon[28247]: pgmap v366: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 MiB/s wr, 2.14k op/s 2024-04-26T01:52:41.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:41 smithi169 ceph-mon[32851]: pgmap v366: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 MiB/s wr, 2.14k op/s 2024-04-26T01:52:41.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:41 smithi053 ceph-mon[32593]: pgmap v366: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 27 MiB/s wr, 2.14k op/s 2024-04-26T01:52:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:43 smithi028 ceph-mon[28247]: pgmap v367: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 871 op/s 2024-04-26T01:52:43.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:43 smithi169 ceph-mon[32851]: pgmap v367: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 871 op/s 2024-04-26T01:52:43.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:43 smithi053 ceph-mon[32593]: pgmap v367: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 871 op/s 2024-04-26T01:52:45.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:45 smithi028 ceph-mon[28247]: pgmap v368: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 13 MiB/s wr, 973 op/s 2024-04-26T01:52:45.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:45 smithi169 ceph-mon[32851]: pgmap v368: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 13 MiB/s wr, 973 op/s 2024-04-26T01:52:45.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:45 smithi053 ceph-mon[32593]: pgmap v368: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 13 MiB/s wr, 973 op/s 2024-04-26T01:52:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 650.0 s, 1750.3 tps, lat 53.591 ms stddev 62.596 2024-04-26T01:52:47.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:47 smithi028 ceph-mon[28247]: pgmap v369: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 14 MiB/s wr, 1.09k op/s 2024-04-26T01:52:47.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:47 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:47.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:47 smithi169 ceph-mon[32851]: pgmap v369: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 14 MiB/s wr, 1.09k op/s 2024-04-26T01:52:47.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:47 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:47.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:47 smithi053 ceph-mon[32593]: pgmap v369: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 14 MiB/s wr, 1.09k op/s 2024-04-26T01:52:47.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:47 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:49.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:49 smithi028 ceph-mon[28247]: pgmap v370: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 12 MiB/s wr, 991 op/s 2024-04-26T01:52:49.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:49 smithi169 ceph-mon[32851]: pgmap v370: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 12 MiB/s wr, 991 op/s 2024-04-26T01:52:49.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:49 smithi053 ceph-mon[32593]: pgmap v370: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 12 MiB/s wr, 991 op/s 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 -- 172.21.15.28:0/1982177110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c1043a0 msgr2=0x7f511c104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 --2- 172.21.15.28:0/1982177110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c1043a0 0x7f511c104780 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f510c009900 tx=0x7f510c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 -- 172.21.15.28:0/1982177110 shutdown_connections 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 --2- 172.21.15.28:0/1982177110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f511c109470 0x7f511c111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 --2- 172.21.15.28:0/1982177110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f511c104d50 0x7f511c108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 --2- 172.21.15.28:0/1982177110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c1043a0 0x7f511c104780 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 -- 172.21.15.28:0/1982177110 >> 172.21.15.28:0/1982177110 conn(0x7f511c0ffc40 msgr2=0x7f511c102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 -- 172.21.15.28:0/1982177110 shutdown_connections 2024-04-26T01:52:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.363+0000 7f5120c00640 1 -- 172.21.15.28:0/1982177110 wait complete. 2024-04-26T01:52:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.364+0000 7f5120c00640 1 Processor -- start 2024-04-26T01:52:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.364+0000 7f5120c00640 1 -- start start 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.364+0000 7f5120c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f511c1043a0 0x7f511c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.364+0000 7f5120c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c109470 0x7f511c1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.364+0000 7f5120c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f511c19a740 0x7f511c19abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5120c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f511c114420 con 0x7f511c109470 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5120c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1142a0 con 0x7f511c19a740 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5120c00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1145a0 con 0x7f511c1043a0 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5111400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c109470 0x7f511c1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5111400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c109470 0x7f511c1a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50052/0 (socket says 172.21.15.28:50052) 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5111400640 1 -- 172.21.15.28:0/629463108 learned_addr learned my addr 172.21.15.28:0/629463108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f511a800640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f511c19a740 0x7f511c19abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5119e00640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f511c1043a0 0x7f511c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5111400640 1 -- 172.21.15.28:0/629463108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f511c1043a0 msgr2=0x7f511c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5111400640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f511c1043a0 0x7f511c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5111400640 1 -- 172.21.15.28:0/629463108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f511c19a740 msgr2=0x7f511c19abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5111400640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f511c19a740 0x7f511c19abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.365+0000 7f5111400640 1 -- 172.21.15.28:0/629463108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f511c19b340 con 0x7f511c109470 2024-04-26T01:52:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.366+0000 7f5111400640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c109470 0x7f511c1a0b70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f510400e9f0 tx=0x7f510400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.366+0000 7f5113400640 1 -- 172.21.15.28:0/629463108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510400ccc0 con 0x7f511c109470 2024-04-26T01:52:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.366+0000 7f5113400640 1 -- 172.21.15.28:0/629463108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51040054d0 con 0x7f511c109470 2024-04-26T01:52:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.366+0000 7f5113400640 1 -- 172.21.15.28:0/629463108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104010430 con 0x7f511c109470 2024-04-26T01:52:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.366+0000 7f5120c00640 1 -- 172.21.15.28:0/629463108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f511c1a73d0 con 0x7f511c109470 2024-04-26T01:52:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.366+0000 7f5120c00640 1 -- 172.21.15.28:0/629463108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f511c1a7780 con 0x7f511c109470 2024-04-26T01:52:50.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.369+0000 7f5123a6c740 1 Processor -- start 2024-04-26T01:52:50.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.370+0000 7f5123a6c740 1 -- start start 2024-04-26T01:52:50.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.370+0000 7f5113400640 1 -- 172.21.15.28:0/629463108 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f51040046c0 con 0x7f511c109470 2024-04-26T01:52:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.370+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0a820 0x557a8ea0ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.370+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.370+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea15a00 0x557a8ea17df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.370+0000 7f5123a6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a8e970900 con 0x557a8ea15a00 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.370+0000 7f5123a6c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557a8e916ef0 con 0x557a8ea0a820 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.370+0000 7f5123a6c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557a8e96c8c0 con 0x557a8ea0b140 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5111400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea15a00 0x557a8ea17df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f511a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0a820 0x557a8ea0ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea154c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:57306/0 (socket says 172.21.15.28:57306) 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 -- 172.21.15.28:0/3163170370 learned_addr learned my addr 172.21.15.28:0/3163170370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 -- 172.21.15.28:0/3163170370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0a820 msgr2=0x557a8ea0ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 --2- 172.21.15.28:0/3163170370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0a820 0x557a8ea0ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 -- 172.21.15.28:0/3163170370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea15a00 msgr2=0x557a8ea17df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 --2- 172.21.15.28:0/3163170370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea15a00 0x557a8ea17df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 -- 172.21.15.28:0/3163170370 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a8e683210 con 0x557a8ea0b140 2024-04-26T01:52:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5119e00640 1 --2- 172.21.15.28:0/3163170370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea154c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f510c000c00 tx=0x7f510c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29c3e74dd1fe330a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.371+0000 7f5113400640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f51000a3dc0 0x7f51000a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.372+0000 7f510b000640 1 -- 172.21.15.28:0/3163170370 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c035040 con 0x557a8ea0b140 2024-04-26T01:52:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.372+0000 7f5113400640 1 -- 172.21.15.28:0/629463108 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f51040d0b90 con 0x7f511c109470 2024-04-26T01:52:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.372+0000 7f510b000640 1 -- 172.21.15.28:0/3163170370 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f510c0054f0 con 0x557a8ea0b140 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.372+0000 7f510b000640 1 -- 172.21.15.28:0/3163170370 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c0035f0 con 0x557a8ea0b140 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.372+0000 7f511a800640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f51000a3dc0 0x7f51000a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 -- 172.21.15.28:0/3163170370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 msgr2=0x557a8ea154c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 --2- 172.21.15.28:0/3163170370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea154c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f510c000c00 tx=0x7f510c004040 comp rx=0 tx=0).stop 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f511a800640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f51000a3dc0 0x7f51000a6280 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5114005fd0 tx=0x7f5114009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 -- 172.21.15.28:0/3163170370 shutdown_connections 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 --2- 172.21.15.28:0/3163170370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea15a00 0x557a8ea17df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 --2- 172.21.15.28:0/3163170370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea154c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 --2- 172.21.15.28:0/3163170370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0a820 0x557a8ea0ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 -- 172.21.15.28:0/3163170370 >> 172.21.15.28:0/3163170370 conn(0x557a8ea06d30 msgr2=0x557a8ea07110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 -- 172.21.15.28:0/3163170370 shutdown_connections 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 -- 172.21.15.28:0/3163170370 wait complete. 2024-04-26T01:52:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.373+0000 7f5123a6c740 1 -- start start 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5123a6c740 1 Processor -- start 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea129e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea12f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 0x557a8ea13460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5123a6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a8ea185d0 con 0x557a8ea0a820 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5123a6c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557a8e92a310 con 0x557a8ea0b140 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5123a6c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557a8ea18750 con 0x557a8ea15a00 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5123a6c740 10 client.0 _ll_register_callbacks cb 0x557a8e8f8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5119e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5111400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea12f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5119e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea129e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50064/0 (socket says 172.21.15.28:50064) 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f5119e00640 1 -- 172.21.15.28:0/2727248231 learned_addr learned my addr 172.21.15.28:0/2727248231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.374+0000 7f511a800640 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 0x557a8ea13460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f5119e00640 1 -- 172.21.15.28:0/2727248231 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 msgr2=0x557a8ea13460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f5119e00640 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 0x557a8ea13460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f5119e00640 1 -- 172.21.15.28:0/2727248231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 msgr2=0x557a8ea12f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f5119e00640 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea12f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f5119e00640 1 -- 172.21.15.28:0/2727248231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a8e74f620 con 0x557a8ea0a820 2024-04-26T01:52:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f5111400640 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea12f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f5119e00640 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea129e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f510c004980 tx=0x7f510c02a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38141d8900ee616c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f50ff400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:52:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.375+0000 7f50ff400640 1 -- 172.21.15.28:0/2727248231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c053060 con 0x557a8ea0a820 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f5123a6c740 1 -- 172.21.15.28:0/2727248231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f510c0054f0 con 0x557a8ea0a820 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f50ff400640 1 -- 172.21.15.28:0/2727248231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f510c004c70 con 0x557a8ea0a820 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f50ff400640 1 -- 172.21.15.28:0/2727248231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f510c035050 con 0x557a8ea0a820 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f5123a6c740 10 client.15963 fetch_fsmap learned FSMap version 45 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f5123a6c740 1 -- 172.21.15.28:0/2727248231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x557a8ea24880 con 0x557a8ea0a820 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f50ff400640 1 -- 172.21.15.28:0/2727248231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f510c0054f0 con 0x557a8ea0a820 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f50ff400640 1 -- 172.21.15.28:0/2727248231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7f510c0039e0 con 0x557a8ea0a820 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f5123a6c740 10 client.15963 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f5123a6c740 10 client.15963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.376+0000 7f5123a6c740 20 client.15963 populate_metadata read hostname 'smithi028' 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.377+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557a8ea25bd0 0x557a8ea27fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.377+0000 7f5123a6c740 4 client.15963 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.377+0000 7f5123a6c740 1 -- 172.21.15.28:0/2727248231 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a8e9848b0 con 0x557a8ea25bd0 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.377+0000 7f5111400640 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557a8ea25bd0 0x557a8ea27fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.377+0000 7f5111400640 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557a8ea25bd0 0x557a8ea27fc0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f353fcf650c5434 server_cookie=393362b1545c19c8 in_seq=0 out_seq=0 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.377+0000 7f50ff400640 10 client.15963 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.378+0000 7f50ff400640 1 -- 172.21.15.28:0/2727248231 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a8e9848b0 con 0x557a8ea25bd0 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.378+0000 7f50ff400640 10 client.15963 handle_command_reply: tid=0 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.378+0000 7f5123a6c740 1 client.15963 shutdown 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.378+0000 7f5123a6c740 1 -- 172.21.15.28:0/2727248231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 msgr2=0x557a8ea129e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.378+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea129e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f510c004980 tx=0x7f510c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 1 -- 172.21.15.28:0/2727248231 shutdown_connections 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557a8ea25bd0 0x557a8ea27fc0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 0x557a8ea13460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea12f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2727248231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea129e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 1 -- 172.21.15.28:0/2727248231 >> 172.21.15.28:0/2727248231 conn(0x557a8ea06d30 msgr2=0x557a8ea0ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f50ff400640 0 client.15963 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 1 -- 172.21.15.28:0/2727248231 shutdown_connections 2024-04-26T01:52:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 1 -- 172.21.15.28:0/2727248231 wait complete. 2024-04-26T01:52:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.379+0000 7f5123a6c740 20 client.15963 trim_cache size 0 max 16384 2024-04-26T01:52:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.397+0000 7f5123a6c740 1 Processor -- start 2024-04-26T01:52:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5123a6c740 1 -- start start 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea21490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea05d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea15a00 0x557a8ea2d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5123a6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1b2640 con 0x557a8ea0a820 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5123a6c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1afe80 con 0x557a8ea15a00 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5123a6c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1adcf0 con 0x557a8ea0b140 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5111400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea05d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5111400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea05d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:57332/0 (socket says 172.21.15.28:57332) 2024-04-26T01:52:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5111400640 1 -- 172.21.15.28:0/446456184 learned_addr learned my addr 172.21.15.28:0/446456184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f511a800640 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea15a00 0x557a8ea2d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.398+0000 7f5119e00640 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea21490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.399+0000 7f5111400640 1 -- 172.21.15.28:0/446456184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea15a00 msgr2=0x557a8ea2d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.399+0000 7f5111400640 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea15a00 0x557a8ea2d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.399+0000 7f5111400640 1 -- 172.21.15.28:0/446456184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 msgr2=0x557a8ea21490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.399+0000 7f5111400640 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea21490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.399+0000 7f5111400640 1 -- 172.21.15.28:0/446456184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a8ea0bab0 con 0x557a8ea0b140 2024-04-26T01:52:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.399+0000 7f5111400640 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea05d10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f510400c6a0 tx=0x7f51040bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36a2653164ba5026 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.399+0000 7f510a600640 1 -- 172.21.15.28:0/446456184 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51040d5040 con 0x557a8ea0b140 2024-04-26T01:52:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.400+0000 7f510a600640 1 -- 172.21.15.28:0/446456184 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5104005680 con 0x557a8ea0b140 2024-04-26T01:52:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.400+0000 7f510a600640 1 -- 172.21.15.28:0/446456184 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51040a0e80 con 0x557a8ea0b140 2024-04-26T01:52:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 -- 172.21.15.28:0/446456184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 msgr2=0x557a8ea05d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea05d10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f510400c6a0 tx=0x7f51040bd040 comp rx=0 tx=0).stop 2024-04-26T01:52:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 -- 172.21.15.28:0/446456184 shutdown_connections 2024-04-26T01:52:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea15a00 0x557a8ea2d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea0b140 0x557a8ea05d10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 --2- 172.21.15.28:0/446456184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea21490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 -- 172.21.15.28:0/446456184 >> 172.21.15.28:0/446456184 conn(0x557a8ea06d30 msgr2=0x557a8ea05600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 -- 172.21.15.28:0/446456184 shutdown_connections 2024-04-26T01:52:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 -- 172.21.15.28:0/446456184 wait complete. 2024-04-26T01:52:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.401+0000 7f5123a6c740 1 -- start start 2024-04-26T01:52:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.402+0000 7f5123a6c740 1 Processor -- start 2024-04-26T01:52:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.402+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea1dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.402+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea1b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.402+0000 7f5123a6c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 0x557a8ea1b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.402+0000 7f5123a6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1ae340 con 0x557a8ea0a820 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.402+0000 7f5123a6c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1af7b0 con 0x557a8ea0b140 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.402+0000 7f5123a6c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1aea10 con 0x557a8ea15a00 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5123a6c740 10 client.0 _ll_register_callbacks cb 0x7f510c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5119e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea1dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5111400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea1b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5119e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea1dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50080/0 (socket says 172.21.15.28:50080) 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5119e00640 1 -- 172.21.15.28:0/2531476338 learned_addr learned my addr 172.21.15.28:0/2531476338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f511a800640 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 0x557a8ea1b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5111400640 1 -- 172.21.15.28:0/2531476338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 msgr2=0x557a8ea1b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5111400640 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 0x557a8ea1b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5111400640 1 -- 172.21.15.28:0/2531476338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 msgr2=0x557a8ea1dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5111400640 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea1dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5111400640 1 -- 172.21.15.28:0/2531476338 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a8ea8ae10 con 0x557a8ea0b140 2024-04-26T01:52:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.403+0000 7f5119e00640 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea1dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f5111400640 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea1b250 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x557a8ea17ef0 tx=0x7f51040ce250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cf28d635c781eca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f50ff400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f50ff400640 1 -- 172.21.15.28:0/2531476338 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51040b5070 con 0x557a8ea0b140 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f5123a6c740 1 -- 172.21.15.28:0/2531476338 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5104005680 con 0x557a8ea0b140 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f50ff400640 1 -- 172.21.15.28:0/2531476338 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f510409ab70 con 0x557a8ea0b140 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f50ff400640 1 -- 172.21.15.28:0/2531476338 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f5104082050 con 0x557a8ea0b140 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f5123a6c740 10 client.26207 fetch_fsmap learned FSMap version 45 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f5123a6c740 1 -- 172.21.15.28:0/2531476338 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x557a8ea8bf80 con 0x557a8ea0b140 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.404+0000 7f50ff400640 1 -- 172.21.15.28:0/2531476338 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5104005680 con 0x557a8ea0b140 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.405+0000 7f50ff400640 1 -- 172.21.15.28:0/2531476338 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7f51040ce500 con 0x557a8ea0b140 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.405+0000 7f5123a6c740 10 client.26207 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.405+0000 7f5123a6c740 10 client.26207 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.405+0000 7f5123a6c740 20 client.26207 populate_metadata read hostname 'smithi028' 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.405+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557a8ea8c6d0 0x557a8ea8eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.405+0000 7f5123a6c740 4 client.26207 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.405+0000 7f5123a6c740 1 -- 172.21.15.28:0/2531476338 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a8e97e320 con 0x557a8ea8c6d0 2024-04-26T01:52:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.405+0000 7f5119e00640 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557a8ea8c6d0 0x557a8ea8eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.406+0000 7f5119e00640 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557a8ea8c6d0 0x557a8ea8eaf0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=661ff10fdd9b61d server_cookie=ebb0b6028e3ba81e in_seq=0 out_seq=0 2024-04-26T01:52:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.406+0000 7f50ff400640 10 client.26207 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f50ff400640 1 -- 172.21.15.28:0/2531476338 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a8e97e320 con 0x557a8ea8c6d0 2024-04-26T01:52:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f50ff400640 10 client.26207 handle_command_reply: tid=0 2024-04-26T01:52:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 client.26207 shutdown 2024-04-26T01:52:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 -- 172.21.15.28:0/2531476338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 msgr2=0x557a8ea1b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea1b250 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x557a8ea17ef0 tx=0x7f51040ce250 comp rx=0 tx=0).stop 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 -- 172.21.15.28:0/2531476338 shutdown_connections 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557a8ea8c6d0 0x557a8ea8eaf0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f50ff400640 0 client.26207 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557a8ea15a00 0x557a8ea1b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a8ea0b140 0x557a8ea1b250 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 --2- 172.21.15.28:0/2531476338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a8ea0a820 0x557a8ea1dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.432+0000 7f5123a6c740 1 -- 172.21.15.28:0/2531476338 >> 172.21.15.28:0/2531476338 conn(0x557a8ea06d30 msgr2=0x557a8ea13210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.433+0000 7f5123a6c740 1 -- 172.21.15.28:0/2531476338 shutdown_connections 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.433+0000 7f5123a6c740 1 -- 172.21.15.28:0/2531476338 wait complete. 2024-04-26T01:52:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.433+0000 7f5123a6c740 20 client.26207 trim_cache size 0 max 16384 2024-04-26T01:52:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:52:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:52:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:52:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:52:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.433+0000 7f5120c00640 1 -- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f51000a3dc0 msgr2=0x7f51000a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.433+0000 7f5120c00640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f51000a3dc0 0x7f51000a6280 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5114005fd0 tx=0x7f5114009500 comp rx=0 tx=0).stop 2024-04-26T01:52:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.433+0000 7f5120c00640 1 -- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c109470 msgr2=0x7f511c1a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.433+0000 7f5120c00640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c109470 0x7f511c1a0b70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f510400e9f0 tx=0x7f510400eeb0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.434+0000 7f5120c00640 1 -- 172.21.15.28:0/629463108 shutdown_connections 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.434+0000 7f5120c00640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f51000a3dc0 0x7f51000a6280 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.434+0000 7f5120c00640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f511c19a740 0x7f511c19abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.434+0000 7f5120c00640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511c109470 0x7f511c1a0b70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.434+0000 7f5120c00640 1 --2- 172.21.15.28:0/629463108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f511c1043a0 0x7f511c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.434+0000 7f5120c00640 1 -- 172.21.15.28:0/629463108 >> 172.21.15.28:0/629463108 conn(0x7f511c0ffc40 msgr2=0x7f511c101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.434+0000 7f5120c00640 1 -- 172.21.15.28:0/629463108 shutdown_connections 2024-04-26T01:52:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:52:50.434+0000 7f5120c00640 1 -- 172.21.15.28:0/629463108 wait complete. 2024-04-26T01:52:51.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 655.0 s, 131.6 tps, lat 810.827 ms stddev 1828.903 2024-04-26T01:52:51.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:51 smithi028 ceph-mon[28247]: pgmap v371: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-04-26T01:52:51.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:51 smithi169 ceph-mon[32851]: pgmap v371: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-04-26T01:52:51.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:51 smithi053 ceph-mon[32593]: pgmap v371: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-04-26T01:52:53.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:53 smithi028 ceph-mon[28247]: pgmap v372: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 5.8 MiB/s wr, 472 op/s 2024-04-26T01:52:53.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:53 smithi169 ceph-mon[32851]: pgmap v372: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 5.8 MiB/s wr, 472 op/s 2024-04-26T01:52:53.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:53 smithi053 ceph-mon[32593]: pgmap v372: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 5.8 MiB/s wr, 472 op/s 2024-04-26T01:52:55.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:55 smithi028 ceph-mon[28247]: pgmap v373: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-04-26T01:52:55.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:55 smithi169 ceph-mon[32851]: pgmap v373: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-04-26T01:52:55.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:55 smithi053 ceph-mon[32593]: pgmap v373: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-04-26T01:52:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 660.0 s, 2698.8 tps, lat 36.948 ms stddev 42.502 2024-04-26T01:52:57.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:57 smithi169 ceph-mon[32851]: pgmap v374: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.4 MiB/s wr, 592 op/s 2024-04-26T01:52:57.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:57 smithi053 ceph-mon[32593]: pgmap v374: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.4 MiB/s wr, 592 op/s 2024-04-26T01:52:58.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:57 smithi028 ceph-mon[28247]: pgmap v374: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.4 MiB/s wr, 592 op/s 2024-04-26T01:52:59.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:52:59 smithi169 ceph-mon[32851]: pgmap v375: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.9 MiB/s wr, 410 op/s 2024-04-26T01:52:59.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:52:59 smithi053 ceph-mon[32593]: pgmap v375: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.9 MiB/s wr, 410 op/s 2024-04-26T01:53:00.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:52:59 smithi028 ceph-mon[28247]: pgmap v375: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.9 MiB/s wr, 410 op/s 2024-04-26T01:53:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 665.0 s, 1984.6 tps, lat 47.700 ms stddev 69.606 2024-04-26T01:53:01.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:01 smithi169 ceph-mon[32851]: pgmap v376: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.9 MiB/s wr, 724 op/s 2024-04-26T01:53:01.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:01 smithi053 ceph-mon[32593]: pgmap v376: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.9 MiB/s wr, 724 op/s 2024-04-26T01:53:02.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:01 smithi028 ceph-mon[28247]: pgmap v376: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.9 MiB/s wr, 724 op/s 2024-04-26T01:53:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.602+0000 7f13a0400640 1 -- 172.21.15.28:0/3920969056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13981021e0 msgr2=0x7f13981025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.602+0000 7f13a0400640 1 --2- 172.21.15.28:0/3920969056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13981021e0 0x7f13981025c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f138c009900 tx=0x7f138c02a410 comp rx=0 tx=0).stop 2024-04-26T01:53:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.603+0000 7f13a0400640 1 -- 172.21.15.28:0/3920969056 shutdown_connections 2024-04-26T01:53:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.603+0000 7f13a0400640 1 --2- 172.21.15.28:0/3920969056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13981072b0 0x7f139810ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.603+0000 7f13a0400640 1 --2- 172.21.15.28:0/3920969056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102b90 0x7f1398106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.603+0000 7f13a0400640 1 --2- 172.21.15.28:0/3920969056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13981021e0 0x7f13981025c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.603+0000 7f13a0400640 1 -- 172.21.15.28:0/3920969056 >> 172.21.15.28:0/3920969056 conn(0x7f13980fda00 msgr2=0x7f13980ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.603+0000 7f13a0400640 1 -- 172.21.15.28:0/3920969056 shutdown_connections 2024-04-26T01:53:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.603+0000 7f13a0400640 1 -- 172.21.15.28:0/3920969056 wait complete. 2024-04-26T01:53:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.604+0000 7f13a0400640 1 Processor -- start 2024-04-26T01:53:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.604+0000 7f13a0400640 1 -- start start 2024-04-26T01:53:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.604+0000 7f13a0400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13981021e0 0x7f13981a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.604+0000 7f13a0400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13981072b0 0x7f13981a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.604+0000 7f13a0400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139819a800 0x7f139819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.604+0000 7f13a0400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1398112240 con 0x7f13981021e0 2024-04-26T01:53:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.604+0000 7f13a0400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f13981120c0 con 0x7f139819a800 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.604+0000 7f13a0400640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f13981123c0 con 0x7f13981072b0 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13981072b0 0x7f13981a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139e400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139819a800 0x7f139819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13981021e0 0x7f13981a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13981072b0 0x7f13981a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37944/0 (socket says 172.21.15.28:37944) 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139e400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139819a800 0x7f139819acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35226/0 (socket says 172.21.15.28:35226) 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 -- 172.21.15.28:0/1307429527 learned_addr learned my addr 172.21.15.28:0/1307429527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 -- 172.21.15.28:0/1307429527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139819a800 msgr2=0x7f139819acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139819a800 0x7f139819acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 -- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13981021e0 msgr2=0x7f13981a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13981021e0 0x7f13981a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 -- 172.21.15.28:0/1307429527 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139819b390 con 0x7f13981072b0 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139e400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139819a800 0x7f139819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.605+0000 7f139d000640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13981072b0 0x7f13981a0bd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1384005fd0 tx=0x7f138400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.606+0000 7f1396000640 1 -- 172.21.15.28:0/1307429527 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384017070 con 0x7f13981072b0 2024-04-26T01:53:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.606+0000 7f13a0400640 1 -- 172.21.15.28:0/1307429527 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139819b680 con 0x7f13981072b0 2024-04-26T01:53:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.606+0000 7f1396000640 1 -- 172.21.15.28:0/1307429527 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1384009c30 con 0x7f13981072b0 2024-04-26T01:53:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.606+0000 7f1396000640 1 -- 172.21.15.28:0/1307429527 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384003d30 con 0x7f13981072b0 2024-04-26T01:53:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.606+0000 7f13a0400640 1 -- 172.21.15.28:0/1307429527 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13981a78b0 con 0x7f13981072b0 2024-04-26T01:53:03.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.610+0000 7f13a2f22740 1 Processor -- start 2024-04-26T01:53:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.610+0000 7f13a2f22740 1 -- start start 2024-04-26T01:53:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 0x555b82225b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b82238cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b8223b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f13a2f22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555b8218b900 con 0x555b82225760 2024-04-26T01:53:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f13a2f22740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555b82131ef0 con 0x555b82226080 2024-04-26T01:53:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f13a2f22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x555b821878c0 con 0x555b82239210 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f139da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b82238cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f139da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b82238cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35242/0 (socket says 172.21.15.28:35242) 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.611+0000 7f139da00640 1 -- 172.21.15.28:0/574904395 learned_addr learned my addr 172.21.15.28:0/574904395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.612+0000 7f139e400640 1 --2- 172.21.15.28:0/574904395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 0x555b82225b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.612+0000 7f139da00640 1 -- 172.21.15.28:0/574904395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 msgr2=0x555b8223b600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.612+0000 7f139da00640 1 --2- 172.21.15.28:0/574904395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b8223b600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.612+0000 7f139da00640 1 -- 172.21.15.28:0/574904395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 msgr2=0x555b82225b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.612+0000 7f139da00640 1 --2- 172.21.15.28:0/574904395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 0x555b82225b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.612+0000 7f139da00640 1 -- 172.21.15.28:0/574904395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b81e9e210 con 0x555b82226080 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.612+0000 7f1396000640 1 -- 172.21.15.28:0/1307429527 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1384003ed0 con 0x7f13981072b0 2024-04-26T01:53:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.612+0000 7f139da00640 1 --2- 172.21.15.28:0/574904395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b82238cd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f138c000c00 tx=0x7f138c004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b60a8f640680d06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.613+0000 7f138aa00640 1 -- 172.21.15.28:0/574904395 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c035040 con 0x555b82226080 2024-04-26T01:53:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.613+0000 7f138aa00640 1 -- 172.21.15.28:0/574904395 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138c0054f0 con 0x555b82226080 2024-04-26T01:53:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.613+0000 7f138aa00640 1 -- 172.21.15.28:0/574904395 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c0035f0 con 0x555b82226080 2024-04-26T01:53:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f13a2f22740 1 -- 172.21.15.28:0/574904395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 msgr2=0x555b82238cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f13a2f22740 1 --2- 172.21.15.28:0/574904395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b82238cd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f138c000c00 tx=0x7f138c004170 comp rx=0 tx=0).stop 2024-04-26T01:53:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f13a2f22740 1 -- 172.21.15.28:0/574904395 shutdown_connections 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f13a2f22740 1 --2- 172.21.15.28:0/574904395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b8223b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f1396000640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f13800a3e40 0x7f13800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f13a2f22740 1 --2- 172.21.15.28:0/574904395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b82238cd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f13a2f22740 1 --2- 172.21.15.28:0/574904395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 0x555b82225b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f13a2f22740 1 -- 172.21.15.28:0/574904395 >> 172.21.15.28:0/574904395 conn(0x555b82221d30 msgr2=0x555b82222110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.614+0000 7f1396000640 1 -- 172.21.15.28:0/1307429527 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8557+0+0 (secure 0 0 0) 0x7f13840d0480 con 0x7f13981072b0 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.615+0000 7f139e400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f13800a3e40 0x7f13800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.615+0000 7f13a2f22740 1 -- 172.21.15.28:0/574904395 shutdown_connections 2024-04-26T01:53:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.615+0000 7f13a2f22740 1 -- 172.21.15.28:0/574904395 wait complete. 2024-04-26T01:53:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.615+0000 7f139e400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f13800a3e40 0x7f13800a6300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1390009fd0 tx=0x7f1390009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.616+0000 7f13a2f22740 1 -- start start 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.616+0000 7f13a2f22740 1 Processor -- start 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.616+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b822361a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.616+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82226080 0x555b822366e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.616+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b82236c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.616+0000 7f13a2f22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555b8223bd80 con 0x555b82226080 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.616+0000 7f13a2f22740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555b82145310 con 0x555b82225760 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.616+0000 7f13a2f22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x555b8223bf00 con 0x555b82239210 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f13a2f22740 10 client.0 _ll_register_callbacks cb 0x555b82113cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b822361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b822361a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35256/0 (socket says 172.21.15.28:35256) 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 -- 172.21.15.28:0/2088625712 learned_addr learned my addr 172.21.15.28:0/2088625712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139e400640 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b82236c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139d000640 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82226080 0x555b822366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 -- 172.21.15.28:0/2088625712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 msgr2=0x555b82236c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b82236c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 -- 172.21.15.28:0/2088625712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82226080 msgr2=0x555b822366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82226080 0x555b822366e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 -- 172.21.15.28:0/2088625712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b81f6a620 con 0x555b82225760 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139d000640 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82226080 0x555b822366e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f139da00640 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b822361a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f138c005c40 tx=0x7f138c009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58dc3064dc6b0df1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.617+0000 7f137f400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.618+0000 7f137f400640 1 -- 172.21.15.28:0/2088625712 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c053060 con 0x555b82225760 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.618+0000 7f137f400640 1 -- 172.21.15.28:0/2088625712 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8557+0+0 (secure 0 0 0) 0x7f138c004980 con 0x555b82225760 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.618+0000 7f13a2f22740 1 -- 172.21.15.28:0/2088625712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f138c0054f0 con 0x555b82225760 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.618+0000 7f137f400640 1 -- 172.21.15.28:0/2088625712 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f138c035050 con 0x555b82225760 2024-04-26T01:53:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.618+0000 7f13a2f22740 10 client.26219 fetch_fsmap learned FSMap version 45 2024-04-26T01:53:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.618+0000 7f13a2f22740 1 -- 172.21.15.28:0/2088625712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x555b8222b280 con 0x555b82225760 2024-04-26T01:53:03.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f137f400640 1 -- 172.21.15.28:0/2088625712 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138c0054f0 con 0x555b82225760 2024-04-26T01:53:03.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f137f400640 1 -- 172.21.15.28:0/2088625712 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7f138c003c80 con 0x555b82225760 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f13a2f22740 10 client.26219 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f13a2f22740 10 client.26219 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f13a2f22740 20 client.26219 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555b8222c740 0x555b8222ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f13a2f22740 4 client.26219 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f13a2f22740 1 -- 172.21.15.28:0/2088625712 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b8219f8b0 con 0x555b8222c740 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.619+0000 7f139d000640 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555b8222c740 0x555b8222ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.620+0000 7f139d000640 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555b8222c740 0x555b8222ec00 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5923a88703c201b server_cookie=b6f5944b8de2492f in_seq=0 out_seq=0 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.620+0000 7f137f400640 10 client.26219 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f137f400640 1 -- 172.21.15.28:0/2088625712 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b8219f8b0 con 0x555b8222c740 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f137f400640 10 client.26219 handle_command_reply: tid=0 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 client.26219 shutdown 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 -- 172.21.15.28:0/2088625712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 msgr2=0x555b822361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b822361a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f138c005c40 tx=0x7f138c009230 comp rx=0 tx=0).stop 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 -- 172.21.15.28:0/2088625712 shutdown_connections 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555b8222c740 0x555b8222ec00 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b82236c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82226080 0x555b822366e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2088625712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b822361a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f137f400640 0 client.26219 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 -- 172.21.15.28:0/2088625712 >> 172.21.15.28:0/2088625712 conn(0x555b82221d30 msgr2=0x555b82227ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 -- 172.21.15.28:0/2088625712 shutdown_connections 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.621+0000 7f13a2f22740 1 -- 172.21.15.28:0/2088625712 wait complete. 2024-04-26T01:53:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.622+0000 7f13a2f22740 20 client.26219 trim_cache size 0 max 16384 2024-04-26T01:53:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.640+0000 7f13a2f22740 1 Processor -- start 2024-04-26T01:53:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f13a2f22740 1 -- start start 2024-04-26T01:53:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b82244c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82226080 0x555b82220d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82239210 0x555b8225b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f13a2f22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13981ae310 con 0x555b82239210 2024-04-26T01:53:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f13a2f22740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f13981abb50 con 0x555b82225760 2024-04-26T01:53:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f13a2f22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f13981a9a40 con 0x555b82226080 2024-04-26T01:53:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f139e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82239210 0x555b8225b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f139d000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82226080 0x555b82220d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f139da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b82244c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f139d000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82226080 0x555b82220d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37954/0 (socket says 172.21.15.28:37954) 2024-04-26T01:53:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.641+0000 7f139d000640 1 -- 172.21.15.28:0/1179784645 learned_addr learned my addr 172.21.15.28:0/1179784645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.642+0000 7f139d000640 1 -- 172.21.15.28:0/1179784645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 msgr2=0x555b82244c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.642+0000 7f139d000640 1 --2- 172.21.15.28:0/1179784645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b82244c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.642+0000 7f139d000640 1 -- 172.21.15.28:0/1179784645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82239210 msgr2=0x555b8225b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.642+0000 7f139d000640 1 --2- 172.21.15.28:0/1179784645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82239210 0x555b8225b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.642+0000 7f139d000640 1 -- 172.21.15.28:0/1179784645 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b8223b9c0 con 0x555b82226080 2024-04-26T01:53:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.642+0000 7f139da00640 1 --2- 172.21.15.28:0/1179784645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b82244c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.642+0000 7f139d000640 1 --2- 172.21.15.28:0/1179784645 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82226080 0x555b82220d10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1384098800 tx=0x7f138400bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64adcee48b1e6b11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.643+0000 7f138a000640 1 -- 172.21.15.28:0/1179784645 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13840a5010 con 0x555b82226080 2024-04-26T01:53:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.643+0000 7f138a000640 1 -- 172.21.15.28:0/1179784645 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138409a440 con 0x555b82226080 2024-04-26T01:53:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.643+0000 7f138a000640 1 -- 172.21.15.28:0/1179784645 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13840041c0 con 0x555b82226080 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 -- 172.21.15.28:0/1179784645 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82226080 msgr2=0x555b82220d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 --2- 172.21.15.28:0/1179784645 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82226080 0x555b82220d10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1384098800 tx=0x7f138400bf60 comp rx=0 tx=0).stop 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 -- 172.21.15.28:0/1179784645 shutdown_connections 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 --2- 172.21.15.28:0/1179784645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82239210 0x555b8225b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 --2- 172.21.15.28:0/1179784645 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82226080 0x555b82220d10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 --2- 172.21.15.28:0/1179784645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82225760 0x555b82244c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 -- 172.21.15.28:0/1179784645 >> 172.21.15.28:0/1179784645 conn(0x555b82221d30 msgr2=0x555b82220600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 -- 172.21.15.28:0/1179784645 shutdown_connections 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 -- 172.21.15.28:0/1179784645 wait complete. 2024-04-26T01:53:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 -- start start 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.644+0000 7f13a2f22740 1 Processor -- start 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 0x555b8223efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b8223f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f13a2f22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b822b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f13a2f22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13981aa090 con 0x555b82225760 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f13a2f22740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f13981ab480 con 0x555b82226080 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f13a2f22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f13981aa6e0 con 0x555b82239210 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f13a2f22740 10 client.0 _ll_register_callbacks cb 0x7f138c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f139da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 0x555b8223efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f139e400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b822b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f139e400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b822b6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37966/0 (socket says 172.21.15.28:37966) 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.645+0000 7f139e400640 1 -- 172.21.15.28:0/2159359228 learned_addr learned my addr 172.21.15.28:0/2159359228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.646+0000 7f139d000640 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b8223f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.646+0000 7f139e400640 1 -- 172.21.15.28:0/2159359228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 msgr2=0x555b8223f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.646+0000 7f139e400640 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b8223f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.646+0000 7f139e400640 1 -- 172.21.15.28:0/2159359228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 msgr2=0x555b8223efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.646+0000 7f139e400640 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 0x555b8223efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.646+0000 7f139e400640 1 -- 172.21.15.28:0/2159359228 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b822b7020 con 0x555b82239210 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.646+0000 7f139e400640 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b822b6ae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1390003fd0 tx=0x7f1390004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c55232b699b96725 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.646+0000 7f137f400640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f137f400640 1 -- 172.21.15.28:0/2159359228 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139002f060 con 0x555b82239210 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f137f400640 1 -- 172.21.15.28:0/2159359228 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8557+0+0 (secure 0 0 0) 0x7f1390004740 con 0x555b82239210 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f13a2f22740 1 -- 172.21.15.28:0/2159359228 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f138409a440 con 0x555b82239210 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f137f400640 1 -- 172.21.15.28:0/2159359228 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f1390033430 con 0x555b82239210 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f13a2f22740 10 client.26245 fetch_fsmap learned FSMap version 45 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f13a2f22740 1 -- 172.21.15.28:0/2159359228 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x555b822b81d0 con 0x555b82239210 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f137f400640 1 -- 172.21.15.28:0/2159359228 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138409a440 con 0x555b82239210 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f137f400640 1 -- 172.21.15.28:0/2159359228 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7f139001e480 con 0x555b82239210 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f13a2f22740 10 client.26245 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f13a2f22740 10 client.26245 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.647+0000 7f13a2f22740 20 client.26245 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.648+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555b822b8950 0x555b822bad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.648+0000 7f13a2f22740 4 client.26245 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.648+0000 7f13a2f22740 1 -- 172.21.15.28:0/2159359228 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555b82199320 con 0x555b822b8950 2024-04-26T01:53:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.648+0000 7f139da00640 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555b822b8950 0x555b822bad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.648+0000 7f139da00640 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555b822b8950 0x555b822bad70 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f4c4bc776d75cb3 server_cookie=f4e35ac3212dd2b2 in_seq=0 out_seq=0 2024-04-26T01:53:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.648+0000 7f137f400640 10 client.26245 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f137f400640 1 -- 172.21.15.28:0/2159359228 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555b82199320 con 0x555b822b8950 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f137f400640 10 client.26245 handle_command_reply: tid=0 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 client.26245 shutdown 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 -- 172.21.15.28:0/2159359228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 msgr2=0x555b822b6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b822b6ae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1390003fd0 tx=0x7f1390004a40 comp rx=0 tx=0).stop 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 -- 172.21.15.28:0/2159359228 shutdown_connections 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x555b822b8950 0x555b822bad70 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555b82239210 0x555b822b6ae0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555b82226080 0x555b8223f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 --2- 172.21.15.28:0/2159359228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555b82225760 0x555b8223efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f137f400640 0 client.26245 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 -- 172.21.15.28:0/2159359228 >> 172.21.15.28:0/2159359228 conn(0x555b82221d30 msgr2=0x555b82220e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 -- 172.21.15.28:0/2159359228 shutdown_connections 2024-04-26T01:53:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.675+0000 7f13a2f22740 1 -- 172.21.15.28:0/2159359228 wait complete. 2024-04-26T01:53:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.676+0000 7f13a2f22740 20 client.26245 trim_cache size 0 max 16384 2024-04-26T01:53:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:53:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:53:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:53:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:53:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.676+0000 7f13a0400640 1 -- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f13800a3e40 msgr2=0x7f13800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.676+0000 7f13a0400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f13800a3e40 0x7f13800a6300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1390009fd0 tx=0x7f1390009500 comp rx=0 tx=0).stop 2024-04-26T01:53:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.676+0000 7f13a0400640 1 -- 172.21.15.28:0/1307429527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13981072b0 msgr2=0x7f13981a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.676+0000 7f13a0400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13981072b0 0x7f13981a0bd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1384005fd0 tx=0x7f138400dde0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.677+0000 7f13a0400640 1 -- 172.21.15.28:0/1307429527 shutdown_connections 2024-04-26T01:53:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.677+0000 7f13a0400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f13800a3e40 0x7f13800a6300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.677+0000 7f13a0400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139819a800 0x7f139819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.677+0000 7f13a0400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13981072b0 0x7f13981a0bd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.677+0000 7f13a0400640 1 --2- 172.21.15.28:0/1307429527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13981021e0 0x7f13981a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.677+0000 7f13a0400640 1 -- 172.21.15.28:0/1307429527 >> 172.21.15.28:0/1307429527 conn(0x7f13980fda00 msgr2=0x7f13980fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.677+0000 7f13a0400640 1 -- 172.21.15.28:0/1307429527 shutdown_connections 2024-04-26T01:53:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:03.677+0000 7f13a0400640 1 -- 172.21.15.28:0/1307429527 wait complete. 2024-04-26T01:53:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:03 smithi169 ceph-mon[32851]: pgmap v377: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.0 MiB/s wr, 535 op/s 2024-04-26T01:53:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:03 smithi169 ceph-mon[32851]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:53:03.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:03 smithi053 ceph-mon[32593]: pgmap v377: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.0 MiB/s wr, 535 op/s 2024-04-26T01:53:03.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:03 smithi053 ceph-mon[32593]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:53:04.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:03 smithi028 ceph-mon[28247]: pgmap v377: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.0 MiB/s wr, 535 op/s 2024-04-26T01:53:04.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:03 smithi028 ceph-mon[28247]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:53:05.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:05 smithi028 ceph-mon[28247]: pgmap v379: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.5 MiB/s wr, 607 op/s 2024-04-26T01:53:05.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:05 smithi169 ceph-mon[32851]: pgmap v379: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.5 MiB/s wr, 607 op/s 2024-04-26T01:53:05.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:05 smithi053 ceph-mon[32593]: pgmap v379: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.5 MiB/s wr, 607 op/s 2024-04-26T01:53:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 670.0 s, 248.4 tps, lat 401.506 ms stddev 245.943 2024-04-26T01:53:07.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:07 smithi169 ceph-mon[32851]: pgmap v380: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 576 op/s 2024-04-26T01:53:07.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:07 smithi053 ceph-mon[32593]: pgmap v380: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 576 op/s 2024-04-26T01:53:08.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:07 smithi028 ceph-mon[28247]: pgmap v380: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 576 op/s 2024-04-26T01:53:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:09 smithi169 ceph-mon[32851]: pgmap v381: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 576 op/s 2024-04-26T01:53:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:09 smithi169 ceph-mon[32851]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:53:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:09 smithi053 ceph-mon[32593]: pgmap v381: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 576 op/s 2024-04-26T01:53:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:09 smithi053 ceph-mon[32593]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:53:09.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:09 smithi028 ceph-mon[28247]: pgmap v381: 97 pgs: 97 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 576 op/s 2024-04-26T01:53:09.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:09 smithi028 ceph-mon[28247]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:53:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 675.0 s, 729.6 tps, lat 143.665 ms stddev 268.218 2024-04-26T01:53:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:11 smithi169 ceph-mon[32851]: pgmap v383: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 24 MiB/s wr, 2.02k op/s 2024-04-26T01:53:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:11 smithi053 ceph-mon[32593]: pgmap v383: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 24 MiB/s wr, 2.02k op/s 2024-04-26T01:53:11.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:11 smithi028 ceph-mon[28247]: pgmap v383: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 24 MiB/s wr, 2.02k op/s 2024-04-26T01:53:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:13 smithi169 ceph-mon[32851]: pgmap v384: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 20 MiB/s wr, 1.66k op/s 2024-04-26T01:53:13.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:13 smithi053 ceph-mon[32593]: pgmap v384: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 20 MiB/s wr, 1.66k op/s 2024-04-26T01:53:13.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:13 smithi028 ceph-mon[28247]: pgmap v384: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 20 MiB/s wr, 1.66k op/s 2024-04-26T01:53:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:15 smithi169 ceph-mon[32851]: pgmap v385: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 19 MiB/s wr, 1.63k op/s 2024-04-26T01:53:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:15 smithi053 ceph-mon[32593]: pgmap v385: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 19 MiB/s wr, 1.63k op/s 2024-04-26T01:53:15.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:15 smithi028 ceph-mon[28247]: pgmap v385: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 19 MiB/s wr, 1.63k op/s 2024-04-26T01:53:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 680.0 s, 1971.4 tps, lat 50.775 ms stddev 61.326 2024-04-26T01:53:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 -- 172.21.15.28:0/1355141270 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff050108f50 msgr2=0x7ff050109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 --2- 172.21.15.28:0/1355141270 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff050108f50 0x7ff050109330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff040009930 tx=0x7ff04002a4d0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 -- 172.21.15.28:0/1355141270 shutdown_connections 2024-04-26T01:53:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 --2- 172.21.15.28:0/1355141270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0501150a0 0x7ff050117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 --2- 172.21.15.28:0/1355141270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff050109870 0x7ff050114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 --2- 172.21.15.28:0/1355141270 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff050108f50 0x7ff050109330 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 -- 172.21.15.28:0/1355141270 >> 172.21.15.28:0/1355141270 conn(0x7ff05006d4b0 msgr2=0x7ff05006d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 -- 172.21.15.28:0/1355141270 shutdown_connections 2024-04-26T01:53:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.850+0000 7ff054c00640 1 -- 172.21.15.28:0/1355141270 wait complete. 2024-04-26T01:53:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.851+0000 7ff054c00640 1 Processor -- start 2024-04-26T01:53:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.851+0000 7ff054c00640 1 -- start start 2024-04-26T01:53:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff054c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050108f50 0x7ff05010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff054c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff050109870 0x7ff05010af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff054c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0501150a0 0x7ff05010b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff054c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff05011aa20 con 0x7ff050108f50 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff054c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff05011a8a0 con 0x7ff050109870 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff054c00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff05011aba0 con 0x7ff0501150a0 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff04d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff050109870 0x7ff05010af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff04de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050108f50 0x7ff05010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff04d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff050109870 0x7ff05010af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:44272/0 (socket says 172.21.15.28:44272) 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff04d400640 1 -- 172.21.15.28:0/1393500559 learned_addr learned my addr 172.21.15.28:0/1393500559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.852+0000 7ff04e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0501150a0 0x7ff05010b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.853+0000 7ff04d400640 1 -- 172.21.15.28:0/1393500559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0501150a0 msgr2=0x7ff05010b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.853+0000 7ff04d400640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0501150a0 0x7ff05010b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.853+0000 7ff04d400640 1 -- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050108f50 msgr2=0x7ff05010a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.853+0000 7ff04d400640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050108f50 0x7ff05010a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.853+0000 7ff04d400640 1 -- 172.21.15.28:0/1393500559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0501af3a0 con 0x7ff050109870 2024-04-26T01:53:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.853+0000 7ff04e800640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0501150a0 0x7ff05010b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.853+0000 7ff04d400640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff050109870 0x7ff05010af00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00d950 tx=0x7ff03c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.854+0000 7ff046a00640 1 -- 172.21.15.28:0/1393500559 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c017030 con 0x7ff050109870 2024-04-26T01:53:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.854+0000 7ff046a00640 1 -- 172.21.15.28:0/1393500559 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff03c004020 con 0x7ff050109870 2024-04-26T01:53:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.854+0000 7ff046a00640 1 -- 172.21.15.28:0/1393500559 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0047a0 con 0x7ff050109870 2024-04-26T01:53:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.854+0000 7ff054c00640 1 -- 172.21.15.28:0/1393500559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0501af690 con 0x7ff050109870 2024-04-26T01:53:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.854+0000 7ff054c00640 1 -- 172.21.15.28:0/1393500559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0501afb90 con 0x7ff050109870 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff0579f7740 1 Processor -- start 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff0579f7740 1 -- start start 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff046a00640 1 -- 172.21.15.28:0/1393500559 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff03c009a30 con 0x7ff050109870 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff0579f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e55a928900 con 0x55e55a9c3140 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff0579f7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e55a8ceef0 con 0x55e55a9cda00 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff0579f7740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55e55a9248c0 con 0x55e55a9c2820 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff04d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff04e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff04de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff04d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:44276/0 (socket says 172.21.15.28:44276) 2024-04-26T01:53:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.859+0000 7ff04d400640 1 -- 172.21.15.28:0/4108223451 learned_addr learned my addr 172.21.15.28:0/4108223451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff04d400640 1 -- 172.21.15.28:0/4108223451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 msgr2=0x55e55a9c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff04d400640 1 --2- 172.21.15.28:0/4108223451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9c2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff04d400640 1 -- 172.21.15.28:0/4108223451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 msgr2=0x55e55a9cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff04d400640 1 --2- 172.21.15.28:0/4108223451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff04d400640 1 -- 172.21.15.28:0/4108223451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e55a63b210 con 0x55e55a9cda00 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff04d400640 1 --2- 172.21.15.28:0/4108223451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cfdf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0cd6e0 tx=0x7ff03c0d8570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71d932268d73efa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff03b000640 1 -- 172.21.15.28:0/4108223451 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0e3070 con 0x55e55a9cda00 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff03b000640 1 -- 172.21.15.28:0/4108223451 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff03c0d22d0 con 0x55e55a9cda00 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.860+0000 7ff03b000640 1 -- 172.21.15.28:0/4108223451 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0de800 con 0x55e55a9cda00 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff046a00640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff0340a3f60 0x7ff0340a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff046a00640 1 -- 172.21.15.28:0/1393500559 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7ff03c0d00b0 con 0x7ff050109870 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff04e800640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff0340a3f60 0x7ff0340a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff0579f7740 1 -- 172.21.15.28:0/4108223451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 msgr2=0x55e55a9cfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff0579f7740 1 --2- 172.21.15.28:0/4108223451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cfdf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0cd6e0 tx=0x7ff03c0d8570 comp rx=0 tx=0).stop 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff0579f7740 1 -- 172.21.15.28:0/4108223451 shutdown_connections 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff0579f7740 1 --2- 172.21.15.28:0/4108223451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cfdf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff0579f7740 1 --2- 172.21.15.28:0/4108223451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff0579f7740 1 --2- 172.21.15.28:0/4108223451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9c2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.861+0000 7ff0579f7740 1 -- 172.21.15.28:0/4108223451 >> 172.21.15.28:0/4108223451 conn(0x55e55a9bed30 msgr2=0x55e55a9bf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.862+0000 7ff0579f7740 1 -- 172.21.15.28:0/4108223451 shutdown_connections 2024-04-26T01:53:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.862+0000 7ff0579f7740 1 -- 172.21.15.28:0/4108223451 wait complete. 2024-04-26T01:53:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.863+0000 7ff04e800640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff0340a3f60 0x7ff0340a6420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff048009fd0 tx=0x7ff048009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.863+0000 7ff0579f7740 1 -- start start 2024-04-26T01:53:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.864+0000 7ff0579f7740 1 Processor -- start 2024-04-26T01:53:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.864+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c2820 0x55e55a9ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.864+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9caee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.864+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.864+0000 7ff0579f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e55a9d0630 con 0x55e55a9c2820 2024-04-26T01:53:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.864+0000 7ff0579f7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e55a8e2310 con 0x55e55a9cda00 2024-04-26T01:53:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.864+0000 7ff0579f7740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55e55a9d07b0 con 0x55e55a9c3140 2024-04-26T01:53:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.864+0000 7ff0579f7740 10 client.0 _ll_register_callbacks cb 0x55e55a8b0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:44286/0 (socket says 172.21.15.28:44286) 2024-04-26T01:53:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04e800640 1 -- 172.21.15.28:0/2282466823 learned_addr learned my addr 172.21.15.28:0/2282466823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04de00640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c2820 0x55e55a9ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04d400640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04e800640 1 -- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 msgr2=0x55e55a9caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04e800640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04e800640 1 -- 172.21.15.28:0/2282466823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c2820 msgr2=0x55e55a9ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04e800640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c2820 0x55e55a9ca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04e800640 1 -- 172.21.15.28:0/2282466823 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e55a707620 con 0x55e55a9cda00 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04de00640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c2820 0x55e55a9ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.865+0000 7ff04d400640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff04e800640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cb420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff04800fec0 tx=0x7ff048004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=494d19b5a472a0c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff02f400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff02f400640 1 -- 172.21.15.28:0/2282466823 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04802e060 con 0x55e55a9cda00 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff0579f7740 1 -- 172.21.15.28:0/2282466823 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff03c0d22d0 con 0x55e55a9cda00 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff02f400640 1 -- 172.21.15.28:0/2282466823 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7ff048018030 con 0x55e55a9cda00 2024-04-26T01:53:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff02f400640 1 -- 172.21.15.28:0/2282466823 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff04801e960 con 0x55e55a9cda00 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff0579f7740 10 client.26237 fetch_fsmap learned FSMap version 45 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff0579f7740 1 -- 172.21.15.28:0/2282466823 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55e55a9e5170 con 0x55e55a9cda00 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.866+0000 7ff02f400640 1 -- 172.21.15.28:0/2282466823 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff03c0d22d0 con 0x55e55a9cda00 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.867+0000 7ff02f400640 1 -- 172.21.15.28:0/2282466823 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7ff0480313b0 con 0x55e55a9cda00 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.867+0000 7ff0579f7740 10 client.26237 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.867+0000 7ff0579f7740 10 client.26237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.867+0000 7ff0579f7740 20 client.26237 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.867+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e55a9e66a0 0x55e55a9e8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.867+0000 7ff0579f7740 4 client.26237 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.867+0000 7ff0579f7740 1 -- 172.21.15.28:0/2282466823 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e55a93c8b0 con 0x55e55a9e66a0 2024-04-26T01:53:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.867+0000 7ff04de00640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e55a9e66a0 0x55e55a9e8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.868+0000 7ff04de00640 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e55a9e66a0 0x55e55a9e8ad0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faecca3fb3f6525 server_cookie=641415c5b4f40a7d in_seq=0 out_seq=0 2024-04-26T01:53:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.868+0000 7ff02f400640 10 client.26237 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.869+0000 7ff02f400640 1 -- 172.21.15.28:0/2282466823 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e55a93c8b0 con 0x55e55a9e66a0 2024-04-26T01:53:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.869+0000 7ff02f400640 10 client.26237 handle_command_reply: tid=0 2024-04-26T01:53:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.869+0000 7ff0579f7740 1 client.26237 shutdown 2024-04-26T01:53:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.870+0000 7ff0579f7740 1 -- 172.21.15.28:0/2282466823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 msgr2=0x55e55a9cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.870+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cb420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff04800fec0 tx=0x7ff048004ab0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.871+0000 7ff0579f7740 1 -- 172.21.15.28:0/2282466823 shutdown_connections 2024-04-26T01:53:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.871+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e55a9e66a0 0x55e55a9e8ad0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.871+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9cb420 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.871+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.871+0000 7ff02f400640 0 client.26237 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:16.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.871+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2282466823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c2820 0x55e55a9ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.871+0000 7ff0579f7740 1 -- 172.21.15.28:0/2282466823 >> 172.21.15.28:0/2282466823 conn(0x55e55a9bed30 msgr2=0x55e55a9c4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:16.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.872+0000 7ff0579f7740 1 -- 172.21.15.28:0/2282466823 shutdown_connections 2024-04-26T01:53:16.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.872+0000 7ff0579f7740 1 -- 172.21.15.28:0/2282466823 wait complete. 2024-04-26T01:53:16.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.872+0000 7ff0579f7740 20 client.26237 trim_cache size 0 max 16384 2024-04-26T01:53:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.893+0000 7ff0579f7740 1 Processor -- start 2024-04-26T01:53:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.893+0000 7ff0579f7740 1 -- start start 2024-04-26T01:53:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9c2820 0x55e55a9e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9c5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9cda00 0x55e55a9bdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff0579f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501b6b80 con 0x55e55a9cda00 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff0579f7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501b43c0 con 0x55e55a9c2820 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff0579f7740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501b2230 con 0x55e55a9c3140 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff04de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9c2820 0x55e55a9e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff04de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9c2820 0x55e55a9e1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:44302/0 (socket says 172.21.15.28:44302) 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff04de00640 1 -- 172.21.15.28:0/3990688298 learned_addr learned my addr 172.21.15.28:0/3990688298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff04d400640 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff04e800640 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9cda00 0x55e55a9bdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff04de00640 1 -- 172.21.15.28:0/3990688298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 msgr2=0x55e55a9c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.894+0000 7ff04de00640 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.895+0000 7ff04de00640 1 -- 172.21.15.28:0/3990688298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9cda00 msgr2=0x55e55a9bdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.895+0000 7ff04de00640 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9cda00 0x55e55a9bdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.895+0000 7ff04de00640 1 -- 172.21.15.28:0/3990688298 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e55a9c8f30 con 0x55e55a9c2820 2024-04-26T01:53:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.895+0000 7ff04de00640 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9c2820 0x55e55a9e1d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff040006010 tx=0x7ff040005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=116cd5283a89aabc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.895+0000 7ff03a600640 1 -- 172.21.15.28:0/3990688298 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040035040 con 0x55e55a9c2820 2024-04-26T01:53:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.895+0000 7ff03a600640 1 -- 172.21.15.28:0/3990688298 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0400458f0 con 0x55e55a9c2820 2024-04-26T01:53:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.895+0000 7ff03a600640 1 -- 172.21.15.28:0/3990688298 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040005540 con 0x55e55a9c2820 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 -- 172.21.15.28:0/3990688298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9c2820 msgr2=0x55e55a9e1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9c2820 0x55e55a9e1d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff040006010 tx=0x7ff040005a20 comp rx=0 tx=0).stop 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 -- 172.21.15.28:0/3990688298 shutdown_connections 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9cda00 0x55e55a9bdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c3140 0x55e55a9c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 --2- 172.21.15.28:0/3990688298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9c2820 0x55e55a9e1d40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 -- 172.21.15.28:0/3990688298 >> 172.21.15.28:0/3990688298 conn(0x55e55a9bed30 msgr2=0x55e55a9bd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 -- 172.21.15.28:0/3990688298 shutdown_connections 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.896+0000 7ff0579f7740 1 -- 172.21.15.28:0/3990688298 wait complete. 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.897+0000 7ff0579f7740 1 -- start start 2024-04-26T01:53:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.897+0000 7ff0579f7740 1 Processor -- start 2024-04-26T01:53:16.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.897+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9dbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9dc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff0579f7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9dca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff0579f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501b2880 con 0x55e55a9c3140 2024-04-26T01:53:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff0579f7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501b3cf0 con 0x55e55a9cda00 2024-04-26T01:53:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff0579f7740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501b2f50 con 0x55e55a9c2820 2024-04-26T01:53:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff0579f7740 10 client.0 _ll_register_callbacks cb 0x7ff04800fd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff04e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9dca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff04de00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9dbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff04de00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9dbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:53080/0 (socket says 172.21.15.28:53080) 2024-04-26T01:53:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff04de00640 1 -- 172.21.15.28:0/2363420901 learned_addr learned my addr 172.21.15.28:0/2363420901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:16.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.898+0000 7ff04d400640 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9dc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.900+0000 7ff04d400640 1 -- 172.21.15.28:0/2363420901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 msgr2=0x55e55a9dbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:16.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.901+0000 7ff04d400640 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9dbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.901+0000 7ff04d400640 1 -- 172.21.15.28:0/2363420901 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 msgr2=0x55e55a9dca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:16.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.901+0000 7ff04d400640 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9dca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:16.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.901+0000 7ff04d400640 1 -- 172.21.15.28:0/2363420901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e55aa4bf50 con 0x55e55a9c3140 2024-04-26T01:53:16.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.902+0000 7ff04d400640 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9dc4b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x55e55a9ca200 tx=0x7ff03c0d0330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4ee1cd0c598af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:16.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.902+0000 7ff02f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:53:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.907+0000 7ff02f400640 1 -- 172.21.15.28:0/2363420901 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c098290 con 0x55e55a9c3140 2024-04-26T01:53:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.908+0000 7ff02f400640 1 -- 172.21.15.28:0/2363420901 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7ff03c009d20 con 0x55e55a9c3140 2024-04-26T01:53:16.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.909+0000 7ff0579f7740 1 -- 172.21.15.28:0/2363420901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0400458f0 con 0x55e55a9c3140 2024-04-26T01:53:16.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.911+0000 7ff02f400640 1 -- 172.21.15.28:0/2363420901 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff03c0d3c50 con 0x55e55a9c3140 2024-04-26T01:53:16.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.911+0000 7ff0579f7740 10 client.16002 fetch_fsmap learned FSMap version 45 2024-04-26T01:53:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.911+0000 7ff0579f7740 1 -- 172.21.15.28:0/2363420901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55e55aa4d260 con 0x55e55a9c3140 2024-04-26T01:53:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.912+0000 7ff02f400640 1 -- 172.21.15.28:0/2363420901 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0400458f0 con 0x55e55a9c3140 2024-04-26T01:53:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.912+0000 7ff02f400640 1 -- 172.21.15.28:0/2363420901 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7ff03c098660 con 0x55e55a9c3140 2024-04-26T01:53:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.912+0000 7ff0579f7740 10 client.16002 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:53:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.912+0000 7ff0579f7740 10 client.16002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.912+0000 7ff0579f7740 20 client.16002 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.912+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e55aa4d590 0x55e55aa4f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.912+0000 7ff0579f7740 4 client.16002 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.912+0000 7ff0579f7740 1 -- 172.21.15.28:0/2363420901 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e55a936320 con 0x55e55aa4d590 2024-04-26T01:53:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.916+0000 7ff04de00640 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e55aa4d590 0x55e55aa4f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.922+0000 7ff04de00640 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e55aa4d590 0x55e55aa4f980 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89522f8740162a6c server_cookie=313f82b8bc412a4b in_seq=0 out_seq=0 2024-04-26T01:53:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:16.922+0000 7ff02f400640 10 client.16002 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.038+0000 7ff02f400640 1 -- 172.21.15.28:0/2363420901 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e55a936320 con 0x55e55aa4d590 2024-04-26T01:53:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.038+0000 7ff02f400640 10 client.16002 handle_command_reply: tid=0 2024-04-26T01:53:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.038+0000 7ff0579f7740 1 client.16002 shutdown 2024-04-26T01:53:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff0579f7740 1 -- 172.21.15.28:0/2363420901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 msgr2=0x55e55a9dc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9dc4b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x55e55a9ca200 tx=0x7ff03c0d0330 comp rx=0 tx=0).stop 2024-04-26T01:53:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff0579f7740 1 -- 172.21.15.28:0/2363420901 shutdown_connections 2024-04-26T01:53:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e55aa4d590 0x55e55aa4f980 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e55a9cda00 0x55e55a9dca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a9c3140 0x55e55a9dc4b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff0579f7740 1 --2- 172.21.15.28:0/2363420901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e55a9c2820 0x55e55a9dbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff0579f7740 1 -- 172.21.15.28:0/2363420901 >> 172.21.15.28:0/2363420901 conn(0x55e55a9bed30 msgr2=0x55e55a9cb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.039+0000 7ff02f400640 0 client.16002 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.040+0000 7ff0579f7740 1 -- 172.21.15.28:0/2363420901 shutdown_connections 2024-04-26T01:53:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.040+0000 7ff0579f7740 1 -- 172.21.15.28:0/2363420901 wait complete. 2024-04-26T01:53:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.040+0000 7ff0579f7740 20 client.16002 trim_cache size 0 max 16384 2024-04-26T01:53:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:53:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:53:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:53:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:53:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.041+0000 7ff054c00640 1 -- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff0340a3f60 msgr2=0x7ff0340a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.041+0000 7ff054c00640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff0340a3f60 0x7ff0340a6420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff048009fd0 tx=0x7ff048009500 comp rx=0 tx=0).stop 2024-04-26T01:53:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.041+0000 7ff054c00640 1 -- 172.21.15.28:0/1393500559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff050109870 msgr2=0x7ff05010af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.041+0000 7ff054c00640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff050109870 0x7ff05010af00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00d950 tx=0x7ff03c00de10 comp rx=0 tx=0).stop 2024-04-26T01:53:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.042+0000 7ff054c00640 1 -- 172.21.15.28:0/1393500559 shutdown_connections 2024-04-26T01:53:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.042+0000 7ff054c00640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7ff0340a3f60 0x7ff0340a6420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.042+0000 7ff054c00640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0501150a0 0x7ff05010b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.042+0000 7ff054c00640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff050109870 0x7ff05010af00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.042+0000 7ff054c00640 1 --2- 172.21.15.28:0/1393500559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050108f50 0x7ff05010a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.042+0000 7ff054c00640 1 -- 172.21.15.28:0/1393500559 >> 172.21.15.28:0/1393500559 conn(0x7ff05006d4b0 msgr2=0x7ff050070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.042+0000 7ff054c00640 1 -- 172.21.15.28:0/1393500559 shutdown_connections 2024-04-26T01:53:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:17.042+0000 7ff054c00640 1 -- 172.21.15.28:0/1393500559 wait complete. 2024-04-26T01:53:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:17 smithi169 ceph-mon[32851]: pgmap v386: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 1.71k op/s 2024-04-26T01:53:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:17 smithi053 ceph-mon[32593]: pgmap v386: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 1.71k op/s 2024-04-26T01:53:17.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:17 smithi028 ceph-mon[28247]: pgmap v386: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 1.71k op/s 2024-04-26T01:53:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:19 smithi169 ceph-mon[32851]: pgmap v387: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 1.71k op/s 2024-04-26T01:53:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:19 smithi053 ceph-mon[32593]: pgmap v387: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 1.71k op/s 2024-04-26T01:53:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:19 smithi028 ceph-mon[28247]: pgmap v387: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 1.71k op/s 2024-04-26T01:53:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 685.0 s, 2152.8 tps, lat 46.752 ms stddev 56.502 2024-04-26T01:53:21.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:21 smithi169 ceph-mon[32851]: pgmap v388: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 MiB/s wr, 1.73k op/s 2024-04-26T01:53:21.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:21 smithi053 ceph-mon[32593]: pgmap v388: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 MiB/s wr, 1.73k op/s 2024-04-26T01:53:21.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:21 smithi028 ceph-mon[28247]: pgmap v388: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 MiB/s wr, 1.73k op/s 2024-04-26T01:53:23.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:23 smithi169 ceph-mon[32851]: pgmap v389: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.8 MiB/s wr, 539 op/s 2024-04-26T01:53:23.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:23 smithi053 ceph-mon[32593]: pgmap v389: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.8 MiB/s wr, 539 op/s 2024-04-26T01:53:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:23 smithi028 ceph-mon[28247]: pgmap v389: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.8 MiB/s wr, 539 op/s 2024-04-26T01:53:25.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:25 smithi169 ceph-mon[32851]: pgmap v390: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.7 MiB/s wr, 636 op/s 2024-04-26T01:53:25.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:25 smithi053 ceph-mon[32593]: pgmap v390: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.7 MiB/s wr, 636 op/s 2024-04-26T01:53:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:25 smithi028 ceph-mon[28247]: pgmap v390: 97 pgs: 97 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.7 MiB/s wr, 636 op/s 2024-04-26T01:53:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 690.0 s, 2088.6 tps, lat 47.372 ms stddev 55.004 2024-04-26T01:53:27.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:27 smithi169 ceph-mon[32851]: pgmap v391: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.4 MiB/s wr, 744 op/s 2024-04-26T01:53:27.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:27 smithi053 ceph-mon[32593]: pgmap v391: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.4 MiB/s wr, 744 op/s 2024-04-26T01:53:27.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:27 smithi028 ceph-mon[28247]: pgmap v391: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.4 MiB/s wr, 744 op/s 2024-04-26T01:53:29.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:29 smithi053 ceph-mon[32593]: pgmap v392: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 583 op/s 2024-04-26T01:53:29.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:29 smithi028 ceph-mon[28247]: pgmap v392: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 583 op/s 2024-04-26T01:53:29.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:29 smithi169 ceph-mon[32851]: pgmap v392: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 583 op/s 2024-04-26T01:53:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.217+0000 7fc789200640 1 -- 172.21.15.28:0/1917390367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7841018f0 msgr2=0x7fc784105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.217+0000 7fc789200640 1 --2- 172.21.15.28:0/1917390367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7841018f0 0x7fc784105db0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009860 tx=0x7fc76c02a3d0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.218+0000 7fc789200640 1 -- 172.21.15.28:0/1917390367 shutdown_connections 2024-04-26T01:53:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.218+0000 7fc789200640 1 --2- 172.21.15.28:0/1917390367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7841063d0 0x7fc784106880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.218+0000 7fc789200640 1 --2- 172.21.15.28:0/1917390367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7841018f0 0x7fc784105db0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.218+0000 7fc789200640 1 --2- 172.21.15.28:0/1917390367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc784100fd0 0x7fc7841013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.218+0000 7fc789200640 1 -- 172.21.15.28:0/1917390367 >> 172.21.15.28:0/1917390367 conn(0x7fc784077f60 msgr2=0x7fc784078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.218+0000 7fc789200640 1 -- 172.21.15.28:0/1917390367 shutdown_connections 2024-04-26T01:53:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.219+0000 7fc789200640 1 -- 172.21.15.28:0/1917390367 wait complete. 2024-04-26T01:53:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.220+0000 7fc789200640 1 Processor -- start 2024-04-26T01:53:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.220+0000 7fc789200640 1 -- start start 2024-04-26T01:53:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.220+0000 7fc789200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc784100fd0 0x7fc7841a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.220+0000 7fc789200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7841018f0 0x7fc7841a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.220+0000 7fc789200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7841063d0 0x7fc78419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc789200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841187d0 con 0x7fc7841063d0 2024-04-26T01:53:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc789200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc784118650 con 0x7fc784100fd0 2024-04-26T01:53:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc789200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc784118950 con 0x7fc7841018f0 2024-04-26T01:53:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7841018f0 0x7fc7841a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc784100fd0 0x7fc7841a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7841018f0 0x7fc7841a4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:41536/0 (socket says 172.21.15.28:41536) 2024-04-26T01:53:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781400640 1 -- 172.21.15.28:0/3263052798 learned_addr learned my addr 172.21.15.28:0/3263052798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc782800640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7841063d0 0x7fc78419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781400640 1 -- 172.21.15.28:0/3263052798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc784100fd0 msgr2=0x7fc7841a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781400640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc784100fd0 0x7fc7841a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781400640 1 -- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7841063d0 msgr2=0x7fc78419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781400640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7841063d0 0x7fc78419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781400640 1 -- 172.21.15.28:0/3263052798 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78419f430 con 0x7fc7841018f0 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.221+0000 7fc781e00640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc784100fd0 0x7fc7841a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.222+0000 7fc781400640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7841018f0 0x7fc7841a4fa0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009980 tx=0x7fc76c032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.222+0000 7fc77aa00640 1 -- 172.21.15.28:0/3263052798 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c040070 con 0x7fc7841018f0 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.222+0000 7fc77aa00640 1 -- 172.21.15.28:0/3263052798 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc76c0318b0 con 0x7fc7841018f0 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.222+0000 7fc77aa00640 1 -- 172.21.15.28:0/3263052798 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c004ad0 con 0x7fc7841018f0 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.222+0000 7fc789200640 1 -- 172.21.15.28:0/3263052798 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc78419f6c0 con 0x7fc7841018f0 2024-04-26T01:53:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.222+0000 7fc789200640 1 -- 172.21.15.28:0/3263052798 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7841ab800 con 0x7fc7841018f0 2024-04-26T01:53:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.239+0000 7fc77aa00640 1 -- 172.21.15.28:0/3263052798 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc76c039040 con 0x7fc7841018f0 2024-04-26T01:53:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.241+0000 7fc77aa00640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc7680a3e40 0x7fc7680a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.241+0000 7fc77aa00640 1 -- 172.21.15.28:0/3263052798 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7fc76c0f1320 con 0x7fc7841018f0 2024-04-26T01:53:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.241+0000 7fc78c03c740 1 Processor -- start 2024-04-26T01:53:30.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.244+0000 7fc78c03c740 1 -- start start 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.244+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 0x56176ad73c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.244+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad74140 0x56176ad7e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.244+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad80df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.244+0000 7fc78c03c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56176acd9900 con 0x56176ad73820 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.244+0000 7fc78c03c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56176ac7fef0 con 0x56176ad7ea00 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.244+0000 7fc78c03c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x56176acd58c0 con 0x56176ad74140 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.245+0000 7fc782800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad80df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.245+0000 7fc782800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad80df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35138/0 (socket says 172.21.15.28:35138) 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.245+0000 7fc781e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 0x56176ad73c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.245+0000 7fc782800640 1 -- 172.21.15.28:0/1975707200 learned_addr learned my addr 172.21.15.28:0/1975707200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.246+0000 7fc782800640 1 -- 172.21.15.28:0/1975707200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad74140 msgr2=0x56176ad7e4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:53:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.246+0000 7fc782800640 1 --2- 172.21.15.28:0/1975707200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad74140 0x56176ad7e4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.246+0000 7fc782800640 1 -- 172.21.15.28:0/1975707200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 msgr2=0x56176ad73c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.246+0000 7fc782800640 1 --2- 172.21.15.28:0/1975707200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 0x56176ad73c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.246+0000 7fc782800640 1 -- 172.21.15.28:0/1975707200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56176a9ec210 con 0x56176ad7ea00 2024-04-26T01:53:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.246+0000 7fc781e00640 1 --2- 172.21.15.28:0/1975707200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 0x56176ad73c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.247+0000 7fc781e00640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc7680a3e40 0x7fc7680a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.247+0000 7fc782800640 1 --2- 172.21.15.28:0/1975707200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad80df0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc77c0099f0 tx=0x7fc77c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17b5a071cb9b10ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.247+0000 7fc773400640 1 -- 172.21.15.28:0/1975707200 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c01d070 con 0x56176ad7ea00 2024-04-26T01:53:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.248+0000 7fc781e00640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc7680a3e40 0x7fc7680a6300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc774009710 tx=0x7fc774004730 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.248+0000 7fc773400640 1 -- 172.21.15.28:0/1975707200 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c003e40 con 0x56176ad7ea00 2024-04-26T01:53:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.248+0000 7fc773400640 1 -- 172.21.15.28:0/1975707200 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c00d9d0 con 0x56176ad7ea00 2024-04-26T01:53:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.249+0000 7fc78c03c740 1 -- 172.21.15.28:0/1975707200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 msgr2=0x56176ad80df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.249+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1975707200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad80df0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc77c0099f0 tx=0x7fc77c0095c0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.250+0000 7fc78c03c740 1 -- 172.21.15.28:0/1975707200 shutdown_connections 2024-04-26T01:53:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.250+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1975707200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad80df0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.250+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1975707200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad74140 0x56176ad7e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.250+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1975707200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 0x56176ad73c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.250+0000 7fc78c03c740 1 -- 172.21.15.28:0/1975707200 >> 172.21.15.28:0/1975707200 conn(0x56176ad6fd30 msgr2=0x56176ad70110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:30.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.250+0000 7fc78c03c740 1 -- 172.21.15.28:0/1975707200 shutdown_connections 2024-04-26T01:53:30.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.250+0000 7fc78c03c740 1 -- 172.21.15.28:0/1975707200 wait complete. 2024-04-26T01:53:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.251+0000 7fc78c03c740 1 -- start start 2024-04-26T01:53:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.251+0000 7fc78c03c740 1 Processor -- start 2024-04-26T01:53:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.252+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad7ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.252+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad7bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.252+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad7ea00 0x56176ad7c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.252+0000 7fc78c03c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56176ad92640 con 0x56176ad7ea00 2024-04-26T01:53:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.252+0000 7fc78c03c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56176ac93310 con 0x56176ad74140 2024-04-26T01:53:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.252+0000 7fc78c03c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x56176ad927c0 con 0x56176ad73820 2024-04-26T01:53:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.253+0000 7fc78c03c740 10 client.0 _ll_register_callbacks cb 0x56176ac61cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.253+0000 7fc782800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad7ea00 0x56176ad7c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.254+0000 7fc781e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad7ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.254+0000 7fc782800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad7ea00 0x56176ad7c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40248/0 (socket says 172.21.15.28:40248) 2024-04-26T01:53:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.254+0000 7fc782800640 1 -- 172.21.15.28:0/2474397203 learned_addr learned my addr 172.21.15.28:0/2474397203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.254+0000 7fc781400640 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad7bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.255+0000 7fc781e00640 1 -- 172.21.15.28:0/2474397203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 msgr2=0x56176ad7bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:30.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.255+0000 7fc781e00640 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad7bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.255+0000 7fc781e00640 1 -- 172.21.15.28:0/2474397203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad7ea00 msgr2=0x56176ad7c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:30.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.255+0000 7fc781e00640 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad7ea00 0x56176ad7c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.255+0000 7fc781e00640 1 -- 172.21.15.28:0/2474397203 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56176aab8620 con 0x56176ad73820 2024-04-26T01:53:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.255+0000 7fc781400640 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad7bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.256+0000 7fc781e00640 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad7ba20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc77400f790 tx=0x7fc774005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed13812b82c05e64 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.256+0000 7fc767e00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:53:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.259+0000 7fc767e00640 1 -- 172.21.15.28:0/2474397203 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77402b060 con 0x56176ad73820 2024-04-26T01:53:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.259+0000 7fc767e00640 1 -- 172.21.15.28:0/2474397203 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7fc774004260 con 0x56176ad73820 2024-04-26T01:53:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.260+0000 7fc78c03c740 1 -- 172.21.15.28:0/2474397203 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56176aced8b0 con 0x56176ad73820 2024-04-26T01:53:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.261+0000 7fc767e00640 1 -- 172.21.15.28:0/2474397203 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc774004460 con 0x56176ad73820 2024-04-26T01:53:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.261+0000 7fc78c03c740 10 client.26269 fetch_fsmap learned FSMap version 45 2024-04-26T01:53:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.261+0000 7fc78c03c740 1 -- 172.21.15.28:0/2474397203 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56176adaf9a0 con 0x56176ad73820 2024-04-26T01:53:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.262+0000 7fc767e00640 1 -- 172.21.15.28:0/2474397203 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56176aced8b0 con 0x56176ad73820 2024-04-26T01:53:30.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.265+0000 7fc767e00640 1 -- 172.21.15.28:0/2474397203 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7fc774026760 con 0x56176ad73820 2024-04-26T01:53:30.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.265+0000 7fc78c03c740 10 client.26269 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:53:30.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.266+0000 7fc78c03c740 10 client.26269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:30.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.266+0000 7fc78c03c740 20 client.26269 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:30.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.266+0000 7fc78c03c740 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56176adb0e10 0x56176adb3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.266+0000 7fc78c03c740 4 client.26269 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:30.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.266+0000 7fc78c03c740 1 -- 172.21.15.28:0/2474397203 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56176ace7320 con 0x56176adb0e10 2024-04-26T01:53:30.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.267+0000 7fc781400640 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56176adb0e10 0x56176adb3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.269+0000 7fc781400640 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56176adb0e10 0x56176adb3200 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8352d06dd69a9a66 server_cookie=c2d714fd61ece07a in_seq=0 out_seq=0 2024-04-26T01:53:30.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.269+0000 7fc767e00640 10 client.26269 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.273+0000 7fc767e00640 1 -- 172.21.15.28:0/2474397203 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56176ace7320 con 0x56176adb0e10 2024-04-26T01:53:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.273+0000 7fc767e00640 10 client.26269 handle_command_reply: tid=0 2024-04-26T01:53:30.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.273+0000 7fc78c03c740 1 client.26269 shutdown 2024-04-26T01:53:30.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc78c03c740 1 -- 172.21.15.28:0/2474397203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 msgr2=0x56176ad7ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc78c03c740 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad7ba20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc77400f790 tx=0x7fc774005f90 comp rx=0 tx=0).stop 2024-04-26T01:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc78c03c740 1 -- 172.21.15.28:0/2474397203 shutdown_connections 2024-04-26T01:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc78c03c740 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56176adb0e10 0x56176adb3200 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc78c03c740 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad7ea00 0x56176ad7c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc78c03c740 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad7bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc78c03c740 1 --2- 172.21.15.28:0/2474397203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad7ba20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc78c03c740 1 -- 172.21.15.28:0/2474397203 >> 172.21.15.28:0/2474397203 conn(0x56176ad6fd30 msgr2=0x56176ad75ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.274+0000 7fc767e00640 0 client.26269 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.275+0000 7fc78c03c740 1 -- 172.21.15.28:0/2474397203 shutdown_connections 2024-04-26T01:53:30.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.275+0000 7fc78c03c740 1 -- 172.21.15.28:0/2474397203 wait complete. 2024-04-26T01:53:30.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.275+0000 7fc78c03c740 20 client.26269 trim_cache size 0 max 16384 2024-04-26T01:53:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.310+0000 7fc78c03c740 1 Processor -- start 2024-04-26T01:53:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.310+0000 7fc78c03c740 1 -- start start 2024-04-26T01:53:30.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 0x56176ad7a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad76230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad7ea00 0x56176ad7c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc78c03c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc78410ad60 con 0x56176ad73820 2024-04-26T01:53:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc78c03c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc7841085a0 con 0x56176ad74140 2024-04-26T01:53:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc78c03c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc78404fce0 con 0x56176ad7ea00 2024-04-26T01:53:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc782800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad7ea00 0x56176ad7c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc782800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad7ea00 0x56176ad7c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:41568/0 (socket says 172.21.15.28:41568) 2024-04-26T01:53:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc782800640 1 -- 172.21.15.28:0/1945845798 learned_addr learned my addr 172.21.15.28:0/1945845798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.311+0000 7fc781400640 1 --2- 172.21.15.28:0/1945845798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad76230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.312+0000 7fc782800640 1 -- 172.21.15.28:0/1945845798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 msgr2=0x56176ad76230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.312+0000 7fc782800640 1 --2- 172.21.15.28:0/1945845798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad76230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.312+0000 7fc782800640 1 -- 172.21.15.28:0/1945845798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 msgr2=0x56176ad7a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.312+0000 7fc782800640 1 --2- 172.21.15.28:0/1945845798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 0x56176ad7a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.312+0000 7fc782800640 1 -- 172.21.15.28:0/1945845798 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56176ad79f70 con 0x56176ad7ea00 2024-04-26T01:53:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.313+0000 7fc782800640 1 --2- 172.21.15.28:0/1945845798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad7ea00 0x56176ad7c2c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc77c006010 tx=0x7fc77c005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6889f6e05137b039 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.314+0000 7fc772a00640 1 -- 172.21.15.28:0/1945845798 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c01d070 con 0x56176ad7ea00 2024-04-26T01:53:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.314+0000 7fc772a00640 1 -- 172.21.15.28:0/1945845798 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c02a070 con 0x56176ad7ea00 2024-04-26T01:53:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.314+0000 7fc772a00640 1 -- 172.21.15.28:0/1945845798 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0053a0 con 0x56176ad7ea00 2024-04-26T01:53:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.316+0000 7fc78c03c740 1 -- 172.21.15.28:0/1945845798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad7ea00 msgr2=0x56176ad7c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.316+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1945845798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad7ea00 0x56176ad7c2c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc77c006010 tx=0x7fc77c005d80 comp rx=0 tx=0).stop 2024-04-26T01:53:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.317+0000 7fc78c03c740 1 -- 172.21.15.28:0/1945845798 shutdown_connections 2024-04-26T01:53:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.317+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1945845798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad7ea00 0x56176ad7c2c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.317+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1945845798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad74140 0x56176ad76230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.317+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1945845798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad73820 0x56176ad7a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.317+0000 7fc78c03c740 1 -- 172.21.15.28:0/1945845798 >> 172.21.15.28:0/1945845798 conn(0x56176ad6fd30 msgr2=0x56176ad6ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.317+0000 7fc78c03c740 1 -- 172.21.15.28:0/1945845798 shutdown_connections 2024-04-26T01:53:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.317+0000 7fc78c03c740 1 -- 172.21.15.28:0/1945845798 wait complete. 2024-04-26T01:53:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.317+0000 7fc78c03c740 1 -- start start 2024-04-26T01:53:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.318+0000 7fc78c03c740 1 Processor -- start 2024-04-26T01:53:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.319+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad97c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.319+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad74140 0x56176ad94fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.319+0000 7fc78c03c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad95520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.319+0000 7fc78c03c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7840738d0 con 0x56176ad74140 2024-04-26T01:53:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.319+0000 7fc78c03c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc784107ed0 con 0x56176ad7ea00 2024-04-26T01:53:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.319+0000 7fc78c03c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc784073f60 con 0x56176ad73820 2024-04-26T01:53:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.320+0000 7fc78c03c740 10 client.0 _ll_register_callbacks cb 0x7fc77400fd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.321+0000 7fc781e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad97c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.321+0000 7fc782800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad95520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.321+0000 7fc782800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad95520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35164/0 (socket says 172.21.15.28:35164) 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.321+0000 7fc782800640 1 -- 172.21.15.28:0/1657420472 learned_addr learned my addr 172.21.15.28:0/1657420472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.322+0000 7fc781e00640 1 -- 172.21.15.28:0/1657420472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 msgr2=0x56176ad95520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.322+0000 7fc781e00640 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad95520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.322+0000 7fc781e00640 1 -- 172.21.15.28:0/1657420472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad74140 msgr2=0x56176ad94fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.322+0000 7fc781e00640 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad74140 0x56176ad94fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.322+0000 7fc781e00640 1 -- 172.21.15.28:0/1657420472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56176ad95b60 con 0x56176ad73820 2024-04-26T01:53:30.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.323+0000 7fc767e00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:53:30.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.323+0000 7fc781e00640 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad97c00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc774026bb0 tx=0x7fc77402f3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a4f9ea979364eca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:30.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.324+0000 7fc767e00640 1 -- 172.21.15.28:0/1657420472 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774035080 con 0x56176ad73820 2024-04-26T01:53:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.324+0000 7fc767e00640 1 -- 172.21.15.28:0/1657420472 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7fc774004260 con 0x56176ad73820 2024-04-26T01:53:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.324+0000 7fc78c03c740 1 -- 172.21.15.28:0/1657420472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc77c02a070 con 0x56176ad73820 2024-04-26T01:53:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.325+0000 7fc767e00640 1 -- 172.21.15.28:0/1657420472 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc774035080 con 0x56176ad73820 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.325+0000 7fc78c03c740 10 client.26275 fetch_fsmap learned FSMap version 45 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.325+0000 7fc78c03c740 1 -- 172.21.15.28:0/1657420472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56176ae171b0 con 0x56176ad73820 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.325+0000 7fc767e00640 1 -- 172.21.15.28:0/1657420472 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc774005bf0 con 0x56176ad73820 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.325+0000 7fc767e00640 1 -- 172.21.15.28:0/1657420472 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7fc77402f790 con 0x56176ad73820 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.326+0000 7fc78c03c740 10 client.26275 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.326+0000 7fc78c03c740 10 client.26275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.326+0000 7fc78c03c740 20 client.26275 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.326+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56176ae17900 0x56176ae19d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.326+0000 7fc78c03c740 4 client.26275 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.326+0000 7fc78c03c740 1 -- 172.21.15.28:0/1657420472 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fc77c02a070 con 0x56176ae17900 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.326+0000 7fc781400640 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56176ae17900 0x56176ae19d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.327+0000 7fc781400640 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56176ae17900 0x56176ae19d20 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=394029d8e1f6e1c0 server_cookie=c2150d09dfc4f8d0 in_seq=0 out_seq=0 2024-04-26T01:53:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.327+0000 7fc767e00640 10 client.26275 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:30.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.345+0000 7fc767e00640 1 -- 172.21.15.28:0/1657420472 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fc77c02a070 con 0x56176ae17900 2024-04-26T01:53:30.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.345+0000 7fc767e00640 10 client.26275 handle_command_reply: tid=0 2024-04-26T01:53:30.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.346+0000 7fc78c03c740 1 client.26275 shutdown 2024-04-26T01:53:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.346+0000 7fc78c03c740 1 -- 172.21.15.28:0/1657420472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 msgr2=0x56176ad97c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.346+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad97c00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc774026bb0 tx=0x7fc77402f3e0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.347+0000 7fc78c03c740 1 -- 172.21.15.28:0/1657420472 shutdown_connections 2024-04-26T01:53:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.347+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56176ae17900 0x56176ae19d20 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.347+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56176ad7ea00 0x56176ad95520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.347+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176ad74140 0x56176ad94fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.347+0000 7fc78c03c740 1 --2- 172.21.15.28:0/1657420472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56176ad73820 0x56176ad97c00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.347+0000 7fc767e00640 0 client.26275 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.347+0000 7fc78c03c740 1 -- 172.21.15.28:0/1657420472 >> 172.21.15.28:0/1657420472 conn(0x56176ad6fd30 msgr2=0x56176ad7d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.348+0000 7fc78c03c740 1 -- 172.21.15.28:0/1657420472 shutdown_connections 2024-04-26T01:53:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.348+0000 7fc78c03c740 1 -- 172.21.15.28:0/1657420472 wait complete. 2024-04-26T01:53:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.348+0000 7fc78c03c740 20 client.26275 trim_cache size 0 max 16384 2024-04-26T01:53:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:53:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:53:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:53:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:53:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.349+0000 7fc789200640 1 -- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc7680a3e40 msgr2=0x7fc7680a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.349+0000 7fc789200640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc7680a3e40 0x7fc7680a6300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc774009710 tx=0x7fc774004730 comp rx=0 tx=0).stop 2024-04-26T01:53:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.349+0000 7fc789200640 1 -- 172.21.15.28:0/3263052798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7841018f0 msgr2=0x7fc7841a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.349+0000 7fc789200640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7841018f0 0x7fc7841a4fa0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009980 tx=0x7fc76c032ec0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.350+0000 7fc789200640 1 -- 172.21.15.28:0/3263052798 shutdown_connections 2024-04-26T01:53:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.350+0000 7fc789200640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc7680a3e40 0x7fc7680a6300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.350+0000 7fc789200640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7841063d0 0x7fc78419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.350+0000 7fc789200640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7841018f0 0x7fc7841a4fa0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.350+0000 7fc789200640 1 --2- 172.21.15.28:0/3263052798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc784100fd0 0x7fc7841a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.350+0000 7fc789200640 1 -- 172.21.15.28:0/3263052798 >> 172.21.15.28:0/3263052798 conn(0x7fc784077f60 msgr2=0x7fc784103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.350+0000 7fc789200640 1 -- 172.21.15.28:0/3263052798 shutdown_connections 2024-04-26T01:53:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:30.350+0000 7fc789200640 1 -- 172.21.15.28:0/3263052798 wait complete. 2024-04-26T01:53:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 695.0 s, 1975.8 tps, lat 51.027 ms stddev 62.021 2024-04-26T01:53:31.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:31 smithi053 ceph-mon[32593]: pgmap v393: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 907 op/s 2024-04-26T01:53:31.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:31 smithi028 ceph-mon[28247]: pgmap v393: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 907 op/s 2024-04-26T01:53:31.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:31 smithi169 ceph-mon[32851]: pgmap v393: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 10 MiB/s wr, 907 op/s 2024-04-26T01:53:33.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:33 smithi053 ceph-mon[32593]: pgmap v394: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.8 MiB/s wr, 606 op/s 2024-04-26T01:53:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:33 smithi028 ceph-mon[28247]: pgmap v394: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.8 MiB/s wr, 606 op/s 2024-04-26T01:53:33.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:33 smithi169 ceph-mon[32851]: pgmap v394: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.8 MiB/s wr, 606 op/s 2024-04-26T01:53:34.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:34 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:34.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:34 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:34.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:34 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:35.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:35 smithi053 ceph-mon[32593]: pgmap v395: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.7 MiB/s wr, 702 op/s 2024-04-26T01:53:35.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:35 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:35.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:35 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:35.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:35 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:53:35.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:35 smithi028 ceph-mon[28247]: pgmap v395: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.7 MiB/s wr, 702 op/s 2024-04-26T01:53:35.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:35 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:35.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:35 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:35.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:35 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:53:35.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:35 smithi169 ceph-mon[32851]: pgmap v395: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.7 MiB/s wr, 702 op/s 2024-04-26T01:53:35.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:35 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:35.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:35 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:35.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:35 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:53:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 700.0 s, 1677.6 tps, lat 59.697 ms stddev 67.090 2024-04-26T01:53:37.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:37 smithi053 ceph-mon[32593]: pgmap v396: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.7 MiB/s wr, 780 op/s 2024-04-26T01:53:37.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:37 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:53:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:37 smithi028 ceph-mon[28247]: pgmap v396: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.7 MiB/s wr, 780 op/s 2024-04-26T01:53:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:37 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:53:37.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:37 smithi169 ceph-mon[32851]: pgmap v396: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.7 MiB/s wr, 780 op/s 2024-04-26T01:53:37.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:37 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:53:39.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:39 smithi053 ceph-mon[32593]: pgmap v397: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.4 MiB/s wr, 595 op/s 2024-04-26T01:53:39.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:39 smithi028 ceph-mon[28247]: pgmap v397: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.4 MiB/s wr, 595 op/s 2024-04-26T01:53:39.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:39 smithi169 ceph-mon[32851]: pgmap v397: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.4 MiB/s wr, 595 op/s 2024-04-26T01:53:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 705.0 s, 2095.8 tps, lat 46.739 ms stddev 56.036 2024-04-26T01:53:41.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:41 smithi028 ceph-mon[28247]: pgmap v398: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.1 MiB/s wr, 864 op/s 2024-04-26T01:53:41.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:41 smithi169 ceph-mon[32851]: pgmap v398: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.1 MiB/s wr, 864 op/s 2024-04-26T01:53:41.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:41 smithi053 ceph-mon[32593]: pgmap v398: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.1 MiB/s wr, 864 op/s 2024-04-26T01:53:43.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.529+0000 7fad57600640 1 -- 172.21.15.28:0/2269040646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad50074b70 msgr2=0x7fad5010e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.529+0000 7fad57600640 1 --2- 172.21.15.28:0/2269040646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad50074b70 0x7fad5010e300 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009900 tx=0x7fad3c02a480 comp rx=0 tx=0).stop 2024-04-26T01:53:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.530+0000 7fad57600640 1 -- 172.21.15.28:0/2269040646 shutdown_connections 2024-04-26T01:53:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.530+0000 7fad57600640 1 --2- 172.21.15.28:0/2269040646 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad5010e840 0x7fad50110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.530+0000 7fad57600640 1 --2- 172.21.15.28:0/2269040646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad50074b70 0x7fad5010e300 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.530+0000 7fad57600640 1 --2- 172.21.15.28:0/2269040646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad50074250 0x7fad50074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.530+0000 7fad57600640 1 -- 172.21.15.28:0/2269040646 >> 172.21.15.28:0/2269040646 conn(0x7fad500793e0 msgr2=0x7fad500797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.530+0000 7fad57600640 1 -- 172.21.15.28:0/2269040646 shutdown_connections 2024-04-26T01:53:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.530+0000 7fad57600640 1 -- 172.21.15.28:0/2269040646 wait complete. 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad57600640 1 Processor -- start 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad57600640 1 -- start start 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad57600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50074250 0x7fad501a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad57600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad50074b70 0x7fad501a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad57600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad5010e840 0x7fad5019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad57600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad50114420 con 0x7fad50074b70 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad57600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fad501142a0 con 0x7fad5010e840 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad57600640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fad501145a0 con 0x7fad50074250 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.531+0000 7fad54c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50074250 0x7fad501a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad54c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50074250 0x7fad501a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:45494/0 (socket says 172.21.15.28:45494) 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad54c00640 1 -- 172.21.15.28:0/1752307620 learned_addr learned my addr 172.21.15.28:0/1752307620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad55600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad5010e840 0x7fad5019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad4fe00640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad50074b70 0x7fad501a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad55600640 1 -- 172.21.15.28:0/1752307620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50074250 msgr2=0x7fad501a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad55600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50074250 0x7fad501a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad55600640 1 -- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad50074b70 msgr2=0x7fad501a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad55600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad50074b70 0x7fad501a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.532+0000 7fad55600640 1 -- 172.21.15.28:0/1752307620 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5019af40 con 0x7fad5010e840 2024-04-26T01:53:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.534+0000 7fad55600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad5010e840 0x7fad5019a680 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fad4800d600 tx=0x7fad4800dac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.535+0000 7fad4d600640 1 -- 172.21.15.28:0/1752307620 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48004230 con 0x7fad5010e840 2024-04-26T01:53:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.535+0000 7fad4d600640 1 -- 172.21.15.28:0/1752307620 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad480043d0 con 0x7fad5010e840 2024-04-26T01:53:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.535+0000 7fad57600640 1 -- 172.21.15.28:0/1752307620 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5019b230 con 0x7fad5010e840 2024-04-26T01:53:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.536+0000 7fad4d600640 1 -- 172.21.15.28:0/1752307620 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48003dd0 con 0x7fad5010e840 2024-04-26T01:53:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.536+0000 7fad57600640 1 -- 172.21.15.28:0/1752307620 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad501a74f0 con 0x7fad5010e840 2024-04-26T01:53:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.541+0000 7fad5a364740 1 Processor -- start 2024-04-26T01:53:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.541+0000 7fad4d600640 1 -- 172.21.15.28:0/1752307620 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fad48010460 con 0x7fad5010e840 2024-04-26T01:53:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.541+0000 7fad5a364740 1 -- start start 2024-04-26T01:53:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.541+0000 7fad5a364740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 0x55c97197a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.541+0000 7fad5a364740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c971984cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad5a364740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c971987600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad5a364740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c9718d7900 con 0x55c971985210 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad5a364740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c97187def0 con 0x55c97197a950 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad5a364740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c9718d38c0 con 0x55c97197a030 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad54c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c971984cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad54c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c971984cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38450/0 (socket says 172.21.15.28:38450) 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad54c00640 1 -- 172.21.15.28:0/2832261007 learned_addr learned my addr 172.21.15.28:0/2832261007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad4fe00640 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 0x55c97197a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad54c00640 1 -- 172.21.15.28:0/2832261007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 msgr2=0x55c97197a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad54c00640 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 0x55c97197a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad54c00640 1 -- 172.21.15.28:0/2832261007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 msgr2=0x55c971987600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad4fe00640 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c971987600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad54c00640 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c971987600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad54c00640 1 -- 172.21.15.28:0/2832261007 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9715ea210 con 0x55c97197a950 2024-04-26T01:53:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.542+0000 7fad4fe00640 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c971987600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:53:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.543+0000 7fad54c00640 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c971984cd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fad40010560 tx=0x7fad40010a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c371dd1c4312970f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.543+0000 7fad45c00640 1 -- 172.21.15.28:0/2832261007 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4000e7d0 con 0x55c97197a950 2024-04-26T01:53:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.543+0000 7fad45c00640 1 -- 172.21.15.28:0/2832261007 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad4000ee20 con 0x55c97197a950 2024-04-26T01:53:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.543+0000 7fad45c00640 1 -- 172.21.15.28:0/2832261007 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad40011e60 con 0x55c97197a950 2024-04-26T01:53:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.543+0000 7fad4d600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fad340a3f20 0x7fad340a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.543+0000 7fad4d600640 1 -- 172.21.15.28:0/1752307620 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7fad480cffd0 con 0x7fad5010e840 2024-04-26T01:53:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.543+0000 7fad4fe00640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fad340a3f20 0x7fad340a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 -- 172.21.15.28:0/2832261007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 msgr2=0x55c971984cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c971984cd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fad40010560 tx=0x7fad40010a20 comp rx=0 tx=0).stop 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 -- 172.21.15.28:0/2832261007 shutdown_connections 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c971987600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c971984cd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 --2- 172.21.15.28:0/2832261007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 0x55c97197a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 -- 172.21.15.28:0/2832261007 >> 172.21.15.28:0/2832261007 conn(0x55c971976540 msgr2=0x55c971976920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 -- 172.21.15.28:0/2832261007 shutdown_connections 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad5a364740 1 -- 172.21.15.28:0/2832261007 wait complete. 2024-04-26T01:53:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.544+0000 7fad4fe00640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fad340a3f20 0x7fad340a63e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fad3c005ab0 tx=0x7fad3c005a20 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.545+0000 7fad5a364740 1 -- start start 2024-04-26T01:53:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.545+0000 7fad5a364740 1 Processor -- start 2024-04-26T01:53:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.546+0000 7fad5a364740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a030 0x55c971982130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.546+0000 7fad5a364740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a950 0x55c971982670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.546+0000 7fad5a364740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c971985210 0x55c971982bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.546+0000 7fad5a364740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c971987e40 con 0x55c97197a030 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.546+0000 7fad5a364740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c971891310 con 0x55c971985210 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.546+0000 7fad5a364740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55c971987fc0 con 0x55c97197a950 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.547+0000 7fad5a364740 10 client.0 _ll_register_callbacks cb 0x55c97185fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.548+0000 7fad4fe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a950 0x55c971982670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.548+0000 7fad54c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a030 0x55c971982130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.548+0000 7fad55600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c971985210 0x55c971982bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.548+0000 7fad55600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c971985210 0x55c971982bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38462/0 (socket says 172.21.15.28:38462) 2024-04-26T01:53:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.548+0000 7fad55600640 1 -- 172.21.15.28:0/2482898579 learned_addr learned my addr 172.21.15.28:0/2482898579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.549+0000 7fad55600640 1 -- 172.21.15.28:0/2482898579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a950 msgr2=0x55c971982670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.549+0000 7fad55600640 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a950 0x55c971982670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.549+0000 7fad55600640 1 -- 172.21.15.28:0/2482898579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a030 msgr2=0x55c971982130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.549+0000 7fad55600640 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a030 0x55c971982130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.549+0000 7fad55600640 1 -- 172.21.15.28:0/2482898579 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9716b6620 con 0x55c971985210 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.549+0000 7fad55600640 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c971985210 0x55c971982bb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fad480989f0 tx=0x7fad4809bbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8352c0654e88576 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.549+0000 7fad3a000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.549+0000 7fad3a000640 1 -- 172.21.15.28:0/2482898579 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480b5060 con 0x55c971985210 2024-04-26T01:53:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad3a000640 1 -- 172.21.15.28:0/2482898579 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7fad48096b90 con 0x55c971985210 2024-04-26T01:53:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad5a364740 1 -- 172.21.15.28:0/2482898579 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad4000ee20 con 0x55c971985210 2024-04-26T01:53:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad3a000640 1 -- 172.21.15.28:0/2482898579 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fad48020040 con 0x55c971985210 2024-04-26T01:53:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad5a364740 10 client.26273 fetch_fsmap learned FSMap version 46 2024-04-26T01:53:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad5a364740 1 -- 172.21.15.28:0/2482898579 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55c971993fb0 con 0x55c971985210 2024-04-26T01:53:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad3a000640 1 -- 172.21.15.28:0/2482898579 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad4000ee20 con 0x55c971985210 2024-04-26T01:53:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad3a000640 1 -- 172.21.15.28:0/2482898579 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7fad480b5290 con 0x55c971985210 2024-04-26T01:53:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad5a364740 10 client.26273 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:53:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad5a364740 10 client.26273 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.550+0000 7fad5a364740 20 client.26273 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.551+0000 7fad5a364740 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c971995470 0x55c971997880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.551+0000 7fad5a364740 4 client.26273 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.551+0000 7fad5a364740 1 -- 172.21.15.28:0/2482898579 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9718eb8b0 con 0x55c971995470 2024-04-26T01:53:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.551+0000 7fad54c00640 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c971995470 0x55c971997880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.552+0000 7fad3a000640 10 client.26273 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.552+0000 7fad54c00640 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c971995470 0x55c971997880 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=725b7785dc9bdc48 server_cookie=c0a1993e60245c00 in_seq=0 out_seq=0 2024-04-26T01:53:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.553+0000 7fad3a000640 1 -- 172.21.15.28:0/2482898579 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9718eb8b0 con 0x55c971995470 2024-04-26T01:53:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.553+0000 7fad3a000640 10 client.26273 handle_command_reply: tid=0 2024-04-26T01:53:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 client.26273 shutdown 2024-04-26T01:53:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 -- 172.21.15.28:0/2482898579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c971985210 msgr2=0x55c971982bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c971985210 0x55c971982bb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fad480989f0 tx=0x7fad4809bbe0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 -- 172.21.15.28:0/2482898579 shutdown_connections 2024-04-26T01:53:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c971995470 0x55c971997880 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad3a000640 0 client.26273 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c971985210 0x55c971982bb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a950 0x55c971982670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 --2- 172.21.15.28:0/2482898579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a030 0x55c971982130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 -- 172.21.15.28:0/2482898579 >> 172.21.15.28:0/2482898579 conn(0x55c971976540 msgr2=0x55c97196ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 -- 172.21.15.28:0/2482898579 shutdown_connections 2024-04-26T01:53:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.554+0000 7fad5a364740 1 -- 172.21.15.28:0/2482898579 wait complete. 2024-04-26T01:53:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.555+0000 7fad5a364740 20 client.26273 trim_cache size 0 max 16384 2024-04-26T01:53:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.592+0000 7fad5a364740 1 Processor -- start 2024-04-26T01:53:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.592+0000 7fad5a364740 1 -- start start 2024-04-26T01:53:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.592+0000 7fad5a364740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 0x55c971990d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.592+0000 7fad5a364740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c9719829d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.592+0000 7fad5a364740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c97199d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.593+0000 7fad54c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 0x55c971990d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.593+0000 7fad4fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c9719829d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.593+0000 7fad5a364740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad501ae250 con 0x55c971985210 2024-04-26T01:53:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.593+0000 7fad5a364740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fad501abac0 con 0x55c97197a950 2024-04-26T01:53:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.593+0000 7fad4fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c9719829d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38464/0 (socket says 172.21.15.28:38464) 2024-04-26T01:53:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.593+0000 7fad4fe00640 1 -- 172.21.15.28:0/1065300496 learned_addr learned my addr 172.21.15.28:0/1065300496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.593+0000 7fad55600640 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c97199d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.593+0000 7fad5a364740 1 -- 172.21.15.28:0/1065300496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fad501a9970 con 0x55c97197a030 2024-04-26T01:53:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.594+0000 7fad4fe00640 1 -- 172.21.15.28:0/1065300496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 msgr2=0x55c971990d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.594+0000 7fad4fe00640 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 0x55c971990d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.594+0000 7fad4fe00640 1 -- 172.21.15.28:0/1065300496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 msgr2=0x55c97199d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.594+0000 7fad4fe00640 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c97199d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.594+0000 7fad4fe00640 1 -- 172.21.15.28:0/1065300496 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c97197a780 con 0x55c97197a950 2024-04-26T01:53:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.594+0000 7fad55600640 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c97199d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.595+0000 7fad4fe00640 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c9719829d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009230 tx=0x7fad3c03f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6babf90da0299d17 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.596+0000 7fad45200640 1 -- 172.21.15.28:0/1065300496 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c003db0 con 0x55c97197a950 2024-04-26T01:53:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.596+0000 7fad45200640 1 -- 172.21.15.28:0/1065300496 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad3c031950 con 0x55c97197a950 2024-04-26T01:53:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.597+0000 7fad5a364740 1 -- 172.21.15.28:0/1065300496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 msgr2=0x55c9719829d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.597+0000 7fad5a364740 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c9719829d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009230 tx=0x7fad3c03f040 comp rx=0 tx=0).stop 2024-04-26T01:53:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.599+0000 7fad5a364740 1 -- 172.21.15.28:0/1065300496 shutdown_connections 2024-04-26T01:53:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.599+0000 7fad5a364740 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c971985210 0x55c97199d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.599+0000 7fad5a364740 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a950 0x55c9719829d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.599+0000 7fad5a364740 1 --2- 172.21.15.28:0/1065300496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c97197a030 0x55c971990d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.599+0000 7fad5a364740 1 -- 172.21.15.28:0/1065300496 >> 172.21.15.28:0/1065300496 conn(0x55c971976540 msgr2=0x55c971976920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.599+0000 7fad5a364740 1 -- 172.21.15.28:0/1065300496 shutdown_connections 2024-04-26T01:53:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.599+0000 7fad5a364740 1 -- 172.21.15.28:0/1065300496 wait complete. 2024-04-26T01:53:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.600+0000 7fad5a364740 1 -- start start 2024-04-26T01:53:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.600+0000 7fad5a364740 1 Processor -- start 2024-04-26T01:53:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.602+0000 7fad5a364740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a030 0x55c97198cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.602+0000 7fad5a364740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a950 0x55c97198d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.603+0000 7fad54c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a030 0x55c97198cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.603+0000 7fad54c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a030 0x55c97198cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38468/0 (socket says 172.21.15.28:38468) 2024-04-26T01:53:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.604+0000 7fad54c00640 1 -- 172.21.15.28:0/2010332889 learned_addr learned my addr 172.21.15.28:0/2010332889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.604+0000 7fad5a364740 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c971985210 0x55c97198a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.604+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad501a9fc0 con 0x55c97197a950 2024-04-26T01:53:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.604+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fad501ab3f0 con 0x55c97197a030 2024-04-26T01:53:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.604+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fad501aa650 con 0x55c971985210 2024-04-26T01:53:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.604+0000 7fad5a364740 10 client.0 _ll_register_callbacks cb 0x7fad4809bdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.604+0000 7fad55600640 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c971985210 0x55c97198a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.605+0000 7fad55600640 1 -- 172.21.15.28:0/2010332889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a030 msgr2=0x55c97198cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.605+0000 7fad4fe00640 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a950 0x55c97198d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.605+0000 7fad55600640 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a030 0x55c97198cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.605+0000 7fad55600640 1 -- 172.21.15.28:0/2010332889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a950 msgr2=0x55c97198d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.605+0000 7fad55600640 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a950 0x55c97198d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.605+0000 7fad55600640 1 -- 172.21.15.28:0/2010332889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c97198ab50 con 0x55c971985210 2024-04-26T01:53:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.605+0000 7fad4fe00640 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a950 0x55c97198d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:53:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.606+0000 7fad55600640 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c971985210 0x55c97198a3c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fad48098480 tx=0x7fad480988a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a51622718c08bfd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.606+0000 7fad3a000640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:53:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.606+0000 7fad3a000640 1 -- 172.21.15.28:0/2010332889 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480a4be0 con 0x55c971985210 2024-04-26T01:53:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.606+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad3c031950 con 0x55c971985210 2024-04-26T01:53:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.608+0000 7fad3a000640 1 -- 172.21.15.28:0/2010332889 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7fad480b5060 con 0x55c971985210 2024-04-26T01:53:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.608+0000 7fad3a000640 1 -- 172.21.15.28:0/2010332889 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fad480b5280 con 0x55c971985210 2024-04-26T01:53:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.608+0000 7fad5a364740 10 client.26293 fetch_fsmap learned FSMap version 46 2024-04-26T01:53:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.608+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55c9719fb4a0 con 0x55c971985210 2024-04-26T01:53:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.609+0000 7fad3a000640 1 -- 172.21.15.28:0/2010332889 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad3c031950 con 0x55c971985210 2024-04-26T01:53:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.609+0000 7fad3a000640 1 -- 172.21.15.28:0/2010332889 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7fad480a14b0 con 0x55c971985210 2024-04-26T01:53:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.609+0000 7fad5a364740 10 client.26293 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:53:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.609+0000 7fad5a364740 10 client.26293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.609+0000 7fad5a364740 20 client.26293 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.609+0000 7fad5a364740 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c971995470 0x55c9719fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.609+0000 7fad5a364740 4 client.26293 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.609+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c9718e5320 con 0x55c971995470 2024-04-26T01:53:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.610+0000 7fad54c00640 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c971995470 0x55c9719fdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.610+0000 7fad54c00640 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c971995470 0x55c9719fdb50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1822906729e30ed4 server_cookie=9c668be22f3bc2f2 in_seq=0 out_seq=0 2024-04-26T01:53:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.610+0000 7fad3a000640 10 client.26293 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:43.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.667+0000 7fad3a000640 1 -- 172.21.15.28:0/2010332889 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c9718e5320 con 0x55c971995470 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.667+0000 7fad3a000640 10 client.26293 handle_command_reply: tid=0 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.667+0000 7fad5a364740 1 client.26293 shutdown 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c971985210 msgr2=0x55c97198a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c971985210 0x55c97198a3c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fad48098480 tx=0x7fad480988a0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 shutdown_connections 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55c971995470 0x55c9719fdb50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c971985210 0x55c97198a3c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c97197a950 0x55c97198d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 --2- 172.21.15.28:0/2010332889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97197a030 0x55c97198cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad3a000640 0 client.26293 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 >> 172.21.15.28:0/2010332889 conn(0x55c971976540 msgr2=0x55c97197b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 shutdown_connections 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 1 -- 172.21.15.28:0/2010332889 wait complete. 2024-04-26T01:53:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.668+0000 7fad5a364740 20 client.26293 trim_cache size 0 max 16384 2024-04-26T01:53:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:53:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:53:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:53:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:53:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 -- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fad340a3f20 msgr2=0x7fad340a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fad340a3f20 0x7fad340a63e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fad3c005ab0 tx=0x7fad3c005a20 comp rx=0 tx=0).stop 2024-04-26T01:53:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 -- 172.21.15.28:0/1752307620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad5010e840 msgr2=0x7fad5019a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad5010e840 0x7fad5019a680 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fad4800d600 tx=0x7fad4800dac0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 -- 172.21.15.28:0/1752307620 shutdown_connections 2024-04-26T01:53:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fad340a3f20 0x7fad340a63e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad5010e840 0x7fad5019a680 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad50074b70 0x7fad501a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 --2- 172.21.15.28:0/1752307620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50074250 0x7fad501a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 -- 172.21.15.28:0/1752307620 >> 172.21.15.28:0/1752307620 conn(0x7fad500793e0 msgr2=0x7fad500fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 -- 172.21.15.28:0/1752307620 shutdown_connections 2024-04-26T01:53:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:43.669+0000 7fad57600640 1 -- 172.21.15.28:0/1752307620 wait complete. 2024-04-26T01:53:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:43 smithi028 ceph-mon[28247]: pgmap v399: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.4 MiB/s wr, 540 op/s 2024-04-26T01:53:43.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:43 smithi169 ceph-mon[32851]: pgmap v399: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.4 MiB/s wr, 540 op/s 2024-04-26T01:53:43.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:43 smithi053 ceph-mon[32593]: pgmap v399: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.4 MiB/s wr, 540 op/s 2024-04-26T01:53:45.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:45 smithi028 ceph-mon[28247]: pgmap v400: 97 pgs: 97 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-04-26T01:53:45.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:45 smithi169 ceph-mon[32851]: pgmap v400: 97 pgs: 97 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-04-26T01:53:45.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:45 smithi053 ceph-mon[32593]: pgmap v400: 97 pgs: 97 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-04-26T01:53:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 710.0 s, 1207.0 tps, lat 84.223 ms stddev 155.279 2024-04-26T01:53:47.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:47 smithi028 ceph-mon[28247]: pgmap v401: 97 pgs: 97 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 MiB/s wr, 1.77k op/s 2024-04-26T01:53:47.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:47 smithi169 ceph-mon[32851]: pgmap v401: 97 pgs: 97 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 MiB/s wr, 1.77k op/s 2024-04-26T01:53:47.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:47 smithi053 ceph-mon[32593]: pgmap v401: 97 pgs: 97 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 MiB/s wr, 1.77k op/s 2024-04-26T01:53:49.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:49 smithi028 ceph-mon[28247]: pgmap v402: 97 pgs: 97 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 23 MiB/s wr, 1.59k op/s 2024-04-26T01:53:49.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:49 smithi169 ceph-mon[32851]: pgmap v402: 97 pgs: 97 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 23 MiB/s wr, 1.59k op/s 2024-04-26T01:53:49.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:49 smithi053 ceph-mon[32593]: pgmap v402: 97 pgs: 97 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 23 MiB/s wr, 1.59k op/s 2024-04-26T01:53:51.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 715.0 s, 2077.1 tps, lat 48.083 ms stddev 56.793 2024-04-26T01:53:51.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:51 smithi028 ceph-mon[28247]: pgmap v403: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 25 MiB/s wr, 1.90k op/s 2024-04-26T01:53:51.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:51 smithi169 ceph-mon[32851]: pgmap v403: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 25 MiB/s wr, 1.90k op/s 2024-04-26T01:53:51.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:51 smithi053 ceph-mon[32593]: pgmap v403: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 25 MiB/s wr, 1.90k op/s 2024-04-26T01:53:53.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:53 smithi028 ceph-mon[28247]: pgmap v404: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 MiB/s wr, 1.64k op/s 2024-04-26T01:53:53.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:53 smithi169 ceph-mon[32851]: pgmap v404: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 MiB/s wr, 1.64k op/s 2024-04-26T01:53:53.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:53 smithi053 ceph-mon[32593]: pgmap v404: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 MiB/s wr, 1.64k op/s 2024-04-26T01:53:55.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:55 smithi028 ceph-mon[28247]: pgmap v405: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 24 MiB/s wr, 1.75k op/s 2024-04-26T01:53:55.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:55 smithi169 ceph-mon[32851]: pgmap v405: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 24 MiB/s wr, 1.75k op/s 2024-04-26T01:53:55.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:55 smithi053 ceph-mon[32593]: pgmap v405: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 24 MiB/s wr, 1.75k op/s 2024-04-26T01:53:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 720.0 s, 1636.3 tps, lat 60.892 ms stddev 68.500 2024-04-26T01:53:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 -- 172.21.15.28:0/2683031914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c108f20 msgr2=0x7f908c1153e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 --2- 172.21.15.28:0/2683031914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c108f20 0x7f908c1153e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9084009be0 tx=0x7f908402a6c0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 -- 172.21.15.28:0/2683031914 shutdown_connections 2024-04-26T01:53:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 --2- 172.21.15.28:0/2683031914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c108f20 0x7f908c1153e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 --2- 172.21.15.28:0/2683031914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f908c108560 0x7f908c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 --2- 172.21.15.28:0/2683031914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c10e620 0x7f908c10ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 -- 172.21.15.28:0/2683031914 >> 172.21.15.28:0/2683031914 conn(0x7f908c0764e0 msgr2=0x7f908c078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 -- 172.21.15.28:0/2683031914 shutdown_connections 2024-04-26T01:53:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.858+0000 7f9091a00640 1 -- 172.21.15.28:0/2683031914 wait complete. 2024-04-26T01:53:56.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.860+0000 7f9091a00640 1 Processor -- start 2024-04-26T01:53:56.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.860+0000 7f9091a00640 1 -- start start 2024-04-26T01:53:56.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.861+0000 7f9091a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f908c108560 0x7f908c073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.861+0000 7f9091a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c108f20 0x7f908c073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.862+0000 7f9091a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c10e620 0x7f908c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.862+0000 7f9083e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c108f20 0x7f908c073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.862+0000 7f9083e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c108f20 0x7f908c073b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46512/0 (socket says 172.21.15.28:46512) 2024-04-26T01:53:56.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.862+0000 7f908aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f908c108560 0x7f908c073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.862+0000 7f908aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f908c108560 0x7f908c073620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57232/0 (socket says 172.21.15.28:57232) 2024-04-26T01:53:56.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.862+0000 7f9091a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1187b0 con 0x7f908c10e620 2024-04-26T01:53:56.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.862+0000 7f9083e00640 1 -- 172.21.15.28:0/1702276160 learned_addr learned my addr 172.21.15.28:0/1702276160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:56.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.863+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f908c118630 con 0x7f908c108560 2024-04-26T01:53:56.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.863+0000 7f908b400640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c10e620 0x7f908c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.863+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f908c118930 con 0x7f908c108f20 2024-04-26T01:53:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.865+0000 7f908aa00640 1 -- 172.21.15.28:0/1702276160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c108f20 msgr2=0x7f908c073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.865+0000 7f908aa00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c108f20 0x7f908c073b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.865+0000 7f908aa00640 1 -- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c10e620 msgr2=0x7f908c06c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.865+0000 7f908aa00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c10e620 0x7f908c06c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.865+0000 7f908aa00640 1 -- 172.21.15.28:0/1702276160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f908c06d100 con 0x7f908c108560 2024-04-26T01:53:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.866+0000 7f9083e00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c108f20 0x7f908c073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.866+0000 7f908b400640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c10e620 0x7f908c06c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.866+0000 7f908aa00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f908c108560 0x7f908c073620 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f907c00e9f0 tx=0x7f907c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.867+0000 7f9083400640 1 -- 172.21.15.28:0/1702276160 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c00ccc0 con 0x7f908c108560 2024-04-26T01:53:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.867+0000 7f9083400640 1 -- 172.21.15.28:0/1702276160 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f907c005020 con 0x7f908c108560 2024-04-26T01:53:56.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.867+0000 7f9083400640 1 -- 172.21.15.28:0/1702276160 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c004a00 con 0x7f908c108560 2024-04-26T01:53:56.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.871+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f908c06d3f0 con 0x7f908c108560 2024-04-26T01:53:56.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.871+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f908c071320 con 0x7f908c108560 2024-04-26T01:53:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.877+0000 7f9094854740 1 Processor -- start 2024-04-26T01:53:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.877+0000 7f9083400640 1 -- 172.21.15.28:0/1702276160 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f907c004ba0 con 0x7f908c108560 2024-04-26T01:53:56.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.879+0000 7f9094854740 1 -- start start 2024-04-26T01:53:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.879+0000 7f9094854740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3ce820 0x5625de3cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.879+0000 7f9094854740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3cf140 0x5625de3d94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.879+0000 7f9094854740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3dbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.879+0000 7f9094854740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625de334900 con 0x5625de3ce820 2024-04-26T01:53:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.879+0000 7f9094854740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5625de2daef0 con 0x5625de3d9a00 2024-04-26T01:53:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.879+0000 7f9094854740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5625de3308c0 con 0x5625de3cf140 2024-04-26T01:53:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.879+0000 7f9083400640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f90700a3f10 0x7f90700a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.880+0000 7f9083400640 1 -- 172.21.15.28:0/1702276160 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7f907c0d0ca0 con 0x7f908c108560 2024-04-26T01:53:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.880+0000 7f908b400640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f90700a3f10 0x7f90700a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.880+0000 7f908b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3ce820 0x5625de3cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.880+0000 7f908b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3ce820 0x5625de3cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57104/0 (socket says 172.21.15.28:57104) 2024-04-26T01:53:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.880+0000 7f908b400640 1 -- 172.21.15.28:0/3049433802 learned_addr learned my addr 172.21.15.28:0/3049433802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.880+0000 7f9083e00640 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3dbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.881+0000 7f908aa00640 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3cf140 0x5625de3d94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.882+0000 7f9083e00640 1 -- 172.21.15.28:0/3049433802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3cf140 msgr2=0x5625de3d94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.882+0000 7f9083e00640 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3cf140 0x5625de3d94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.882+0000 7f9083e00640 1 -- 172.21.15.28:0/3049433802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3ce820 msgr2=0x5625de3cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.882+0000 7f9083e00640 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3ce820 0x5625de3cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.882+0000 7f9083e00640 1 -- 172.21.15.28:0/3049433802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625de047210 con 0x5625de3d9a00 2024-04-26T01:53:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.882+0000 7f9083e00640 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3dbdf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f908c06def0 tx=0x7f90740119a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca535532a0b2d81f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.883+0000 7f907be00640 1 -- 172.21.15.28:0/3049433802 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907401e070 con 0x5625de3d9a00 2024-04-26T01:53:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.883+0000 7f907be00640 1 -- 172.21.15.28:0/3049433802 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90740035f0 con 0x5625de3d9a00 2024-04-26T01:53:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.883+0000 7f907be00640 1 -- 172.21.15.28:0/3049433802 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907400f7c0 con 0x5625de3d9a00 2024-04-26T01:53:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.885+0000 7f9094854740 1 -- 172.21.15.28:0/3049433802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 msgr2=0x5625de3dbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.885+0000 7f908b400640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f90700a3f10 0x7f90700a63d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9084005f00 tx=0x7f9084005e30 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.885+0000 7f9094854740 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3dbdf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f908c06def0 tx=0x7f90740119a0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.886+0000 7f9094854740 1 -- 172.21.15.28:0/3049433802 shutdown_connections 2024-04-26T01:53:56.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.886+0000 7f9094854740 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3dbdf0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.886+0000 7f9094854740 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3cf140 0x5625de3d94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.886+0000 7f9094854740 1 --2- 172.21.15.28:0/3049433802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3ce820 0x5625de3cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.886+0000 7f9094854740 1 -- 172.21.15.28:0/3049433802 >> 172.21.15.28:0/3049433802 conn(0x5625de3cad30 msgr2=0x5625de3cb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:56.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.888+0000 7f9094854740 1 -- 172.21.15.28:0/3049433802 shutdown_connections 2024-04-26T01:53:56.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.888+0000 7f9094854740 1 -- 172.21.15.28:0/3049433802 wait complete. 2024-04-26T01:53:56.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.889+0000 7f9094854740 1 -- start start 2024-04-26T01:53:56.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.889+0000 7f9094854740 1 Processor -- start 2024-04-26T01:53:56.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.890+0000 7f9094854740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3d6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.890+0000 7f9094854740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.890+0000 7f9094854740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3d7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.890+0000 7f9094854740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625de3dc630 con 0x5625de3cf140 2024-04-26T01:53:56.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.890+0000 7f9094854740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5625de2ee310 con 0x5625de3d9a00 2024-04-26T01:53:56.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.890+0000 7f9094854740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5625de3dc7b0 con 0x5625de3ce820 2024-04-26T01:53:56.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.890+0000 7f9094854740 10 client.0 _ll_register_callbacks cb 0x5625de2bccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.891+0000 7f908aa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3d6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.891+0000 7f908aa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3d6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46526/0 (socket says 172.21.15.28:46526) 2024-04-26T01:53:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.891+0000 7f908aa00640 1 -- 172.21.15.28:0/3507822562 learned_addr learned my addr 172.21.15.28:0/3507822562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.891+0000 7f908b400640 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3d7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.891+0000 7f9083e00640 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.892+0000 7f908aa00640 1 -- 172.21.15.28:0/3507822562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 msgr2=0x5625de3d7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.892+0000 7f908aa00640 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3d7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.892+0000 7f908aa00640 1 -- 172.21.15.28:0/3507822562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 msgr2=0x5625de3d6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.892+0000 7f908aa00640 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.892+0000 7f908aa00640 1 -- 172.21.15.28:0/3507822562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625de113620 con 0x5625de3ce820 2024-04-26T01:53:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.892+0000 7f9083e00640 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.892+0000 7f908aa00640 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3d6980 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f907c00c6a0 tx=0x7f907c09d420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90a3172543ff413f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.892+0000 7f9078c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:53:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.893+0000 7f9078c00640 1 -- 172.21.15.28:0/3507822562 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0b5020 con 0x5625de3ce820 2024-04-26T01:53:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.893+0000 7f9078c00640 1 -- 172.21.15.28:0/3507822562 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7f907c0bd070 con 0x5625de3ce820 2024-04-26T01:53:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.893+0000 7f9094854740 1 -- 172.21.15.28:0/3507822562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90740035f0 con 0x5625de3ce820 2024-04-26T01:53:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.894+0000 7f9078c00640 1 -- 172.21.15.28:0/3507822562 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f907c0ce610 con 0x5625de3ce820 2024-04-26T01:53:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.895+0000 7f9094854740 10 client.26305 fetch_fsmap learned FSMap version 46 2024-04-26T01:53:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.895+0000 7f9094854740 1 -- 172.21.15.28:0/3507822562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5625de3e88d0 con 0x5625de3ce820 2024-04-26T01:53:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.895+0000 7f9078c00640 1 -- 172.21.15.28:0/3507822562 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90740035f0 con 0x5625de3ce820 2024-04-26T01:53:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.896+0000 7f9078c00640 1 -- 172.21.15.28:0/3507822562 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f907c01f070 con 0x5625de3ce820 2024-04-26T01:53:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.896+0000 7f9094854740 10 client.26305 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:53:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.896+0000 7f9094854740 10 client.26305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.896+0000 7f9094854740 20 client.26305 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.896+0000 7f9094854740 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5625de3e9d40 0x5625de3ec130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.896+0000 7f9094854740 4 client.26305 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.897+0000 7f9094854740 1 -- 172.21.15.28:0/3507822562 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625de3488b0 con 0x5625de3e9d40 2024-04-26T01:53:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.897+0000 7f9083e00640 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5625de3e9d40 0x5625de3ec130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.898+0000 7f9078c00640 10 client.26305 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.899+0000 7f9083e00640 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5625de3e9d40 0x5625de3ec130 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d171da4a4da0a05d server_cookie=b08274acc76d0341 in_seq=0 out_seq=0 2024-04-26T01:53:56.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.899+0000 7f9078c00640 1 -- 172.21.15.28:0/3507822562 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625de3488b0 con 0x5625de3e9d40 2024-04-26T01:53:56.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9078c00640 10 client.26305 handle_command_reply: tid=0 2024-04-26T01:53:56.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 client.26305 shutdown 2024-04-26T01:53:56.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 -- 172.21.15.28:0/3507822562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 msgr2=0x5625de3d6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3d6980 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f907c00c6a0 tx=0x7f907c09d420 comp rx=0 tx=0).stop 2024-04-26T01:53:56.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 -- 172.21.15.28:0/3507822562 shutdown_connections 2024-04-26T01:53:56.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5625de3e9d40 0x5625de3ec130 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3d7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 --2- 172.21.15.28:0/3507822562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3d6980 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9078c00640 0 client.26305 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.900+0000 7f9094854740 1 -- 172.21.15.28:0/3507822562 >> 172.21.15.28:0/3507822562 conn(0x5625de3cad30 msgr2=0x5625de3d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.901+0000 7f9094854740 1 -- 172.21.15.28:0/3507822562 shutdown_connections 2024-04-26T01:53:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.901+0000 7f9094854740 1 -- 172.21.15.28:0/3507822562 wait complete. 2024-04-26T01:53:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.901+0000 7f9094854740 20 client.26305 trim_cache size 0 max 16384 2024-04-26T01:53:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.936+0000 7f9094854740 1 Processor -- start 2024-04-26T01:53:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.936+0000 7f9094854740 1 -- start start 2024-04-26T01:53:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f9094854740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f9094854740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f9094854740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f9094854740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1bf390 con 0x5625de3cf140 2024-04-26T01:53:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f9094854740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1bcc00 con 0x5625de3d9a00 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f9094854740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1baa70 con 0x5625de3ce820 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908aa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908aa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46528/0 (socket says 172.21.15.28:46528) 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908aa00640 1 -- 172.21.15.28:0/3975761838 learned_addr learned my addr 172.21.15.28:0/3975761838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908b400640 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f9083e00640 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908aa00640 1 -- 172.21.15.28:0/3975761838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 msgr2=0x5625de3f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908aa00640 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908aa00640 1 -- 172.21.15.28:0/3975761838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 msgr2=0x5625de3d11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908aa00640 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.937+0000 7f908aa00640 1 -- 172.21.15.28:0/3975761838 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625de3cbf60 con 0x5625de3ce820 2024-04-26T01:53:56.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.938+0000 7f908aa00640 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e54f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f907c00e9c0 tx=0x7f907c0a59d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7c46defc30cb321 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:56.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.939+0000 7f907b400640 1 -- 172.21.15.28:0/3975761838 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0b5020 con 0x5625de3ce820 2024-04-26T01:53:56.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.939+0000 7f907b400640 1 -- 172.21.15.28:0/3975761838 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f907c0bd030 con 0x5625de3ce820 2024-04-26T01:53:56.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.940+0000 7f907b400640 1 -- 172.21.15.28:0/3975761838 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0b5020 con 0x5625de3ce820 2024-04-26T01:53:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 -- 172.21.15.28:0/3975761838 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 msgr2=0x5625de3e54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e54f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f907c00e9c0 tx=0x7f907c0a59d0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 -- 172.21.15.28:0/3975761838 shutdown_connections 2024-04-26T01:53:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3d11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 --2- 172.21.15.28:0/3975761838 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e54f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 -- 172.21.15.28:0/3975761838 >> 172.21.15.28:0/3975761838 conn(0x5625de3cad30 msgr2=0x5625de3c9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 -- 172.21.15.28:0/3975761838 shutdown_connections 2024-04-26T01:53:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.941+0000 7f9094854740 1 -- 172.21.15.28:0/3975761838 wait complete. 2024-04-26T01:53:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.942+0000 7f9094854740 1 -- start start 2024-04-26T01:53:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.942+0000 7f9094854740 1 Processor -- start 2024-04-26T01:53:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.942+0000 7f9094854740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.942+0000 7f9094854740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3e1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f9094854740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3deed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f9094854740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1bb0c0 con 0x5625de3cf140 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f9094854740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1bc530 con 0x5625de3d9a00 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f9094854740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1bb790 con 0x5625de3ce820 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f9094854740 10 client.0 _ll_register_callbacks cb 0x7f907c09c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908aa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908aa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46544/0 (socket says 172.21.15.28:46544) 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908aa00640 1 -- 172.21.15.28:0/1181730641 learned_addr learned my addr 172.21.15.28:0/1181730641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:53:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f9083e00640 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3e1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908b400640 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3deed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908aa00640 1 -- 172.21.15.28:0/1181730641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 msgr2=0x5625de3deed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908aa00640 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3deed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908aa00640 1 -- 172.21.15.28:0/1181730641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 msgr2=0x5625de3e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908aa00640 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3e1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.943+0000 7f908aa00640 1 -- 172.21.15.28:0/1181730641 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625de3df700 con 0x5625de3ce820 2024-04-26T01:53:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.944+0000 7f908aa00640 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e1950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f907c099040 tx=0x7f907c005460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1490c6b9078a7339 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.944+0000 7f9078c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:53:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.944+0000 7f9078c00640 1 -- 172.21.15.28:0/1181730641 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0b5070 con 0x5625de3ce820 2024-04-26T01:53:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.944+0000 7f9078c00640 1 -- 172.21.15.28:0/1181730641 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8513+0+0 (secure 0 0 0) 0x7f907c09c030 con 0x5625de3ce820 2024-04-26T01:53:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.944+0000 7f9094854740 1 -- 172.21.15.28:0/1181730641 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f907c0bd030 con 0x5625de3ce820 2024-04-26T01:53:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.945+0000 7f9078c00640 1 -- 172.21.15.28:0/1181730641 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f907c0a0f90 con 0x5625de3ce820 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.945+0000 7f9094854740 10 client.26311 fetch_fsmap learned FSMap version 46 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.945+0000 7f9094854740 1 -- 172.21.15.28:0/1181730641 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5625de3df700 con 0x5625de3ce820 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.945+0000 7f9078c00640 1 -- 172.21.15.28:0/1181730641 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f907c0bd030 con 0x5625de3ce820 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.945+0000 7f9078c00640 1 -- 172.21.15.28:0/1181730641 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f907c082070 con 0x5625de3ce820 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.946+0000 7f9094854740 10 client.26311 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.946+0000 7f9094854740 10 client.26311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.946+0000 7f9094854740 20 client.26311 populate_metadata read hostname 'smithi028' 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.946+0000 7f9094854740 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5625de4501b0 0x5625de4525d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.946+0000 7f9094854740 4 client.26311 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.946+0000 7f9094854740 1 -- 172.21.15.28:0/1181730641 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625de342320 con 0x5625de4501b0 2024-04-26T01:53:56.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.946+0000 7f9083e00640 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5625de4501b0 0x5625de4525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:56.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.947+0000 7f9083e00640 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5625de4501b0 0x5625de4525d0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=545e6ba4b4626d3 server_cookie=d4e1f665aa6a94b1 in_seq=0 out_seq=0 2024-04-26T01:53:56.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.947+0000 7f9078c00640 10 client.26311 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.972+0000 7f9078c00640 1 -- 172.21.15.28:0/1181730641 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5625de342320 con 0x5625de4501b0 2024-04-26T01:53:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.972+0000 7f9078c00640 10 client.26311 handle_command_reply: tid=0 2024-04-26T01:53:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.972+0000 7f9094854740 1 client.26311 shutdown 2024-04-26T01:53:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 -- 172.21.15.28:0/1181730641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 msgr2=0x5625de3e1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e1950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f907c099040 tx=0x7f907c005460 comp rx=0 tx=0).stop 2024-04-26T01:53:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 -- 172.21.15.28:0/1181730641 shutdown_connections 2024-04-26T01:53:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5625de4501b0 0x5625de4525d0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5625de3d9a00 0x5625de3deed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625de3cf140 0x5625de3e1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 --2- 172.21.15.28:0/1181730641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5625de3ce820 0x5625de3e1950 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 -- 172.21.15.28:0/1181730641 >> 172.21.15.28:0/1181730641 conn(0x5625de3cad30 msgr2=0x5625de3c9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9078c00640 0 client.26311 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:53:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.973+0000 7f9094854740 1 -- 172.21.15.28:0/1181730641 shutdown_connections 2024-04-26T01:53:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.974+0000 7f9094854740 1 -- 172.21.15.28:0/1181730641 wait complete. 2024-04-26T01:53:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.974+0000 7f9094854740 20 client.26311 trim_cache size 0 max 16384 2024-04-26T01:53:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:53:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:53:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:53:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:53:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.975+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f90700a3f10 msgr2=0x7f90700a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f90700a3f10 0x7f90700a63d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9084005f00 tx=0x7f9084005e30 comp rx=0 tx=0).stop 2024-04-26T01:53:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f908c108560 msgr2=0x7f908c073620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f908c108560 0x7f908c073620 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f907c00e9f0 tx=0x7f907c00eeb0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 shutdown_connections 2024-04-26T01:53:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f90700a3f10 0x7f90700a63d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f908c10e620 0x7f908c06c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c108f20 0x7f908c073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 --2- 172.21.15.28:0/1702276160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f908c108560 0x7f908c073620 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.976+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 >> 172.21.15.28:0/1702276160 conn(0x7f908c0764e0 msgr2=0x7f908c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.977+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 shutdown_connections 2024-04-26T01:53:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:53:56.977+0000 7f9091a00640 1 -- 172.21.15.28:0/1702276160 wait complete. 2024-04-26T01:53:57.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:57 smithi169 ceph-mon[32851]: pgmap v406: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-04-26T01:53:57.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:57 smithi053 ceph-mon[32593]: pgmap v406: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-04-26T01:53:58.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:57 smithi028 ceph-mon[28247]: pgmap v406: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-04-26T01:53:59.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:53:59 smithi169 ceph-mon[32851]: pgmap v407: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.6 MiB/s wr, 576 op/s 2024-04-26T01:53:59.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:53:59 smithi053 ceph-mon[32593]: pgmap v407: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.6 MiB/s wr, 576 op/s 2024-04-26T01:54:00.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:53:59 smithi028 ceph-mon[28247]: pgmap v407: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.6 MiB/s wr, 576 op/s 2024-04-26T01:54:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 725.0 s, 1944.0 tps, lat 49.806 ms stddev 62.658 2024-04-26T01:54:01.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:01 smithi169 ceph-mon[32851]: pgmap v408: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.2 MiB/s wr, 874 op/s 2024-04-26T01:54:01.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:01 smithi053 ceph-mon[32593]: pgmap v408: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.2 MiB/s wr, 874 op/s 2024-04-26T01:54:02.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:01 smithi028 ceph-mon[28247]: pgmap v408: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.2 MiB/s wr, 874 op/s 2024-04-26T01:54:03.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:03 smithi169 ceph-mon[32851]: pgmap v409: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.3 MiB/s wr, 564 op/s 2024-04-26T01:54:03.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:03 smithi169 ceph-mon[32851]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:54:03.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:03 smithi053 ceph-mon[32593]: pgmap v409: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.3 MiB/s wr, 564 op/s 2024-04-26T01:54:03.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:03 smithi053 ceph-mon[32593]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:54:04.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:03 smithi028 ceph-mon[28247]: pgmap v409: 97 pgs: 97 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.3 MiB/s wr, 564 op/s 2024-04-26T01:54:04.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:03 smithi028 ceph-mon[28247]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:54:05.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:05 smithi028 ceph-mon[28247]: pgmap v411: 97 pgs: 97 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.3 MiB/s wr, 660 op/s 2024-04-26T01:54:05.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:05 smithi169 ceph-mon[32851]: pgmap v411: 97 pgs: 97 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.3 MiB/s wr, 660 op/s 2024-04-26T01:54:05.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:05 smithi053 ceph-mon[32593]: pgmap v411: 97 pgs: 97 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.3 MiB/s wr, 660 op/s 2024-04-26T01:54:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 730.0 s, 304.8 tps, lat 332.669 ms stddev 200.765 2024-04-26T01:54:07.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:07 smithi169 ceph-mon[32851]: pgmap v412: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 556 op/s 2024-04-26T01:54:07.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:07 smithi053 ceph-mon[32593]: pgmap v412: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 556 op/s 2024-04-26T01:54:08.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:07 smithi028 ceph-mon[28247]: pgmap v412: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 556 op/s 2024-04-26T01:54:09.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:09 smithi169 ceph-mon[32851]: pgmap v413: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 556 op/s 2024-04-26T01:54:09.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:09 smithi169 ceph-mon[32851]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:54:09.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:09 smithi053 ceph-mon[32593]: pgmap v413: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 556 op/s 2024-04-26T01:54:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:09 smithi053 ceph-mon[32593]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:54:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:09 smithi028 ceph-mon[28247]: pgmap v413: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 556 op/s 2024-04-26T01:54:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:09 smithi028 ceph-mon[28247]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:54:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.188+0000 7f63f4a00640 1 -- 172.21.15.28:0/399244376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0108f50 msgr2=0x7f63f0109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.188+0000 7f63f4a00640 1 --2- 172.21.15.28:0/399244376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0108f50 0x7f63f0109330 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009890 tx=0x7f63e002a410 comp rx=0 tx=0).stop 2024-04-26T01:54:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.188+0000 7f63f4a00640 1 -- 172.21.15.28:0/399244376 shutdown_connections 2024-04-26T01:54:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.188+0000 7f63f4a00640 1 --2- 172.21.15.28:0/399244376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63f01150a0 0x7f63f0117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.188+0000 7f63f4a00640 1 --2- 172.21.15.28:0/399244376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63f0109870 0x7f63f0114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.188+0000 7f63f4a00640 1 --2- 172.21.15.28:0/399244376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0108f50 0x7f63f0109330 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.188+0000 7f63f4a00640 1 -- 172.21.15.28:0/399244376 >> 172.21.15.28:0/399244376 conn(0x7f63f006d020 msgr2=0x7f63f006d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.189+0000 7f63f4a00640 1 -- 172.21.15.28:0/399244376 shutdown_connections 2024-04-26T01:54:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.189+0000 7f63f4a00640 1 -- 172.21.15.28:0/399244376 wait complete. 2024-04-26T01:54:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.190+0000 7f63f4a00640 1 Processor -- start 2024-04-26T01:54:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.191+0000 7f63f4a00640 1 -- start start 2024-04-26T01:54:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.193+0000 7f63f4a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63f0108f50 0x7f63f010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.193+0000 7f63f4a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0109870 0x7f63f010af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.193+0000 7f63f4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63f01150a0 0x7f63f010b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.193+0000 7f63f4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63f011aa20 con 0x7f63f01150a0 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.193+0000 7f63f4a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f63f011a8a0 con 0x7f63f0108f50 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.193+0000 7f63f4a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f63f011aba0 con 0x7f63f0109870 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.193+0000 7f63ed400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0109870 0x7f63f010af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.194+0000 7f63ed400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0109870 0x7f63f010af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:39132/0 (socket says 172.21.15.28:39132) 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.194+0000 7f63ed400640 1 -- 172.21.15.28:0/1302297808 learned_addr learned my addr 172.21.15.28:0/1302297808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.194+0000 7f63ee800640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63f01150a0 0x7f63f010b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.194+0000 7f63ed400640 1 -- 172.21.15.28:0/1302297808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63f0108f50 msgr2=0x7f63f010a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.194+0000 7f63ed400640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63f0108f50 0x7f63f010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.194+0000 7f63ed400640 1 -- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63f01150a0 msgr2=0x7f63f010b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.194+0000 7f63ed400640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63f01150a0 0x7f63f010b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.194+0000 7f63ed400640 1 -- 172.21.15.28:0/1302297808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f01af400 con 0x7f63f0109870 2024-04-26T01:54:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.195+0000 7f63ed400640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0109870 0x7f63f010af00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f63dc00d950 tx=0x7f63dc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.195+0000 7f63e6a00640 1 -- 172.21.15.28:0/1302297808 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63dc017070 con 0x7f63f0109870 2024-04-26T01:54:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.196+0000 7f63e6a00640 1 -- 172.21.15.28:0/1302297808 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63dc009c30 con 0x7f63f0109870 2024-04-26T01:54:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.196+0000 7f63e6a00640 1 -- 172.21.15.28:0/1302297808 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63dc00f620 con 0x7f63f0109870 2024-04-26T01:54:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.197+0000 7f63f4a00640 1 -- 172.21.15.28:0/1302297808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63f01af6f0 con 0x7f63f0109870 2024-04-26T01:54:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.197+0000 7f63f4a00640 1 -- 172.21.15.28:0/1302297808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63f01afbf0 con 0x7f63f0109870 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.214+0000 7f63e6a00640 1 -- 172.21.15.28:0/1302297808 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f63dc010460 con 0x7f63f0109870 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.215+0000 7f63f79ca740 1 Processor -- start 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.216+0000 7f63f79ca740 1 -- start start 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.216+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6df4ef0 0x55afc6df52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.216+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6df0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.216+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6e08c80 0x55afc6e0b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.217+0000 7f63f79ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55afc6d52900 con 0x55afc6df4ef0 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.217+0000 7f63ed400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6e08c80 0x55afc6e0b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.217+0000 7f63f79ca740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55afc6cf8ef0 con 0x55afc6df5810 2024-04-26T01:54:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.217+0000 7f63f79ca740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55afc6d4e8c0 con 0x55afc6e08c80 2024-04-26T01:54:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.217+0000 7f63ed400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6e08c80 0x55afc6e0b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:39140/0 (socket says 172.21.15.28:39140) 2024-04-26T01:54:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.217+0000 7f63ede00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6df0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.217+0000 7f63ed400640 1 -- 172.21.15.28:0/1755163208 learned_addr learned my addr 172.21.15.28:0/1755163208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.218+0000 7f63ede00640 1 -- 172.21.15.28:0/1755163208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6e08c80 msgr2=0x55afc6e0b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.218+0000 7f63ede00640 1 --2- 172.21.15.28:0/1755163208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6e08c80 0x55afc6e0b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.218+0000 7f63ede00640 1 -- 172.21.15.28:0/1755163208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6df4ef0 msgr2=0x55afc6df52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.218+0000 7f63e6a00640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f63d40a3e40 0x7f63d40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.217+0000 7f63ee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6df4ef0 0x55afc6df52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.220+0000 7f63ede00640 1 --2- 172.21.15.28:0/1755163208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6df4ef0 0x55afc6df52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.220+0000 7f63ede00640 1 -- 172.21.15.28:0/1755163208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afc6a65210 con 0x55afc6df5810 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.220+0000 7f63ee800640 1 --2- 172.21.15.28:0/1755163208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6df4ef0 0x55afc6df52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:54:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.220+0000 7f63ede00640 1 --2- 172.21.15.28:0/1755163208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6df0740 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009890 tx=0x7f63e0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10d89c43b69408f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.221+0000 7f63db000640 1 -- 172.21.15.28:0/1755163208 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e0004020 con 0x55afc6df5810 2024-04-26T01:54:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.221+0000 7f63ed400640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f63d40a3e40 0x7f63d40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.221+0000 7f63e6a00640 1 -- 172.21.15.28:0/1302297808 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f63dc0cfdc0 con 0x7f63f0109870 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.224+0000 7f63db000640 1 -- 172.21.15.28:0/1755163208 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e00309c0 con 0x55afc6df5810 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.225+0000 7f63f79ca740 1 -- 172.21.15.28:0/1755163208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 msgr2=0x55afc6df0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.225+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1755163208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6df0740 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009890 tx=0x7f63e0005dc0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.225+0000 7f63ed400640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f63d40a3e40 0x7f63d40a6300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f63dc098730 tx=0x7f63dc000bc0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.224+0000 7f63db000640 1 -- 172.21.15.28:0/1755163208 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e0005570 con 0x55afc6df5810 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.225+0000 7f63f79ca740 1 -- 172.21.15.28:0/1755163208 shutdown_connections 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.225+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1755163208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6e08c80 0x55afc6e0b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.225+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1755163208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6df0740 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.225+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1755163208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6df4ef0 0x55afc6df52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.226+0000 7f63f79ca740 1 -- 172.21.15.28:0/1755163208 >> 172.21.15.28:0/1755163208 conn(0x55afc6df1540 msgr2=0x55afc6df1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.226+0000 7f63f79ca740 1 -- 172.21.15.28:0/1755163208 shutdown_connections 2024-04-26T01:54:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.226+0000 7f63f79ca740 1 -- 172.21.15.28:0/1755163208 wait complete. 2024-04-26T01:54:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.227+0000 7f63f79ca740 1 -- start start 2024-04-26T01:54:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.227+0000 7f63f79ca740 1 Processor -- start 2024-04-26T01:54:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.229+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6dedbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.229+0000 7f63ee800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6dedbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.229+0000 7f63ee800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6dedbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:39150/0 (socket says 172.21.15.28:39150) 2024-04-26T01:54:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.229+0000 7f63ee800640 1 -- 172.21.15.28:0/4287388542 learned_addr learned my addr 172.21.15.28:0/4287388542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.229+0000 7f63f79ca740 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6dee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.230+0000 7f63f79ca740 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6dee630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.230+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55afc6e0b610 con 0x55afc6e08c80 2024-04-26T01:54:10.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.230+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55afc6d0c310 con 0x55afc6df5810 2024-04-26T01:54:10.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.230+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55afc6e0b790 con 0x55afc6df4ef0 2024-04-26T01:54:10.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.230+0000 7f63ede00640 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6dee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.230+0000 7f63f79ca740 10 client.0 _ll_register_callbacks cb 0x55afc6cdacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:10.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.231+0000 7f63ede00640 1 -- 172.21.15.28:0/4287388542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 msgr2=0x55afc6dedbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.231+0000 7f63ede00640 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6dedbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.231+0000 7f63ede00640 1 -- 172.21.15.28:0/4287388542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 msgr2=0x55afc6dee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.231+0000 7f63ede00640 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6dee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.231+0000 7f63ede00640 1 -- 172.21.15.28:0/4287388542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afc6b31620 con 0x55afc6df5810 2024-04-26T01:54:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.233+0000 7f63ede00640 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6dee0f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63e0000c00 tx=0x7f63e0030d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb94463998cfcef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.233+0000 7f63cf400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:54:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.234+0000 7f63cf400640 1 -- 172.21.15.28:0/4287388542 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e0053060 con 0x55afc6df5810 2024-04-26T01:54:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.234+0000 7f63cf400640 1 -- 172.21.15.28:0/4287388542 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f63e0005500 con 0x55afc6df5810 2024-04-26T01:54:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.234+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63e00309c0 con 0x55afc6df5810 2024-04-26T01:54:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.235+0000 7f63cf400640 1 -- 172.21.15.28:0/4287388542 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f63e00046c0 con 0x55afc6df5810 2024-04-26T01:54:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.235+0000 7f63f79ca740 10 client.26306 fetch_fsmap learned FSMap version 46 2024-04-26T01:54:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.235+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55afc6e17820 con 0x55afc6df5810 2024-04-26T01:54:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.236+0000 7f63cf400640 1 -- 172.21.15.28:0/4287388542 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e00309c0 con 0x55afc6df5810 2024-04-26T01:54:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.236+0000 7f63cf400640 1 -- 172.21.15.28:0/4287388542 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f63e003c700 con 0x55afc6df5810 2024-04-26T01:54:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.237+0000 7f63f79ca740 10 client.26306 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:54:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.237+0000 7f63f79ca740 10 client.26306 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.237+0000 7f63f79ca740 20 client.26306 populate_metadata read hostname 'smithi028' 2024-04-26T01:54:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.237+0000 7f63f79ca740 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55afc6e18b70 0x55afc6e1af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.237+0000 7f63f79ca740 4 client.26306 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.237+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afc6d668b0 con 0x55afc6e18b70 2024-04-26T01:54:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.237+0000 7f63ee800640 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55afc6e18b70 0x55afc6e1af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.238+0000 7f63ee800640 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55afc6e18b70 0x55afc6e1af60 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=827f87397b11d486 server_cookie=bc15e19890c74f59 in_seq=0 out_seq=0 2024-04-26T01:54:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.238+0000 7f63cf400640 10 client.26306 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.239+0000 7f63cf400640 1 -- 172.21.15.28:0/4287388542 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55afc6d668b0 con 0x55afc6e18b70 2024-04-26T01:54:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.239+0000 7f63cf400640 10 client.26306 handle_command_reply: tid=0 2024-04-26T01:54:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.240+0000 7f63f79ca740 1 client.26306 shutdown 2024-04-26T01:54:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.240+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 msgr2=0x55afc6dee0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.240+0000 7f63f79ca740 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6dee0f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63e0000c00 tx=0x7f63e0030d80 comp rx=0 tx=0).stop 2024-04-26T01:54:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.241+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 shutdown_connections 2024-04-26T01:54:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.241+0000 7f63f79ca740 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55afc6e18b70 0x55afc6e1af60 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.241+0000 7f63f79ca740 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6dee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.241+0000 7f63f79ca740 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df5810 0x55afc6dee0f0 secure :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63e0000c00 tx=0x7f63e0030d80 comp rx=0 tx=0).stop 2024-04-26T01:54:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.241+0000 7f63f79ca740 1 --2- 172.21.15.28:0/4287388542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6dedbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.241+0000 7f63cf400640 0 client.26306 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.241+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 >> 172.21.15.28:0/4287388542 conn(0x55afc6df1540 msgr2=0x55afc6df72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.241+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 shutdown_connections 2024-04-26T01:54:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.242+0000 7f63f79ca740 1 -- 172.21.15.28:0/4287388542 wait complete. 2024-04-26T01:54:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.242+0000 7f63f79ca740 20 client.26306 trim_cache size 0 max 16384 2024-04-26T01:54:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.283+0000 7f63f79ca740 1 Processor -- start 2024-04-26T01:54:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.283+0000 7f63f79ca740 1 -- start start 2024-04-26T01:54:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.283+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e14550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.283+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6de7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.284+0000 7f63ee800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e14550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.284+0000 7f63ee800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e14550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:39154/0 (socket says 172.21.15.28:39154) 2024-04-26T01:54:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.284+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e20680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.284+0000 7f63f79ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63f01b6a70 con 0x55afc6e08c80 2024-04-26T01:54:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.284+0000 7f63f79ca740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f63f01b42e0 con 0x55afc6df03f0 2024-04-26T01:54:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.284+0000 7f63f79ca740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f63f01b2150 con 0x55afc6df4ef0 2024-04-26T01:54:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.285+0000 7f63ee800640 1 -- 172.21.15.28:0/3231012392 learned_addr learned my addr 172.21.15.28:0/3231012392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.285+0000 7f63ee800640 1 --2- 172.21.15.28:0/3231012392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e20680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.286+0000 7f63ee800640 1 -- 172.21.15.28:0/3231012392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 msgr2=0x55afc6e20680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.286+0000 7f63ee800640 1 --2- 172.21.15.28:0/3231012392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e20680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.286+0000 7f63ee800640 1 -- 172.21.15.28:0/3231012392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 msgr2=0x55afc6de7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.286+0000 7f63ee800640 1 --2- 172.21.15.28:0/3231012392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6de7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.286+0000 7f63ee800640 1 -- 172.21.15.28:0/3231012392 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afc6df1360 con 0x55afc6df4ef0 2024-04-26T01:54:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.287+0000 7f63ee800640 1 --2- 172.21.15.28:0/3231012392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e14550 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009d40 tx=0x7f63e8009e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc7c44deb33bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.287+0000 7f63da600640 1 -- 172.21.15.28:0/3231012392 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e800f500 con 0x55afc6df4ef0 2024-04-26T01:54:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.287+0000 7f63da600640 1 -- 172.21.15.28:0/3231012392 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e800f500 con 0x55afc6df4ef0 2024-04-26T01:54:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.287+0000 7f63da600640 1 -- 172.21.15.28:0/3231012392 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8005210 con 0x55afc6df4ef0 2024-04-26T01:54:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.288+0000 7f63f79ca740 1 -- 172.21.15.28:0/3231012392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 msgr2=0x55afc6e14550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.288+0000 7f63f79ca740 1 --2- 172.21.15.28:0/3231012392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e14550 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009d40 tx=0x7f63e8009e00 comp rx=0 tx=0).stop 2024-04-26T01:54:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.289+0000 7f63f79ca740 1 -- 172.21.15.28:0/3231012392 shutdown_connections 2024-04-26T01:54:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.289+0000 7f63f79ca740 1 --2- 172.21.15.28:0/3231012392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e20680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.289+0000 7f63f79ca740 1 --2- 172.21.15.28:0/3231012392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6de7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.289+0000 7f63f79ca740 1 --2- 172.21.15.28:0/3231012392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e14550 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.289+0000 7f63f79ca740 1 -- 172.21.15.28:0/3231012392 >> 172.21.15.28:0/3231012392 conn(0x55afc6df1540 msgr2=0x55afc6de7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.289+0000 7f63f79ca740 1 -- 172.21.15.28:0/3231012392 shutdown_connections 2024-04-26T01:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.290+0000 7f63f79ca740 1 -- 172.21.15.28:0/3231012392 wait complete. 2024-04-26T01:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.290+0000 7f63f79ca740 1 -- start start 2024-04-26T01:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.290+0000 7f63f79ca740 1 Processor -- start 2024-04-26T01:54:10.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.291+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e10b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63ee800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e10b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e0e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63f79ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6e0e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63f79ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63f01b27a0 con 0x55afc6e08c80 2024-04-26T01:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63f79ca740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f63f01b3c10 con 0x55afc6df03f0 2024-04-26T01:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63ee800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e10b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:49800/0 (socket says 172.21.15.28:49800) 2024-04-26T01:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63ee800640 1 -- 172.21.15.28:0/1969783191 learned_addr learned my addr 172.21.15.28:0/1969783191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f63f01b2e70 con 0x55afc6df4ef0 2024-04-26T01:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.292+0000 7f63f79ca740 10 client.0 _ll_register_callbacks cb 0x7f63e00094c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.293+0000 7f63ede00640 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e0e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.293+0000 7f63ee800640 1 -- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 msgr2=0x55afc6e0e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.294+0000 7f63ee800640 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e0e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.294+0000 7f63ee800640 1 -- 172.21.15.28:0/1969783191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 msgr2=0x55afc6e0e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.294+0000 7f63ee800640 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6e0e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.294+0000 7f63ee800640 1 -- 172.21.15.28:0/1969783191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afc6e0ead0 con 0x55afc6df03f0 2024-04-26T01:54:10.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.294+0000 7f63ede00640 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e0e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:54:10.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.294+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63e800f500 con 0x55afc6df03f0 2024-04-26T01:54:10.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.295+0000 7f63ee800640 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e10b20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f63e8020040 tx=0x7f63e801d8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f75e9e02ec3905be server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.295+0000 7f63cf400640 10 client.26312 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:54:10.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.295+0000 7f63cf400640 1 -- 172.21.15.28:0/1969783191 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e802f060 con 0x55afc6df03f0 2024-04-26T01:54:10.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.296+0000 7f63cf400640 1 -- 172.21.15.28:0/1969783191 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f63e800e650 con 0x55afc6df03f0 2024-04-26T01:54:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.296+0000 7f63cf400640 1 -- 172.21.15.28:0/1969783191 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f63e800e870 con 0x55afc6df03f0 2024-04-26T01:54:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.296+0000 7f63f79ca740 10 client.26312 fetch_fsmap learned FSMap version 46 2024-04-26T01:54:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.296+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({fsmap=46}) -- 0x55afc6e7eb80 con 0x55afc6df03f0 2024-04-26T01:54:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.297+0000 7f63cf400640 1 -- 172.21.15.28:0/1969783191 <== mon.2 v2:172.21.15.53:3300/0 4 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f63e0030670 con 0x55afc6df03f0 2024-04-26T01:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.297+0000 7f63f79ca740 10 client.26312 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.297+0000 7f63f79ca740 10 client.26312 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.297+0000 7f63f79ca740 20 client.26312 populate_metadata read hostname 'smithi028' 2024-04-26T01:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.297+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55afc6e7f120 0x55afc6e81540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.297+0000 7f63f79ca740 4 client.26312 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.297+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55afc6d60320 con 0x55afc6e7f120 2024-04-26T01:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.297+0000 7f63ede00640 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55afc6e7f120 0x55afc6e81540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.299+0000 7f63ede00640 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55afc6e7f120 0x55afc6e81540 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=255eec39e0525ec0 server_cookie=177b088fbef2835d in_seq=0 out_seq=0 2024-04-26T01:54:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.299+0000 7f63cf400640 10 client.26312 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.316+0000 7f63cf400640 1 -- 172.21.15.28:0/1969783191 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55afc6d60320 con 0x55afc6e7f120 2024-04-26T01:54:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.316+0000 7f63cf400640 10 client.26312 handle_command_reply: tid=0 2024-04-26T01:54:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.316+0000 7f63f79ca740 1 client.26312 shutdown 2024-04-26T01:54:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 msgr2=0x55afc6e10b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e10b20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f63e8020040 tx=0x7f63e801d8b0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 shutdown_connections 2024-04-26T01:54:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55afc6e7f120 0x55afc6e81540 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55afc6e08c80 0x55afc6e0e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afc6df4ef0 0x55afc6e0e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63f79ca740 1 --2- 172.21.15.28:0/1969783191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55afc6df03f0 0x55afc6e10b20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63cf400640 0 client.26312 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.317+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 >> 172.21.15.28:0/1969783191 conn(0x55afc6df1540 msgr2=0x55afc6dec480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.318+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 shutdown_connections 2024-04-26T01:54:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.318+0000 7f63f79ca740 1 -- 172.21.15.28:0/1969783191 wait complete. 2024-04-26T01:54:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.319+0000 7f63f79ca740 20 client.26312 trim_cache size 0 max 16384 2024-04-26T01:54:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:54:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:54:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:54:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:54:10.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.320+0000 7f63f4a00640 1 -- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f63d40a3e40 msgr2=0x7f63d40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.320+0000 7f63f4a00640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f63d40a3e40 0x7f63d40a6300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f63dc098730 tx=0x7f63dc000bc0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.320+0000 7f63f4a00640 1 -- 172.21.15.28:0/1302297808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0109870 msgr2=0x7f63f010af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.320+0000 7f63f4a00640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0109870 0x7f63f010af00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f63dc00d950 tx=0x7f63dc00de10 comp rx=0 tx=0).stop 2024-04-26T01:54:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.321+0000 7f63f4a00640 1 -- 172.21.15.28:0/1302297808 shutdown_connections 2024-04-26T01:54:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.321+0000 7f63f4a00640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f63d40a3e40 0x7f63d40a6300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.321+0000 7f63f4a00640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63f01150a0 0x7f63f010b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.321+0000 7f63f4a00640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f0109870 0x7f63f010af00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.321+0000 7f63f4a00640 1 --2- 172.21.15.28:0/1302297808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63f0108f50 0x7f63f010a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.321+0000 7f63f4a00640 1 -- 172.21.15.28:0/1302297808 >> 172.21.15.28:0/1302297808 conn(0x7f63f006d020 msgr2=0x7f63f0070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.321+0000 7f63f4a00640 1 -- 172.21.15.28:0/1302297808 shutdown_connections 2024-04-26T01:54:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:10.321+0000 7f63f4a00640 1 -- 172.21.15.28:0/1302297808 wait complete. 2024-04-26T01:54:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 735.0 s, 1208.8 tps, lat 84.036 ms stddev 183.179 2024-04-26T01:54:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:11 smithi169 ceph-mon[32851]: pgmap v415: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 MiB/s wr, 1.87k op/s 2024-04-26T01:54:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:11 smithi053 ceph-mon[32593]: pgmap v415: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 MiB/s wr, 1.87k op/s 2024-04-26T01:54:11.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:11 smithi028 ceph-mon[28247]: pgmap v415: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 MiB/s wr, 1.87k op/s 2024-04-26T01:54:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:13 smithi169 ceph-mon[32851]: pgmap v416: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 1.54k op/s 2024-04-26T01:54:13.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:13 smithi053 ceph-mon[32593]: pgmap v416: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 1.54k op/s 2024-04-26T01:54:13.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:13 smithi028 ceph-mon[28247]: pgmap v416: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 1.54k op/s 2024-04-26T01:54:15.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:15 smithi169 ceph-mon[32851]: pgmap v417: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 16 MiB/s wr, 1.51k op/s 2024-04-26T01:54:15.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:15 smithi053 ceph-mon[32593]: pgmap v417: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 16 MiB/s wr, 1.51k op/s 2024-04-26T01:54:15.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:15 smithi028 ceph-mon[28247]: pgmap v417: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 16 MiB/s wr, 1.51k op/s 2024-04-26T01:54:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 740.0 s, 1864.8 tps, lat 54.212 ms stddev 72.762 2024-04-26T01:54:17.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:17 smithi169 ceph-mon[32851]: pgmap v418: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 18 MiB/s wr, 1.62k op/s 2024-04-26T01:54:17.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:17 smithi053 ceph-mon[32593]: pgmap v418: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 18 MiB/s wr, 1.62k op/s 2024-04-26T01:54:17.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:17 smithi028 ceph-mon[28247]: pgmap v418: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 18 MiB/s wr, 1.62k op/s 2024-04-26T01:54:19.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:19 smithi169 ceph-mon[32851]: pgmap v419: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 18 MiB/s wr, 1.62k op/s 2024-04-26T01:54:19.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:19 smithi053 ceph-mon[32593]: pgmap v419: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 18 MiB/s wr, 1.62k op/s 2024-04-26T01:54:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:19 smithi028 ceph-mon[28247]: pgmap v419: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 18 MiB/s wr, 1.62k op/s 2024-04-26T01:54:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 745.0 s, 2404.4 tps, lat 41.255 ms stddev 48.756 2024-04-26T01:54:21.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:21 smithi053 ceph-mon[32593]: pgmap v420: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 1.69k op/s 2024-04-26T01:54:21.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:21 smithi028 ceph-mon[28247]: pgmap v420: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 1.69k op/s 2024-04-26T01:54:21.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:21 smithi169 ceph-mon[32851]: pgmap v420: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 1.69k op/s 2024-04-26T01:54:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.507+0000 7f0e67400640 1 -- 172.21.15.28:0/1176501813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60100d60 msgr2=0x7f0e60101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.507+0000 7f0e67400640 1 --2- 172.21.15.28:0/1176501813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60100d60 0x7f0e60101140 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009900 tx=0x7f0e5402a3f0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.508+0000 7f0e67400640 1 -- 172.21.15.28:0/1176501813 shutdown_connections 2024-04-26T01:54:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.508+0000 7f0e67400640 1 --2- 172.21.15.28:0/1176501813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e6010e910 0x7f0e60110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.508+0000 7f0e67400640 1 --2- 172.21.15.28:0/1176501813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e60101680 0x7f0e6010e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.508+0000 7f0e67400640 1 --2- 172.21.15.28:0/1176501813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60100d60 0x7f0e60101140 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.508+0000 7f0e67400640 1 -- 172.21.15.28:0/1176501813 >> 172.21.15.28:0/1176501813 conn(0x7f0e60077fd0 msgr2=0x7f0e600783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.508+0000 7f0e67400640 1 -- 172.21.15.28:0/1176501813 shutdown_connections 2024-04-26T01:54:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.508+0000 7f0e67400640 1 -- 172.21.15.28:0/1176501813 wait complete. 2024-04-26T01:54:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.510+0000 7f0e67400640 1 Processor -- start 2024-04-26T01:54:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.510+0000 7f0e67400640 1 -- start start 2024-04-26T01:54:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.511+0000 7f0e67400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e60100d60 0x7f0e6010e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.511+0000 7f0e67400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60101680 0x7f0e6010c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.511+0000 7f0e67400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e6010e910 0x7f0e6010c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.511+0000 7f0e67400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e60103580 con 0x7f0e60100d60 2024-04-26T01:54:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.511+0000 7f0e67400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e60103400 con 0x7f0e60101680 2024-04-26T01:54:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.511+0000 7f0e67400640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e60103700 con 0x7f0e6010e910 2024-04-26T01:54:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60101680 0x7f0e6010c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e64a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e60100d60 0x7f0e6010e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60101680 0x7f0e6010c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53176/0 (socket says 172.21.15.28:53176) 2024-04-26T01:54:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3276662965 learned_addr learned my addr 172.21.15.28:0/3276662965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e65400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e6010e910 0x7f0e6010c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3276662965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e6010e910 msgr2=0x7f0e6010c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e6010e910 0x7f0e6010c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e60100d60 msgr2=0x7f0e6010e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e60100d60 0x7f0e6010e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.512+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3276662965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6010cdb0 con 0x7f0e60101680 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.513+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60101680 0x7f0e6010c140 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00b4a0 tx=0x7f0e4c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.513+0000 7f0e5d600640 1 -- 172.21.15.28:0/3276662965 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c014070 con 0x7f0e60101680 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.513+0000 7f0e5d600640 1 -- 172.21.15.28:0/3276662965 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e4c010ac0 con 0x7f0e60101680 2024-04-26T01:54:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.513+0000 7f0e5d600640 1 -- 172.21.15.28:0/3276662965 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c017c30 con 0x7f0e60101680 2024-04-26T01:54:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.513+0000 7f0e67400640 1 -- 172.21.15.28:0/3276662965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e601ab410 con 0x7f0e60101680 2024-04-26T01:54:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.513+0000 7f0e67400640 1 -- 172.21.15.28:0/3276662965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e601ab880 con 0x7f0e60101680 2024-04-26T01:54:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.518+0000 7f0e6a283740 1 Processor -- start 2024-04-26T01:54:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.519+0000 7f0e6a283740 1 -- start start 2024-04-26T01:54:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.519+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c6820 0x5590b11c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.520+0000 7f0e5d600640 1 -- 172.21.15.28:0/3276662965 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0e4c017070 con 0x7f0e60101680 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.520+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11c7140 0x5590b11d14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.520+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11d3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.520+0000 7f0e6a283740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590b112c900 con 0x5590b11c6820 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.520+0000 7f0e6a283740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5590b10d2ef0 con 0x5590b11d1a00 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.520+0000 7f0e6a283740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5590b11288c0 con 0x5590b11c7140 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.520+0000 7f0e65400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c6820 0x5590b11c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.521+0000 7f0e65400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c6820 0x5590b11c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50290/0 (socket says 172.21.15.28:50290) 2024-04-26T01:54:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.521+0000 7f0e65400640 1 -- 172.21.15.28:0/2391391947 learned_addr learned my addr 172.21.15.28:0/2391391947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.522+0000 7f0e5d600640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e480a3e40 0x7f0e480a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.523+0000 7f0e65400640 1 -- 172.21.15.28:0/2391391947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11c7140 msgr2=0x5590b11d14c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:54:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.523+0000 7f0e65400640 1 --2- 172.21.15.28:0/2391391947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11c7140 0x5590b11d14c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.523+0000 7f0e65400640 1 -- 172.21.15.28:0/2391391947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 msgr2=0x5590b11d3df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:54:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.523+0000 7f0e65400640 1 --2- 172.21.15.28:0/2391391947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11d3df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.523+0000 7f0e65400640 1 -- 172.21.15.28:0/2391391947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590b0e3f210 con 0x5590b11c6820 2024-04-26T01:54:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.523+0000 7f0e5d600640 1 -- 172.21.15.28:0/3276662965 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f0e4c0cf120 con 0x7f0e60101680 2024-04-26T01:54:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.527+0000 7f0e65400640 1 --2- 172.21.15.28:0/2391391947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c6820 0x5590b11c6c00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009f80 tx=0x7f0e58010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c1a5fde1ee612c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.528+0000 7f0e65400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e480a3e40 0x7f0e480a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.528+0000 7f0e51c00640 1 -- 172.21.15.28:0/2391391947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e58003b70 con 0x5590b11c6820 2024-04-26T01:54:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.528+0000 7f0e51c00640 1 -- 172.21.15.28:0/2391391947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e58003d10 con 0x5590b11c6820 2024-04-26T01:54:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.529+0000 7f0e6a283740 1 -- 172.21.15.28:0/2391391947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c6820 msgr2=0x5590b11c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.529+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2391391947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c6820 0x5590b11c6c00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009f80 tx=0x7f0e58010860 comp rx=0 tx=0).stop 2024-04-26T01:54:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.530+0000 7f0e51c00640 1 -- 172.21.15.28:0/2391391947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e58011ac0 con 0x5590b11c6820 2024-04-26T01:54:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.531+0000 7f0e6a283740 1 -- 172.21.15.28:0/2391391947 shutdown_connections 2024-04-26T01:54:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.532+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2391391947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11d3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.532+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2391391947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11c7140 0x5590b11d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.532+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2391391947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c6820 0x5590b11c6c00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.532+0000 7f0e6a283740 1 -- 172.21.15.28:0/2391391947 >> 172.21.15.28:0/2391391947 conn(0x5590b11c2d30 msgr2=0x5590b11c3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.532+0000 7f0e65400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e480a3e40 0x7f0e480a6300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009f80 tx=0x7f0e58010860 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.533+0000 7f0e6a283740 1 -- 172.21.15.28:0/2391391947 shutdown_connections 2024-04-26T01:54:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.533+0000 7f0e6a283740 1 -- 172.21.15.28:0/2391391947 wait complete. 2024-04-26T01:54:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.534+0000 7f0e6a283740 1 -- start start 2024-04-26T01:54:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.534+0000 7f0e6a283740 1 Processor -- start 2024-04-26T01:54:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11cebc0 0x5590b11cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e6a283740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590b11d4630 con 0x5590b11c7140 2024-04-26T01:54:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e6a283740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5590b10e6310 con 0x5590b11d1a00 2024-04-26T01:54:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e6a283740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5590b11d47b0 con 0x5590b11cebc0 2024-04-26T01:54:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e6a283740 10 client.0 _ll_register_callbacks cb 0x5590b10b4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53184/0 (socket says 172.21.15.28:53184) 2024-04-26T01:54:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.535+0000 7f0e5fe00640 1 -- 172.21.15.28:0/2879189013 learned_addr learned my addr 172.21.15.28:0/2879189013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.536+0000 7f0e65400640 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11cebc0 0x5590b11cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.536+0000 7f0e64a00640 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.536+0000 7f0e5fe00640 1 -- 172.21.15.28:0/2879189013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11cebc0 msgr2=0x5590b11cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.536+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11cebc0 0x5590b11cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.536+0000 7f0e5fe00640 1 -- 172.21.15.28:0/2879189013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 msgr2=0x5590b11d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.536+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.536+0000 7f0e5fe00640 1 -- 172.21.15.28:0/2879189013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590b0f0b620 con 0x5590b11d1a00 2024-04-26T01:54:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.536+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11ce680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c09ae70 tx=0x7f0e4c017360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98122f054353cf9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.537+0000 7f0e46000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:54:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.537+0000 7f0e46000640 1 -- 172.21.15.28:0/2879189013 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0b4060 con 0x5590b11d1a00 2024-04-26T01:54:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.537+0000 7f0e46000640 1 -- 172.21.15.28:0/2879189013 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f0e4c0977e0 con 0x5590b11d1a00 2024-04-26T01:54:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.537+0000 7f0e6a283740 1 -- 172.21.15.28:0/2879189013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e58003d10 con 0x5590b11d1a00 2024-04-26T01:54:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.537+0000 7f0e46000640 1 -- 172.21.15.28:0/2879189013 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0e4c09b170 con 0x5590b11d1a00 2024-04-26T01:54:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.537+0000 7f0e6a283740 10 client.26321 fetch_fsmap learned FSMap version 46 2024-04-26T01:54:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.537+0000 7f0e6a283740 1 -- 172.21.15.28:0/2879189013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5590b11e0840 con 0x5590b11d1a00 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.537+0000 7f0e46000640 1 -- 172.21.15.28:0/2879189013 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e58003d10 con 0x5590b11d1a00 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.538+0000 7f0e46000640 1 -- 172.21.15.28:0/2879189013 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f0e4c0cca90 con 0x5590b11d1a00 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.538+0000 7f0e6a283740 10 client.26321 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.538+0000 7f0e6a283740 10 client.26321 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.538+0000 7f0e6a283740 20 client.26321 populate_metadata read hostname 'smithi028' 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.538+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5590b11e1d00 0x5590b11e40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.538+0000 7f0e6a283740 4 client.26321 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.538+0000 7f0e6a283740 1 -- 172.21.15.28:0/2879189013 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590b11408b0 con 0x5590b11e1d00 2024-04-26T01:54:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.538+0000 7f0e64a00640 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5590b11e1d00 0x5590b11e40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.540+0000 7f0e64a00640 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5590b11e1d00 0x5590b11e40f0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40dfa2f2b1087abd server_cookie=f214d07d20b80595 in_seq=0 out_seq=0 2024-04-26T01:54:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.540+0000 7f0e46000640 10 client.26321 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.541+0000 7f0e46000640 1 -- 172.21.15.28:0/2879189013 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590b11408b0 con 0x5590b11e1d00 2024-04-26T01:54:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.541+0000 7f0e46000640 10 client.26321 handle_command_reply: tid=0 2024-04-26T01:54:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.541+0000 7f0e6a283740 1 client.26321 shutdown 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.541+0000 7f0e6a283740 1 -- 172.21.15.28:0/2879189013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 msgr2=0x5590b11ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.541+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11ce680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c09ae70 tx=0x7f0e4c017360 comp rx=0 tx=0).stop 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.541+0000 7f0e6a283740 1 -- 172.21.15.28:0/2879189013 shutdown_connections 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5590b11e1d00 0x5590b11e40f0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e46000640 0 client.26321 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11cebc0 0x5590b11cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11d1a00 0x5590b11ce680 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2879189013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e6a283740 1 -- 172.21.15.28:0/2879189013 >> 172.21.15.28:0/2879189013 conn(0x5590b11c2d30 msgr2=0x5590b11c81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e6a283740 1 -- 172.21.15.28:0/2879189013 shutdown_connections 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e6a283740 1 -- 172.21.15.28:0/2879189013 wait complete. 2024-04-26T01:54:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.542+0000 7f0e6a283740 20 client.26321 trim_cache size 0 max 16384 2024-04-26T01:54:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.567+0000 7f0e6a283740 1 Processor -- start 2024-04-26T01:54:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.567+0000 7f0e6a283740 1 -- start start 2024-04-26T01:54:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.567+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11dd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.567+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11e99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.567+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 0x5590b11e9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.567+0000 7f0e6a283740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e601b6a30 con 0x5590b11c7140 2024-04-26T01:54:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.567+0000 7f0e6a283740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e601b42a0 con 0x5590b11cebc0 2024-04-26T01:54:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.567+0000 7f0e6a283740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e601b2110 con 0x5590b11d1a00 2024-04-26T01:54:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.568+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11e99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.568+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11e99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53188/0 (socket says 172.21.15.28:53188) 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e5fe00640 1 -- 172.21.15.28:0/2683904020 learned_addr learned my addr 172.21.15.28:0/2683904020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e65400640 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 0x5590b11e9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e5fe00640 1 -- 172.21.15.28:0/2683904020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 msgr2=0x5590b11e9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 0x5590b11e9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e5fe00640 1 -- 172.21.15.28:0/2683904020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 msgr2=0x5590b11dd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e64a00640 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11dd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11dd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e5fe00640 1 -- 172.21.15.28:0/2683904020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590b11e4460 con 0x5590b11cebc0 2024-04-26T01:54:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.569+0000 7f0e64a00640 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11dd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:54:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.570+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11e99c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c09b100 tx=0x7f0e4c099680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45fb4fedd48cf188 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.570+0000 7f0e51200640 1 -- 172.21.15.28:0/2683904020 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0a3de0 con 0x5590b11cebc0 2024-04-26T01:54:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.570+0000 7f0e51200640 1 -- 172.21.15.28:0/2683904020 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e4c09a830 con 0x5590b11cebc0 2024-04-26T01:54:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.570+0000 7f0e51200640 1 -- 172.21.15.28:0/2683904020 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0971f0 con 0x5590b11cebc0 2024-04-26T01:54:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.571+0000 7f0e6a283740 1 -- 172.21.15.28:0/2683904020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 msgr2=0x5590b11e99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.571+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11e99c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c09b100 tx=0x7f0e4c099680 comp rx=0 tx=0).stop 2024-04-26T01:54:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.572+0000 7f0e6a283740 1 -- 172.21.15.28:0/2683904020 shutdown_connections 2024-04-26T01:54:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.572+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 0x5590b11e9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.572+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11e99c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.572+0000 7f0e6a283740 1 --2- 172.21.15.28:0/2683904020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11dd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.572+0000 7f0e6a283740 1 -- 172.21.15.28:0/2683904020 >> 172.21.15.28:0/2683904020 conn(0x5590b11c2d30 msgr2=0x5590b11c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.572+0000 7f0e6a283740 1 -- 172.21.15.28:0/2683904020 shutdown_connections 2024-04-26T01:54:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.572+0000 7f0e6a283740 1 -- 172.21.15.28:0/2683904020 wait complete. 2024-04-26T01:54:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.573+0000 7f0e6a283740 1 -- start start 2024-04-26T01:54:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.573+0000 7f0e6a283740 1 Processor -- start 2024-04-26T01:54:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.574+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.574+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.574+0000 7f0e6a283740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 0x5590b11d8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e6a283740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e601b2760 con 0x5590b11c7140 2024-04-26T01:54:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e6a283740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e601b3bd0 con 0x5590b11cebc0 2024-04-26T01:54:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e6a283740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0e601b2e30 con 0x5590b11d1a00 2024-04-26T01:54:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e6a283740 10 client.0 _ll_register_callbacks cb 0x7f0e4c09aad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11d3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53194/0 (socket says 172.21.15.28:53194) 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e5fe00640 1 -- 172.21.15.28:0/1323952997 learned_addr learned my addr 172.21.15.28:0/1323952997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e64a00640 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e65400640 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 0x5590b11d8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e5fe00640 1 -- 172.21.15.28:0/1323952997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 msgr2=0x5590b11d8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 0x5590b11d8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.575+0000 7f0e5fe00640 1 -- 172.21.15.28:0/1323952997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 msgr2=0x5590b11d2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.576+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.576+0000 7f0e5fe00640 1 -- 172.21.15.28:0/1323952997 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590b11d85a0 con 0x5590b11cebc0 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.576+0000 7f0e64a00640 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.576+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11d3020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c09f3c0 tx=0x7f0e4c00bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aae84de4c7052e0c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.576+0000 7f0e46000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:54:23.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.576+0000 7f0e46000640 1 -- 172.21.15.28:0/1323952997 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0b4070 con 0x5590b11cebc0 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.576+0000 7f0e46000640 1 -- 172.21.15.28:0/1323952997 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f0e4c0971a0 con 0x5590b11cebc0 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.576+0000 7f0e6a283740 1 -- 172.21.15.28:0/1323952997 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e4c09a830 con 0x5590b11cebc0 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e46000640 1 -- 172.21.15.28:0/1323952997 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0e4c0a3de0 con 0x5590b11cebc0 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e6a283740 10 client.26327 fetch_fsmap learned FSMap version 46 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e6a283740 1 -- 172.21.15.28:0/1323952997 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5590b12488c0 con 0x5590b11cebc0 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e46000640 1 -- 172.21.15.28:0/1323952997 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e4c09a830 con 0x5590b11cebc0 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e46000640 1 -- 172.21.15.28:0/1323952997 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f0e4c095ce0 con 0x5590b11cebc0 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e6a283740 10 client.26327 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e6a283740 10 client.26327 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e6a283740 20 client.26327 populate_metadata read hostname 'smithi028' 2024-04-26T01:54:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e6a283740 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5590b11e1d00 0x5590b124ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e6a283740 4 client.26327 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.577+0000 7f0e6a283740 1 -- 172.21.15.28:0/1323952997 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5590b113a320 con 0x5590b11e1d00 2024-04-26T01:54:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.578+0000 7f0e64a00640 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5590b11e1d00 0x5590b124ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.579+0000 7f0e64a00640 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5590b11e1d00 0x5590b124ad90 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d6ae903f626300c server_cookie=e3f0e9a32b35ae10 in_seq=0 out_seq=0 2024-04-26T01:54:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.579+0000 7f0e46000640 10 client.26327 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e46000640 1 -- 172.21.15.28:0/1323952997 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5590b113a320 con 0x5590b11e1d00 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e46000640 10 client.26327 handle_command_reply: tid=0 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 client.26327 shutdown 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 -- 172.21.15.28:0/1323952997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 msgr2=0x5590b11d3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11d3020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c09f3c0 tx=0x7f0e4c00bf90 comp rx=0 tx=0).stop 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 -- 172.21.15.28:0/1323952997 shutdown_connections 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5590b11e1d00 0x5590b124ad90 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590b11d1a00 0x5590b11d8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590b11cebc0 0x5590b11d3020 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 --2- 172.21.15.28:0/1323952997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590b11c7140 0x5590b11d2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 -- 172.21.15.28:0/1323952997 >> 172.21.15.28:0/1323952997 conn(0x5590b11c2d30 msgr2=0x5590b11ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e46000640 0 client.26327 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.594+0000 7f0e6a283740 1 -- 172.21.15.28:0/1323952997 shutdown_connections 2024-04-26T01:54:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.595+0000 7f0e6a283740 1 -- 172.21.15.28:0/1323952997 wait complete. 2024-04-26T01:54:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.595+0000 7f0e6a283740 20 client.26327 trim_cache size 0 max 16384 2024-04-26T01:54:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:54:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:54:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:54:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:54:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 -- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e480a3e40 msgr2=0x7f0e480a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e480a3e40 0x7f0e480a6300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009f80 tx=0x7f0e58010860 comp rx=0 tx=0).stop 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 -- 172.21.15.28:0/3276662965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60101680 msgr2=0x7f0e6010c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60101680 0x7f0e6010c140 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00b4a0 tx=0x7f0e4c00b960 comp rx=0 tx=0).stop 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 -- 172.21.15.28:0/3276662965 shutdown_connections 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0e480a3e40 0x7f0e480a6300 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e6010e910 0x7f0e6010c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e60101680 0x7f0e6010c140 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 --2- 172.21.15.28:0/3276662965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e60100d60 0x7f0e6010e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.597+0000 7f0e67400640 1 -- 172.21.15.28:0/3276662965 >> 172.21.15.28:0/3276662965 conn(0x7f0e60077fd0 msgr2=0x7f0e6010fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.598+0000 7f0e67400640 1 -- 172.21.15.28:0/3276662965 shutdown_connections 2024-04-26T01:54:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:23.598+0000 7f0e67400640 1 -- 172.21.15.28:0/3276662965 wait complete. 2024-04-26T01:54:23.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:23 smithi053 ceph-mon[32593]: pgmap v421: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-04-26T01:54:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:23 smithi028 ceph-mon[28247]: pgmap v421: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-04-26T01:54:23.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:23 smithi169 ceph-mon[32851]: pgmap v421: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-04-26T01:54:25.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:25 smithi053 ceph-mon[32593]: pgmap v422: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 687 op/s 2024-04-26T01:54:25.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:25 smithi028 ceph-mon[28247]: pgmap v422: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 687 op/s 2024-04-26T01:54:25.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:25 smithi169 ceph-mon[32851]: pgmap v422: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 687 op/s 2024-04-26T01:54:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 750.0 s, 2173.8 tps, lat 46.085 ms stddev 60.810 2024-04-26T01:54:27.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:27 smithi053 ceph-mon[32593]: pgmap v423: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.2 MiB/s wr, 747 op/s 2024-04-26T01:54:27.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:27 smithi028 ceph-mon[28247]: pgmap v423: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.2 MiB/s wr, 747 op/s 2024-04-26T01:54:27.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:27 smithi169 ceph-mon[32851]: pgmap v423: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.2 MiB/s wr, 747 op/s 2024-04-26T01:54:29.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:29 smithi053 ceph-mon[32593]: pgmap v424: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.6 MiB/s wr, 587 op/s 2024-04-26T01:54:29.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:29 smithi028 ceph-mon[28247]: pgmap v424: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.6 MiB/s wr, 587 op/s 2024-04-26T01:54:29.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:29 smithi169 ceph-mon[32851]: pgmap v424: 97 pgs: 97 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.6 MiB/s wr, 587 op/s 2024-04-26T01:54:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 755.0 s, 2529.8 tps, lat 39.401 ms stddev 49.039 2024-04-26T01:54:31.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:31 smithi053 ceph-mon[32593]: pgmap v425: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.8 MiB/s wr, 900 op/s 2024-04-26T01:54:31.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:31 smithi028 ceph-mon[28247]: pgmap v425: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.8 MiB/s wr, 900 op/s 2024-04-26T01:54:31.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:31 smithi169 ceph-mon[32851]: pgmap v425: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.8 MiB/s wr, 900 op/s 2024-04-26T01:54:33.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:33 smithi053 ceph-mon[32593]: pgmap v426: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.7 MiB/s wr, 579 op/s 2024-04-26T01:54:33.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:33 smithi028 ceph-mon[28247]: pgmap v426: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.7 MiB/s wr, 579 op/s 2024-04-26T01:54:33.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:33 smithi169 ceph-mon[32851]: pgmap v426: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.7 MiB/s wr, 579 op/s 2024-04-26T01:54:35.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:35 smithi028 ceph-mon[28247]: pgmap v427: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.6 MiB/s wr, 747 op/s 2024-04-26T01:54:35.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:35 smithi169 ceph-mon[32851]: pgmap v427: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.6 MiB/s wr, 747 op/s 2024-04-26T01:54:35.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:35 smithi053 ceph-mon[32593]: pgmap v427: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.6 MiB/s wr, 747 op/s 2024-04-26T01:54:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 760.0 s, 1571.4 tps, lat 62.240 ms stddev 66.753 2024-04-26T01:54:36.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:36 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 -- 172.21.15.28:0/2350195957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076480 msgr2=0x7fedb8076860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 --2- 172.21.15.28:0/2350195957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076480 0x7fedb8076860 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7feda8009930 tx=0x7feda802a460 comp rx=0 tx=0).stop 2024-04-26T01:54:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 -- 172.21.15.28:0/2350195957 shutdown_connections 2024-04-26T01:54:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 --2- 172.21.15.28:0/2350195957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb8112c00 0x7fedb8114ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 --2- 172.21.15.28:0/2350195957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedb8076da0 0x7fedb81126c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 --2- 172.21.15.28:0/2350195957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076480 0x7fedb8076860 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 -- 172.21.15.28:0/2350195957 >> 172.21.15.28:0/2350195957 conn(0x7fedb80ffa70 msgr2=0x7fedb8101e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 -- 172.21.15.28:0/2350195957 shutdown_connections 2024-04-26T01:54:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.799+0000 7fedbce00640 1 -- 172.21.15.28:0/2350195957 wait complete. 2024-04-26T01:54:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.801+0000 7fedbce00640 1 Processor -- start 2024-04-26T01:54:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.802+0000 7fedbce00640 1 -- start start 2024-04-26T01:54:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.803+0000 7fedbce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedb8076480 0x7fedb81089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.803+0000 7fedbce00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076da0 0x7fedb8108f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.803+0000 7fedbce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb8112c00 0x7fedb8072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.804+0000 7fedbce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedb8104770 con 0x7fedb8112c00 2024-04-26T01:54:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.804+0000 7fedbce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81045f0 con 0x7fedb8076480 2024-04-26T01:54:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.804+0000 7fedb6a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076da0 0x7fedb8108f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.804+0000 7fedb6a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076da0 0x7fedb8108f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51808/0 (socket says 172.21.15.28:51808) 2024-04-26T01:54:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.804+0000 7fedb6a00640 1 -- 172.21.15.28:0/1620053582 learned_addr learned my addr 172.21.15.28:0/1620053582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:36.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.804+0000 7fedb7e00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb8112c00 0x7fedb8072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.804+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81048f0 con 0x7fedb8076da0 2024-04-26T01:54:36.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.805+0000 7fedb7400640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedb8076480 0x7fedb81089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.805+0000 7fedb7400640 1 -- 172.21.15.28:0/1620053582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076da0 msgr2=0x7fedb8108f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.805+0000 7fedb7400640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076da0 0x7fedb8108f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.805+0000 7fedb7400640 1 -- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb8112c00 msgr2=0x7fedb8072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.805+0000 7fedb7400640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb8112c00 0x7fedb8072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.805+0000 7fedb7400640 1 -- 172.21.15.28:0/1620053582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb80735c0 con 0x7fedb8076480 2024-04-26T01:54:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.805+0000 7fedb7400640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedb8076480 0x7fedb81089f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feda802aa50 tx=0x7feda8003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.806+0000 7fedafe00640 1 -- 172.21.15.28:0/1620053582 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda803f070 con 0x7fedb8076480 2024-04-26T01:54:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.806+0000 7fedafe00640 1 -- 172.21.15.28:0/1620053582 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda8031d60 con 0x7fedb8076480 2024-04-26T01:54:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.806+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb8073850 con 0x7fedb8076480 2024-04-26T01:54:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.806+0000 7fedafe00640 1 -- 172.21.15.28:0/1620053582 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda8035030 con 0x7fedb8076480 2024-04-26T01:54:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.806+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb8110910 con 0x7fedb8076480 2024-04-26T01:54:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.810+0000 7fedafe00640 1 -- 172.21.15.28:0/1620053582 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feda8042070 con 0x7fedb8076480 2024-04-26T01:54:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.811+0000 7fedc1217740 1 Processor -- start 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.812+0000 7fedc1217740 1 -- start start 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.812+0000 7fedc1217740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdbed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.812+0000 7fedc1217740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.812+0000 7fedc1217740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcdcbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.812+0000 7fedc1217740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558bfcd24900 con 0x558bfcdbe9b0 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.812+0000 7fedc1217740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558bfcccaef0 con 0x558bfcdc9b90 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.812+0000 7fedc1217740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558bfcd208c0 con 0x558bfcdbf2d0 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.813+0000 7fedb7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.813+0000 7fedb7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51820/0 (socket says 172.21.15.28:51820) 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.813+0000 7fedb7400640 1 -- 172.21.15.28:0/508557113 learned_addr learned my addr 172.21.15.28:0/508557113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.813+0000 7fedb7e00640 1 --2- 172.21.15.28:0/508557113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdbed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.814+0000 7fedb7e00640 1 -- 172.21.15.28:0/508557113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbf2d0 msgr2=0x558bfcdc9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.814+0000 7fedafe00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fed9c0a3d90 0x7fed9c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.814+0000 7fedb7e00640 1 --2- 172.21.15.28:0/508557113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.814+0000 7fedb7e00640 1 -- 172.21.15.28:0/508557113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 msgr2=0x558bfcdcbf80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:54:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.814+0000 7fedb7e00640 1 --2- 172.21.15.28:0/508557113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcdcbf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.814+0000 7fedb7e00640 1 -- 172.21.15.28:0/508557113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bfca37210 con 0x558bfcdbe9b0 2024-04-26T01:54:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.814+0000 7fedafe00640 1 -- 172.21.15.28:0/1620053582 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7feda80f10b0 con 0x7fedb8076480 2024-04-26T01:54:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.815+0000 7fedb7e00640 1 --2- 172.21.15.28:0/508557113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdbed90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009da0 tx=0x7fedb000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdeba898fd67d73a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.816+0000 7fedb7e00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fed9c0a3d90 0x7fed9c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.816+0000 7fedacc00640 1 -- 172.21.15.28:0/508557113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0007800 con 0x558bfcdbe9b0 2024-04-26T01:54:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.817+0000 7fedacc00640 1 -- 172.21.15.28:0/508557113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedb0015410 con 0x558bfcdbe9b0 2024-04-26T01:54:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.817+0000 7fedacc00640 1 -- 172.21.15.28:0/508557113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb000a620 con 0x558bfcdbe9b0 2024-04-26T01:54:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.818+0000 7fedc1217740 1 -- 172.21.15.28:0/508557113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 msgr2=0x558bfcdbed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.818+0000 7fedc1217740 1 --2- 172.21.15.28:0/508557113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdbed90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009da0 tx=0x7fedb000c7d0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.818+0000 7fedb7e00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fed9c0a3d90 0x7fed9c0a6250 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fedb0018000 tx=0x7fedb000acd0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.818+0000 7fedc1217740 1 -- 172.21.15.28:0/508557113 shutdown_connections 2024-04-26T01:54:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.818+0000 7fedc1217740 1 --2- 172.21.15.28:0/508557113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcdcbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.819+0000 7fedc1217740 1 --2- 172.21.15.28:0/508557113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.819+0000 7fedc1217740 1 --2- 172.21.15.28:0/508557113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdbed90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.819+0000 7fedc1217740 1 -- 172.21.15.28:0/508557113 >> 172.21.15.28:0/508557113 conn(0x558bfcdbaec0 msgr2=0x558bfcdbb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.819+0000 7fedc1217740 1 -- 172.21.15.28:0/508557113 shutdown_connections 2024-04-26T01:54:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.819+0000 7fedc1217740 1 -- 172.21.15.28:0/508557113 wait complete. 2024-04-26T01:54:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.820+0000 7fedc1217740 1 -- start start 2024-04-26T01:54:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.820+0000 7fedc1217740 1 Processor -- start 2024-04-26T01:54:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.821+0000 7fedc1217740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdc6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.821+0000 7fedc1217740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.821+0000 7fedb7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdc6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.821+0000 7fedb7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdc6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51822/0 (socket says 172.21.15.28:51822) 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.821+0000 7fedb6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.821+0000 7fedb6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52416/0 (socket says 172.21.15.28:52416) 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.821+0000 7fedc1217740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcdc7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedb7400640 1 -- 172.21.15.28:0/3675912781 learned_addr learned my addr 172.21.15.28:0/3675912781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558bfcdcc7c0 con 0x558bfcdbf2d0 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558bfccde310 con 0x558bfcdc9b90 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x558bfcdcc940 con 0x558bfcdbe9b0 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedb7e00640 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcdc7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedc1217740 10 client.0 _ll_register_callbacks cb 0x558bfccaccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedb7400640 1 -- 172.21.15.28:0/3675912781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 msgr2=0x558bfcdc7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedb7400640 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcdc7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedb7400640 1 -- 172.21.15.28:0/3675912781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 msgr2=0x558bfcdc70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedb7400640 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedb7400640 1 -- 172.21.15.28:0/3675912781 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bfcb03620 con 0x558bfcdbe9b0 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.822+0000 7fedb7400640 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdc6ba0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7feda80b94b0 tx=0x7feda8031f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d49f951a9f7113bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.823+0000 7feda1600640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.823+0000 7feda1600640 1 -- 172.21.15.28:0/3675912781 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda8049070 con 0x558bfcdbe9b0 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.823+0000 7feda1600640 1 -- 172.21.15.28:0/3675912781 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7feda80eea80 con 0x558bfcdbe9b0 2024-04-26T01:54:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.823+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedb0015410 con 0x558bfcdbe9b0 2024-04-26T01:54:36.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.823+0000 7feda1600640 1 -- 172.21.15.28:0/3675912781 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7feda80bce20 con 0x558bfcdbe9b0 2024-04-26T01:54:36.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7fedc1217740 10 client.26350 fetch_fsmap learned FSMap version 46 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x558bfcdcc810 con 0x558bfcdbe9b0 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7feda1600640 1 -- 172.21.15.28:0/3675912781 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedb0015410 con 0x558bfcdbe9b0 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7feda1600640 1 -- 172.21.15.28:0/3675912781 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7feda80b96c0 con 0x558bfcdbe9b0 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7fedc1217740 10 client.26350 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7fedc1217740 10 client.26350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7fedc1217740 20 client.26350 populate_metadata read hostname 'smithi028' 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7fedc1217740 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bfcdd9f50 0x558bfcddc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7fedc1217740 4 client.26350 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.824+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bfcd388b0 con 0x558bfcdd9f50 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.825+0000 7fedb6a00640 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bfcdd9f50 0x558bfcddc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.825+0000 7fedb6a00640 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bfcdd9f50 0x558bfcddc3f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17a15b67666abc4f server_cookie=b7e28b4b14c66e7d in_seq=0 out_seq=0 2024-04-26T01:54:36.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.826+0000 7feda1600640 10 client.26350 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.826+0000 7feda1600640 1 -- 172.21.15.28:0/3675912781 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558bfcd388b0 con 0x558bfcdd9f50 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.826+0000 7feda1600640 10 client.26350 handle_command_reply: tid=0 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 client.26350 shutdown 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 msgr2=0x558bfcdc6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdc6ba0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7feda80b94b0 tx=0x7feda8031f10 comp rx=0 tx=0).stop 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 shutdown_connections 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bfcdd9f50 0x558bfcddc3f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcdc7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 --2- 172.21.15.28:0/3675912781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdc6ba0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 >> 172.21.15.28:0/3675912781 conn(0x558bfcdbaec0 msgr2=0x558bfcdc0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7feda1600640 0 client.26350 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:36.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 shutdown_connections 2024-04-26T01:54:36.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 1 -- 172.21.15.28:0/3675912781 wait complete. 2024-04-26T01:54:36.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.827+0000 7fedc1217740 20 client.26350 trim_cache size 0 max 16384 2024-04-26T01:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.850+0000 7fedc1217740 1 Processor -- start 2024-04-26T01:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.850+0000 7fedc1217740 1 -- start start 2024-04-26T01:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.850+0000 7fedc1217740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdba7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.850+0000 7fedc1217740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedc1217740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcde1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedc1217740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81c79e0 con 0x558bfcdbf2d0 2024-04-26T01:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedc1217740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81c5250 con 0x558bfcdc9b90 2024-04-26T01:54:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedc1217740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81c30c0 con 0x558bfcdbe9b0 2024-04-26T01:54:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedb7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcde1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedb7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcde1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:60544/0 (socket says 172.21.15.28:60544) 2024-04-26T01:54:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedb7e00640 1 -- 172.21.15.28:0/485168796 learned_addr learned my addr 172.21.15.28:0/485168796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedb6a00640 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.851+0000 7fedb7400640 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdba7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.852+0000 7fedb6a00640 1 -- 172.21.15.28:0/485168796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 msgr2=0x558bfcdba7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.852+0000 7fedb6a00640 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdba7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.852+0000 7fedb6a00640 1 -- 172.21.15.28:0/485168796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 msgr2=0x558bfcde1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.852+0000 7fedb6a00640 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcde1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.852+0000 7fedb6a00640 1 -- 172.21.15.28:0/485168796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bfcdbace0 con 0x558bfcdbf2d0 2024-04-26T01:54:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.852+0000 7fedb7e00640 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcde1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.853+0000 7fedb6a00640 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc7440 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x558bfcdbc2e0 tx=0x7feda4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2620b189c0878ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.855+0000 7feda3e00640 1 -- 172.21.15.28:0/485168796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda40154e0 con 0x558bfcdbf2d0 2024-04-26T01:54:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.855+0000 7feda3e00640 1 -- 172.21.15.28:0/485168796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda401e030 con 0x558bfcdbf2d0 2024-04-26T01:54:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.855+0000 7feda3e00640 1 -- 172.21.15.28:0/485168796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4022960 con 0x558bfcdbf2d0 2024-04-26T01:54:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.856+0000 7fedc1217740 1 -- 172.21.15.28:0/485168796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 msgr2=0x558bfcdc7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.856+0000 7fedc1217740 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc7440 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x558bfcdbc2e0 tx=0x7feda4008040 comp rx=0 tx=0).stop 2024-04-26T01:54:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.856+0000 7fedc1217740 1 -- 172.21.15.28:0/485168796 shutdown_connections 2024-04-26T01:54:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.856+0000 7fedc1217740 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfcdc9b90 0x558bfcde1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.856+0000 7fedc1217740 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbf2d0 0x558bfcdc7440 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.856+0000 7fedc1217740 1 --2- 172.21.15.28:0/485168796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdbe9b0 0x558bfcdba7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.856+0000 7fedc1217740 1 -- 172.21.15.28:0/485168796 >> 172.21.15.28:0/485168796 conn(0x558bfcdbaec0 msgr2=0x558bfcdc0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:36.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.856+0000 7fedc1217740 1 -- 172.21.15.28:0/485168796 shutdown_connections 2024-04-26T01:54:36.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.857+0000 7fedc1217740 1 -- 172.21.15.28:0/485168796 wait complete. 2024-04-26T01:54:36.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.857+0000 7fedc1217740 1 -- start start 2024-04-26T01:54:36.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.857+0000 7fedc1217740 1 Processor -- start 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedc1217740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdcfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedc1217740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdc9b90 0x558bfcdd01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedc1217740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfce3d530 0x558bfce3f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedc1217740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81c3710 con 0x558bfcdbe9b0 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedc1217740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81c4b80 con 0x558bfce3d530 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedc1217740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81c3de0 con 0x558bfcdc9b90 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedc1217740 10 client.0 _ll_register_callbacks cb 0x7feda80f48f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedb6a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdc9b90 0x558bfcdd01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedb7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdcfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedb6a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdc9b90 0x558bfcdd01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:51832/0 (socket says 172.21.15.28:51832) 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedb6a00640 1 -- 172.21.15.28:0/3416090426 learned_addr learned my addr 172.21.15.28:0/3416090426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:36.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.858+0000 7fedb7e00640 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfce3d530 0x558bfce3f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.859+0000 7fedb6a00640 1 -- 172.21.15.28:0/3416090426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfce3d530 msgr2=0x558bfce3f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.859+0000 7fedb6a00640 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfce3d530 0x558bfce3f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.859+0000 7fedb6a00640 1 -- 172.21.15.28:0/3416090426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 msgr2=0x558bfcdcfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.859+0000 7fedb6a00640 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdcfc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.859+0000 7fedb6a00640 1 -- 172.21.15.28:0/3416090426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bfce3fe60 con 0x558bfcdc9b90 2024-04-26T01:54:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.860+0000 7fedb7e00640 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfce3d530 0x558bfce3f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.860+0000 7fedb7400640 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdcfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.860+0000 7fedb6a00640 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdc9b90 0x558bfcdd01a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feda4000c00 tx=0x7feda4022670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9655cd635ce9af03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.860+0000 7feda1600640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:54:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.861+0000 7feda1600640 1 -- 172.21.15.28:0/3416090426 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4029070 con 0x558bfcdc9b90 2024-04-26T01:54:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.862+0000 7feda1600640 1 -- 172.21.15.28:0/3416090426 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7feda400f040 con 0x558bfcdc9b90 2024-04-26T01:54:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.862+0000 7fedc1217740 1 -- 172.21.15.28:0/3416090426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feda401e030 con 0x558bfcdc9b90 2024-04-26T01:54:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.862+0000 7feda1600640 1 -- 172.21.15.28:0/3416090426 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7feda4035050 con 0x558bfcdc9b90 2024-04-26T01:54:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.862+0000 7fedc1217740 10 client.26356 fetch_fsmap learned FSMap version 46 2024-04-26T01:54:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.862+0000 7fedc1217740 1 -- 172.21.15.28:0/3416090426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x558bfce41170 con 0x558bfcdc9b90 2024-04-26T01:54:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.862+0000 7feda1600640 1 -- 172.21.15.28:0/3416090426 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda401e030 con 0x558bfcdc9b90 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7feda1600640 1 -- 172.21.15.28:0/3416090426 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7feda4015c00 con 0x558bfcdc9b90 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7fedc1217740 10 client.26356 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7fedc1217740 10 client.26356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7fedc1217740 20 client.26356 populate_metadata read hostname 'smithi028' 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7fedc1217740 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bfce417c0 0x558bfce43be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7fedc1217740 4 client.26356 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7fedc1217740 1 -- 172.21.15.28:0/3416090426 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558bfcd32320 con 0x558bfce417c0 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7fedb7400640 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bfce417c0 0x558bfce43be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:36.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7fedb7400640 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bfce417c0 0x558bfce43be0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f49f1b342487b0 server_cookie=d50bf09fb198d946 in_seq=0 out_seq=0 2024-04-26T01:54:36.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.863+0000 7feda1600640 10 client.26356 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:36.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.883+0000 7feda1600640 1 -- 172.21.15.28:0/3416090426 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558bfcd32320 con 0x558bfce417c0 2024-04-26T01:54:36.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.883+0000 7feda1600640 10 client.26356 handle_command_reply: tid=0 2024-04-26T01:54:36.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.883+0000 7fedc1217740 1 client.26356 shutdown 2024-04-26T01:54:36.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7fedc1217740 1 -- 172.21.15.28:0/3416090426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdc9b90 msgr2=0x558bfcdd01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7fedc1217740 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdc9b90 0x558bfcdd01a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feda4000c00 tx=0x7feda4022670 comp rx=0 tx=0).stop 2024-04-26T01:54:36.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7fedc1217740 1 -- 172.21.15.28:0/3416090426 shutdown_connections 2024-04-26T01:54:36.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7fedc1217740 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x558bfce417c0 0x558bfce43be0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7fedc1217740 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bfce3d530 0x558bfce3f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7fedc1217740 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558bfcdc9b90 0x558bfcdd01a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7fedc1217740 1 --2- 172.21.15.28:0/3416090426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bfcdbe9b0 0x558bfcdcfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7fedc1217740 1 -- 172.21.15.28:0/3416090426 >> 172.21.15.28:0/3416090426 conn(0x558bfcdbaec0 msgr2=0x558bfcdc83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:36.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.884+0000 7feda1600640 0 client.26356 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:36.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.885+0000 7fedc1217740 1 -- 172.21.15.28:0/3416090426 shutdown_connections 2024-04-26T01:54:36.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.885+0000 7fedc1217740 1 -- 172.21.15.28:0/3416090426 wait complete. 2024-04-26T01:54:36.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.885+0000 7fedc1217740 20 client.26356 trim_cache size 0 max 16384 2024-04-26T01:54:36.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:54:36.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:54:36.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:54:36.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fed9c0a3d90 msgr2=0x7fed9c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fed9c0a3d90 0x7fed9c0a6250 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fedb0018000 tx=0x7fedb000acd0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedb8076480 msgr2=0x7fedb81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedb8076480 0x7fedb81089f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feda802aa50 tx=0x7feda8003cd0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 shutdown_connections 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fed9c0a3d90 0x7fed9c0a6250 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb8112c00 0x7fedb8072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb8076da0 0x7fedb8108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 --2- 172.21.15.28:0/1620053582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedb8076480 0x7fedb81089f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.886+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 >> 172.21.15.28:0/1620053582 conn(0x7fedb80ffa70 msgr2=0x7fedb8101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.887+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 shutdown_connections 2024-04-26T01:54:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:36.887+0000 7fedbce00640 1 -- 172.21.15.28:0/1620053582 wait complete. 2024-04-26T01:54:36.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:36 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:36.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:36 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:37 smithi028 ceph-mon[28247]: pgmap v428: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.2 MiB/s wr, 809 op/s 2024-04-26T01:54:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:37 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:37 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:37 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:54:37.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:37 smithi169 ceph-mon[32851]: pgmap v428: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.2 MiB/s wr, 809 op/s 2024-04-26T01:54:37.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:37 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:37.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:37 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:37.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:37 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:54:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:37 smithi053 ceph-mon[32593]: pgmap v428: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.2 MiB/s wr, 809 op/s 2024-04-26T01:54:37.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:37 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:37.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:37 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:37.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:37 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:54:39.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:39 smithi028 ceph-mon[28247]: pgmap v429: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 639 op/s 2024-04-26T01:54:39.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:39 smithi169 ceph-mon[32851]: pgmap v429: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 639 op/s 2024-04-26T01:54:39.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:39 smithi053 ceph-mon[32593]: pgmap v429: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 639 op/s 2024-04-26T01:54:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 765.0 s, 1683.4 tps, lat 60.674 ms stddev 80.516 2024-04-26T01:54:41.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:41 smithi028 ceph-mon[28247]: pgmap v430: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 8.8 MiB/s wr, 898 op/s 2024-04-26T01:54:41.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:41 smithi169 ceph-mon[32851]: pgmap v430: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 8.8 MiB/s wr, 898 op/s 2024-04-26T01:54:41.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:41 smithi053 ceph-mon[32593]: pgmap v430: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 8.8 MiB/s wr, 898 op/s 2024-04-26T01:54:43.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:43 smithi028 ceph-mon[28247]: pgmap v431: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 5.6 MiB/s wr, 585 op/s 2024-04-26T01:54:43.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:43 smithi169 ceph-mon[32851]: pgmap v431: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 5.6 MiB/s wr, 585 op/s 2024-04-26T01:54:43.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:43 smithi053 ceph-mon[32593]: pgmap v431: 97 pgs: 97 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 5.6 MiB/s wr, 585 op/s 2024-04-26T01:54:45.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:45 smithi028 ceph-mon[28247]: pgmap v432: 97 pgs: 97 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 14 MiB/s wr, 1.15k op/s 2024-04-26T01:54:45.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:45 smithi169 ceph-mon[32851]: pgmap v432: 97 pgs: 97 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 14 MiB/s wr, 1.15k op/s 2024-04-26T01:54:45.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:45 smithi053 ceph-mon[32593]: pgmap v432: 97 pgs: 97 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 14 MiB/s wr, 1.15k op/s 2024-04-26T01:54:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 770.0 s, 1087.6 tps, lat 90.725 ms stddev 175.301 2024-04-26T01:54:47.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:47 smithi028 ceph-mon[28247]: pgmap v433: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 MiB/s wr, 1.73k op/s 2024-04-26T01:54:47.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:47 smithi169 ceph-mon[32851]: pgmap v433: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 MiB/s wr, 1.73k op/s 2024-04-26T01:54:47.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:47 smithi053 ceph-mon[32593]: pgmap v433: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 MiB/s wr, 1.73k op/s 2024-04-26T01:54:49.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:49 smithi028 ceph-mon[28247]: pgmap v434: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 MiB/s wr, 1.57k op/s 2024-04-26T01:54:49.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:49 smithi169 ceph-mon[32851]: pgmap v434: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 MiB/s wr, 1.57k op/s 2024-04-26T01:54:49.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:49 smithi053 ceph-mon[32593]: pgmap v434: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 MiB/s wr, 1.57k op/s 2024-04-26T01:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 -- 172.21.15.28:0/2370753924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd0104d50 msgr2=0x7f8fd0108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2370753924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd0104d50 0x7f8fd0108d90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0009900 tx=0x7f8fc002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 -- 172.21.15.28:0/2370753924 shutdown_connections 2024-04-26T01:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2370753924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd0109470 0x7f8fd0111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2370753924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd0104d50 0x7f8fd0108d90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2370753924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fd01043a0 0x7f8fd0104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 -- 172.21.15.28:0/2370753924 >> 172.21.15.28:0/2370753924 conn(0x7f8fd00ffc40 msgr2=0x7f8fd0102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 -- 172.21.15.28:0/2370753924 shutdown_connections 2024-04-26T01:54:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.065+0000 7f8fd7600640 1 -- 172.21.15.28:0/2370753924 wait complete. 2024-04-26T01:54:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.066+0000 7f8fd7600640 1 Processor -- start 2024-04-26T01:54:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.067+0000 7f8fd7600640 1 -- start start 2024-04-26T01:54:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.067+0000 7f8fd7600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fd01043a0 0x7f8fd01a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.067+0000 7f8fd7600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd0109470 0x7f8fd01a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.067+0000 7f8fd7600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd019a700 0x7f8fd019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.067+0000 7f8fd7600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd01143e0 con 0x7f8fd019a700 2024-04-26T01:54:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.067+0000 7f8fd7600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd0114260 con 0x7f8fd01043a0 2024-04-26T01:54:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.067+0000 7f8fd7600640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd0114560 con 0x7f8fd0109470 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fd01043a0 0x7f8fd01a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fd01043a0 0x7f8fd01a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50000/0 (socket says 172.21.15.28:50000) 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd4c00640 1 -- 172.21.15.28:0/2266727809 learned_addr learned my addr 172.21.15.28:0/2266727809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fcfe00640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd0109470 0x7f8fd01a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd5600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd019a700 0x7f8fd019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd4c00640 1 -- 172.21.15.28:0/2266727809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd0109470 msgr2=0x7f8fd01a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd4c00640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd0109470 0x7f8fd01a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd4c00640 1 -- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd019a700 msgr2=0x7f8fd019ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd4c00640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd019a700 0x7f8fd019ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd4c00640 1 -- 172.21.15.28:0/2266727809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd019b300 con 0x7f8fd01043a0 2024-04-26T01:54:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.068+0000 7f8fd5600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd019a700 0x7f8fd019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.069+0000 7f8fd4c00640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fd01043a0 0x7f8fd01a05f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800ebc0 tx=0x7f8fb800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.070+0000 7f8fcd600640 1 -- 172.21.15.28:0/2266727809 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8010630 con 0x7f8fd01043a0 2024-04-26T01:54:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.070+0000 7f8fcd600640 1 -- 172.21.15.28:0/2266727809 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fb8010c80 con 0x7f8fd01043a0 2024-04-26T01:54:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.070+0000 7f8fcd600640 1 -- 172.21.15.28:0/2266727809 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8018b60 con 0x7f8fd01043a0 2024-04-26T01:54:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.070+0000 7f8fd7600640 1 -- 172.21.15.28:0/2266727809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd0076390 con 0x7f8fd01043a0 2024-04-26T01:54:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.070+0000 7f8fd7600640 1 -- 172.21.15.28:0/2266727809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd00768b0 con 0x7f8fd01043a0 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.077+0000 7f8fda3e1740 1 Processor -- start 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.077+0000 7f8fda3e1740 1 -- start start 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.078+0000 7f8fcd600640 1 -- 172.21.15.28:0/2266727809 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8fb8018d00 con 0x7f8fd01043a0 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.080+0000 7f8fcd600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8fb00a3fa0 0x7f8fb00a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.080+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 0x557e7a403ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.080+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 0x557e7a41f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.080+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a421e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.080+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557e7a369900 con 0x557e7a404020 2024-04-26T01:54:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.080+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557e7a30fef0 con 0x557e7a41fa10 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.080+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557e7a3658c0 con 0x557e7a403700 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.081+0000 7f8fd5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a421e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.081+0000 7f8fd5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a421e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50014/0 (socket says 172.21.15.28:50014) 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.081+0000 7f8fd5600640 1 -- 172.21.15.28:0/1957311040 learned_addr learned my addr 172.21.15.28:0/1957311040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.083+0000 7f8fd5600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8fb00a3fa0 0x7f8fb00a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.083+0000 7f8fcd600640 1 -- 172.21.15.28:0/2266727809 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f8fb80d0e00 con 0x7f8fd01043a0 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.084+0000 7f8fd5600640 1 -- 172.21.15.28:0/1957311040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 msgr2=0x557e7a403ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.084+0000 7f8fd5600640 1 --2- 172.21.15.28:0/1957311040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 0x557e7a403ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.084+0000 7f8fd5600640 1 -- 172.21.15.28:0/1957311040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 msgr2=0x557e7a41f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.084+0000 7f8fd5600640 1 --2- 172.21.15.28:0/1957311040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 0x557e7a41f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.084+0000 7f8fd5600640 1 -- 172.21.15.28:0/1957311040 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e7a07c210 con 0x557e7a41fa10 2024-04-26T01:54:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.088+0000 7f8fd5600640 1 --2- 172.21.15.28:0/1957311040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a421e00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8033840 tx=0x7f8fc8033c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=487725c2eac5044c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.088+0000 7f8fd5600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8fb00a3fa0 0x7f8fb00a6460 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8034010 tx=0x7f8fc8009000 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.088+0000 7f8fc5c00640 1 -- 172.21.15.28:0/1957311040 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8043070 con 0x557e7a41fa10 2024-04-26T01:54:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.088+0000 7f8fc5c00640 1 -- 172.21.15.28:0/1957311040 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc80047a0 con 0x557e7a41fa10 2024-04-26T01:54:50.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.090+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1957311040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 msgr2=0x557e7a421e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.090+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1957311040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a421e00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8033840 tx=0x7f8fc8033c40 comp rx=0 tx=0).stop 2024-04-26T01:54:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.091+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1957311040 shutdown_connections 2024-04-26T01:54:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.091+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1957311040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a421e00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.091+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1957311040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 0x557e7a41f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.091+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1957311040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 0x557e7a403ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.091+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1957311040 >> 172.21.15.28:0/1957311040 conn(0x557e7a3ffd30 msgr2=0x557e7a400110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.092+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1957311040 shutdown_connections 2024-04-26T01:54:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.092+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1957311040 wait complete. 2024-04-26T01:54:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.093+0000 7f8fda3e1740 1 -- start start 2024-04-26T01:54:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.093+0000 7f8fda3e1740 1 Processor -- start 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.095+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 0x557e7a410cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.095+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a404020 0x557e7a411200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.095+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a41fa10 0x557e7a40c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a404020 0x557e7a411200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fd4c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 0x557e7a410cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a404020 0x557e7a411200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50026/0 (socket says 172.21.15.28:50026) 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557e7a406690 con 0x557e7a41fa10 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fcfe00640 1 -- 172.21.15.28:0/1558768869 learned_addr learned my addr 172.21.15.28:0/1558768869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557e7a323310 con 0x557e7a404020 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557e7a406810 con 0x557e7a403700 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fda3e1740 10 client.0 _ll_register_callbacks cb 0x557e7a2f1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fd5600640 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a41fa10 0x557e7a40c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fd4c00640 1 -- 172.21.15.28:0/1558768869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a404020 msgr2=0x557e7a411200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fd4c00640 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a404020 0x557e7a411200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fd4c00640 1 -- 172.21.15.28:0/1558768869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a41fa10 msgr2=0x557e7a40c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fd4c00640 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a41fa10 0x557e7a40c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.096+0000 7f8fd4c00640 1 -- 172.21.15.28:0/1558768869 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e7a148620 con 0x557e7a403700 2024-04-26T01:54:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.097+0000 7f8fd4c00640 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 0x557e7a410cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80ce4f0 tx=0x7f8fb809cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997989bd8e9faf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.097+0000 7f8fbe000640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:54:50.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.100+0000 7f8fbe000640 1 -- 172.21.15.28:0/1558768869 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb80ce980 con 0x557e7a403700 2024-04-26T01:54:50.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.102+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc80047a0 con 0x557e7a403700 2024-04-26T01:54:50.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.102+0000 7f8fbe000640 1 -- 172.21.15.28:0/1558768869 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f8fb80bd2c0 con 0x557e7a403700 2024-04-26T01:54:50.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.103+0000 7f8fbe000640 1 -- 172.21.15.28:0/1558768869 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f8fb809d440 con 0x557e7a403700 2024-04-26T01:54:50.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.103+0000 7f8fda3e1740 10 client.26365 fetch_fsmap learned FSMap version 46 2024-04-26T01:54:50.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.103+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x557e7a41afe0 con 0x557e7a403700 2024-04-26T01:54:50.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.103+0000 7f8fbe000640 1 -- 172.21.15.28:0/1558768869 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc80047a0 con 0x557e7a403700 2024-04-26T01:54:50.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.104+0000 7f8fbe000640 1 -- 172.21.15.28:0/1558768869 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f8fb809c260 con 0x557e7a403700 2024-04-26T01:54:50.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.104+0000 7f8fda3e1740 10 client.26365 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:54:50.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.104+0000 7f8fda3e1740 10 client.26365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:50.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.104+0000 7f8fda3e1740 20 client.26365 populate_metadata read hostname 'smithi028' 2024-04-26T01:54:50.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.104+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557e7a41c4a0 0x557e7a41e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.104+0000 7f8fda3e1740 4 client.26365 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:50.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.104+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e7a37d8b0 con 0x557e7a41c4a0 2024-04-26T01:54:50.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.104+0000 7f8fcfe00640 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557e7a41c4a0 0x557e7a41e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.106+0000 7f8fbe000640 10 client.26365 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:50.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.106+0000 7f8fcfe00640 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557e7a41c4a0 0x557e7a41e8c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b21398b480d14587 server_cookie=a3bb8318484be213 in_seq=0 out_seq=0 2024-04-26T01:54:50.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.107+0000 7f8fbe000640 1 -- 172.21.15.28:0/1558768869 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557e7a37d8b0 con 0x557e7a41c4a0 2024-04-26T01:54:50.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.107+0000 7f8fbe000640 10 client.26365 handle_command_reply: tid=0 2024-04-26T01:54:50.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.108+0000 7f8fda3e1740 1 client.26365 shutdown 2024-04-26T01:54:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.111+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 msgr2=0x557e7a410cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.111+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 0x557e7a410cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80ce4f0 tx=0x7f8fb809cbd0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.111+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 shutdown_connections 2024-04-26T01:54:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.112+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557e7a41c4a0 0x557e7a41e8c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.112+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a41fa10 0x557e7a40c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.112+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a404020 0x557e7a411200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.112+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/1558768869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a403700 0x557e7a410cc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.112+0000 7f8fbe000640 0 client.26365 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:50.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.112+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 >> 172.21.15.28:0/1558768869 conn(0x557e7a3ffd30 msgr2=0x557e7a405b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.112+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 shutdown_connections 2024-04-26T01:54:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.113+0000 7f8fda3e1740 1 -- 172.21.15.28:0/1558768869 wait complete. 2024-04-26T01:54:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.113+0000 7f8fda3e1740 20 client.26365 trim_cache size 0 max 16384 2024-04-26T01:54:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.143+0000 7f8fda3e1740 1 Processor -- start 2024-04-26T01:54:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fda3e1740 1 -- start start 2024-04-26T01:54:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a403700 0x557e7a40b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 0x557e7a410870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a41fa10 0x557e7a410db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd0072950 con 0x557e7a404020 2024-04-26T01:54:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd00701c0 con 0x557e7a403700 2024-04-26T01:54:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd006e050 con 0x557e7a41fa10 2024-04-26T01:54:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fd4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a403700 0x557e7a40b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fd4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a403700 0x557e7a40b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50038/0 (socket says 172.21.15.28:50038) 2024-04-26T01:54:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.145+0000 7f8fd5600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a41fa10 0x557e7a410db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.145+0000 7f8fd4c00640 1 -- 172.21.15.28:0/2526888168 learned_addr learned my addr 172.21.15.28:0/2526888168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.144+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 0x557e7a410870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.145+0000 7f8fd5600640 1 -- 172.21.15.28:0/2526888168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a403700 msgr2=0x557e7a40b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.145+0000 7f8fd5600640 1 --2- 172.21.15.28:0/2526888168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a403700 0x557e7a40b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.145+0000 7f8fd5600640 1 -- 172.21.15.28:0/2526888168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 msgr2=0x557e7a410870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.145+0000 7f8fd5600640 1 --2- 172.21.15.28:0/2526888168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 0x557e7a410870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.145+0000 7f8fd5600640 1 -- 172.21.15.28:0/2526888168 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e7a400ac0 con 0x557e7a41fa10 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.145+0000 7f8fcfe00640 1 --2- 172.21.15.28:0/2526888168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 0x557e7a410870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.146+0000 7f8fd5600640 1 --2- 172.21.15.28:0/2526888168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a41fa10 0x557e7a410db0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x557e7a3fadf0 tx=0x7f8fc802e2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1bb35966c8ce8a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.146+0000 7f8fc5200640 1 -- 172.21.15.28:0/2526888168 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8054070 con 0x557e7a41fa10 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.146+0000 7f8fc5200640 1 -- 172.21.15.28:0/2526888168 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc80054f0 con 0x557e7a41fa10 2024-04-26T01:54:50.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.146+0000 7f8fc5200640 1 -- 172.21.15.28:0/2526888168 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8043070 con 0x557e7a41fa10 2024-04-26T01:54:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 -- 172.21.15.28:0/2526888168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a41fa10 msgr2=0x557e7a410db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/2526888168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a41fa10 0x557e7a410db0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x557e7a3fadf0 tx=0x7f8fc802e2e0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 -- 172.21.15.28:0/2526888168 shutdown_connections 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/2526888168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a41fa10 0x557e7a410db0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/2526888168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a404020 0x557e7a410870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/2526888168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a403700 0x557e7a40b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 -- 172.21.15.28:0/2526888168 >> 172.21.15.28:0/2526888168 conn(0x557e7a3ffd30 msgr2=0x557e7a40a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 -- 172.21.15.28:0/2526888168 shutdown_connections 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.147+0000 7f8fda3e1740 1 -- 172.21.15.28:0/2526888168 wait complete. 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.148+0000 7f8fda3e1740 1 -- start start 2024-04-26T01:54:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.148+0000 7f8fda3e1740 1 Processor -- start 2024-04-26T01:54:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.148+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a403700 0x557e7a4264a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a404020 0x557e7a4269e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fda3e1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a483cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd006e6a0 con 0x557e7a403700 2024-04-26T01:54:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd006faf0 con 0x557e7a41fa10 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fda3e1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd006ed50 con 0x557e7a404020 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fda3e1740 10 client.0 _ll_register_callbacks cb 0x7f8fb80988c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fd4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a403700 0x557e7a4264a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a404020 0x557e7a4269e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fd5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a483cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fd5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a483cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:50040/0 (socket says 172.21.15.28:50040) 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.149+0000 7f8fd5600640 1 -- 172.21.15.28:0/3911097311 learned_addr learned my addr 172.21.15.28:0/3911097311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.150+0000 7f8fd5600640 1 -- 172.21.15.28:0/3911097311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a404020 msgr2=0x557e7a4269e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.150+0000 7f8fd5600640 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a404020 0x557e7a4269e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.150+0000 7f8fd5600640 1 -- 172.21.15.28:0/3911097311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a403700 msgr2=0x557e7a4264a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.150+0000 7f8fd5600640 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a403700 0x557e7a4264a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.150+0000 7f8fd5600640 1 -- 172.21.15.28:0/3911097311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e7a4841f0 con 0x557e7a41fa10 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.150+0000 7f8fd5600640 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a483cb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8fc802ef10 tx=0x7f8fc80317f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c1c391ab161c1b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.150+0000 7f8fbe000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.151+0000 7f8fbe000640 1 -- 172.21.15.28:0/3911097311 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8054070 con 0x557e7a41fa10 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.151+0000 7f8fbe000640 1 -- 172.21.15.28:0/3911097311 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8513+0+0 (secure 0 0 0) 0x7f8fc803f470 con 0x557e7a41fa10 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.151+0000 7f8fda3e1740 1 -- 172.21.15.28:0/3911097311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc80054f0 con 0x557e7a41fa10 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.151+0000 7f8fbe000640 1 -- 172.21.15.28:0/3911097311 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f8fc8043040 con 0x557e7a41fa10 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.151+0000 7f8fda3e1740 10 client.26357 fetch_fsmap learned FSMap version 46 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.151+0000 7f8fda3e1740 1 -- 172.21.15.28:0/3911097311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x557e7a485470 con 0x557e7a41fa10 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.151+0000 7f8fbe000640 1 -- 172.21.15.28:0/3911097311 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc80054f0 con 0x557e7a41fa10 2024-04-26T01:54:50.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.152+0000 7f8fbe000640 1 -- 172.21.15.28:0/3911097311 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f8fc804e030 con 0x557e7a41fa10 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.152+0000 7f8fda3e1740 10 client.26357 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.152+0000 7f8fda3e1740 10 client.26357 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.152+0000 7f8fda3e1740 20 client.26357 populate_metadata read hostname 'smithi028' 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.152+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557e7a4858c0 0x557e7a487ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.152+0000 7f8fda3e1740 4 client.26357 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.152+0000 7f8fda3e1740 1 -- 172.21.15.28:0/3911097311 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557e7a377320 con 0x557e7a4858c0 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.152+0000 7f8fd4c00640 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557e7a4858c0 0x557e7a487ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.153+0000 7f8fd4c00640 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557e7a4858c0 0x557e7a487ce0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71b4f685f877919d server_cookie=7ac4883d8e88a073 in_seq=0 out_seq=0 2024-04-26T01:54:50.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.153+0000 7f8fbe000640 10 client.26357 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.168+0000 7f8fbe000640 1 -- 172.21.15.28:0/3911097311 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557e7a377320 con 0x557e7a4858c0 2024-04-26T01:54:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.168+0000 7f8fbe000640 10 client.26357 handle_command_reply: tid=0 2024-04-26T01:54:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 client.26357 shutdown 2024-04-26T01:54:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 -- 172.21.15.28:0/3911097311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 msgr2=0x557e7a483cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a483cb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8fc802ef10 tx=0x7f8fc80317f0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 -- 172.21.15.28:0/3911097311 shutdown_connections 2024-04-26T01:54:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557e7a4858c0 0x557e7a487ce0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557e7a41fa10 0x557e7a483cb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fbe000640 0 client.26357 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:54:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557e7a404020 0x557e7a4269e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 --2- 172.21.15.28:0/3911097311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e7a403700 0x557e7a4264a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 -- 172.21.15.28:0/3911097311 >> 172.21.15.28:0/3911097311 conn(0x557e7a3ffd30 msgr2=0x557e7a3fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 -- 172.21.15.28:0/3911097311 shutdown_connections 2024-04-26T01:54:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 1 -- 172.21.15.28:0/3911097311 wait complete. 2024-04-26T01:54:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.169+0000 7f8fda3e1740 20 client.26357 trim_cache size 0 max 16384 2024-04-26T01:54:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:54:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:54:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:54:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:54:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.170+0000 7f8fd7600640 1 -- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8fb00a3fa0 msgr2=0x7f8fb00a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.170+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8fb00a3fa0 0x7f8fb00a6460 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8034010 tx=0x7f8fc8009000 comp rx=0 tx=0).stop 2024-04-26T01:54:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.170+0000 7f8fd7600640 1 -- 172.21.15.28:0/2266727809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fd01043a0 msgr2=0x7f8fd01a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.170+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fd01043a0 0x7f8fd01a05f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800ebc0 tx=0x7f8fb800c6a0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.171+0000 7f8fd7600640 1 -- 172.21.15.28:0/2266727809 shutdown_connections 2024-04-26T01:54:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.171+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8fb00a3fa0 0x7f8fb00a6460 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.171+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd019a700 0x7f8fd019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.171+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd0109470 0x7f8fd01a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.171+0000 7f8fd7600640 1 --2- 172.21.15.28:0/2266727809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fd01043a0 0x7f8fd01a05f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.171+0000 7f8fd7600640 1 -- 172.21.15.28:0/2266727809 >> 172.21.15.28:0/2266727809 conn(0x7f8fd00ffc40 msgr2=0x7f8fd0101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.171+0000 7f8fd7600640 1 -- 172.21.15.28:0/2266727809 shutdown_connections 2024-04-26T01:54:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:54:50.171+0000 7f8fd7600640 1 -- 172.21.15.28:0/2266727809 wait complete. 2024-04-26T01:54:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 775.0 s, 1235.6 tps, lat 81.220 ms stddev 94.523 2024-04-26T01:54:51.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:51 smithi028 ceph-mon[28247]: pgmap v435: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 1.85k op/s 2024-04-26T01:54:51.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:51 smithi169 ceph-mon[32851]: pgmap v435: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 1.85k op/s 2024-04-26T01:54:51.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:51 smithi053 ceph-mon[32593]: pgmap v435: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 1.85k op/s 2024-04-26T01:54:53.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:53 smithi028 ceph-mon[28247]: pgmap v436: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 1.59k op/s 2024-04-26T01:54:53.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:53 smithi169 ceph-mon[32851]: pgmap v436: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 1.59k op/s 2024-04-26T01:54:53.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:53 smithi053 ceph-mon[32593]: pgmap v436: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 1.59k op/s 2024-04-26T01:54:55.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:55 smithi028 ceph-mon[28247]: pgmap v437: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.67k op/s 2024-04-26T01:54:55.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:55 smithi169 ceph-mon[32851]: pgmap v437: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.67k op/s 2024-04-26T01:54:55.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:55 smithi053 ceph-mon[32593]: pgmap v437: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.67k op/s 2024-04-26T01:54:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 780.0 s, 1974.6 tps, lat 51.253 ms stddev 61.019 2024-04-26T01:54:57.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:57 smithi028 ceph-mon[28247]: pgmap v438: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-04-26T01:54:57.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:57 smithi169 ceph-mon[32851]: pgmap v438: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-04-26T01:54:57.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:57 smithi053 ceph-mon[32593]: pgmap v438: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-04-26T01:54:59.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:54:59 smithi169 ceph-mon[32851]: pgmap v439: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-26T01:54:59.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:54:59 smithi053 ceph-mon[32593]: pgmap v439: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-26T01:55:00.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:54:59 smithi028 ceph-mon[28247]: pgmap v439: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-26T01:55:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 785.0 s, 2212.8 tps, lat 40.791 ms stddev 49.568 2024-04-26T01:55:01.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:01 smithi169 ceph-mon[32851]: pgmap v440: 97 pgs: 97 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.3 MiB/s wr, 841 op/s 2024-04-26T01:55:01.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:01 smithi053 ceph-mon[32593]: pgmap v440: 97 pgs: 97 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.3 MiB/s wr, 841 op/s 2024-04-26T01:55:02.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:01 smithi028 ceph-mon[28247]: pgmap v440: 97 pgs: 97 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.3 MiB/s wr, 841 op/s 2024-04-26T01:55:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 -- 172.21.15.28:0/3739087977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f901043c0 msgr2=0x7f4f901047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 --2- 172.21.15.28:0/3739087977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f901043c0 0x7f4f901047a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009ca0 tx=0x7f4f8002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 -- 172.21.15.28:0/3739087977 shutdown_connections 2024-04-26T01:55:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 --2- 172.21.15.28:0/3739087977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f90109490 0x7f4f90111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 --2- 172.21.15.28:0/3739087977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90104d70 0x7f4f90108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 --2- 172.21.15.28:0/3739087977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f901043c0 0x7f4f901047a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 -- 172.21.15.28:0/3739087977 >> 172.21.15.28:0/3739087977 conn(0x7f4f900ffc80 msgr2=0x7f4f901020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 -- 172.21.15.28:0/3739087977 shutdown_connections 2024-04-26T01:55:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.327+0000 7f4f95a00640 1 -- 172.21.15.28:0/3739087977 wait complete. 2024-04-26T01:55:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.329+0000 7f4f95a00640 1 Processor -- start 2024-04-26T01:55:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.329+0000 7f4f95a00640 1 -- start start 2024-04-26T01:55:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.329+0000 7f4f95a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f901043c0 0x7f4f901a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.329+0000 7f4f95a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f90109490 0x7f4f901a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.329+0000 7f4f95a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f9019a710 0x7f4f9019ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.329+0000 7f4f95a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f90114450 con 0x7f4f901043c0 2024-04-26T01:55:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.329+0000 7f4f95a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901142d0 con 0x7f4f90109490 2024-04-26T01:55:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.329+0000 7f4f95a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901145d0 con 0x7f4f9019a710 2024-04-26T01:55:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f9019a710 0x7f4f9019ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f90109490 0x7f4f901a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f9019a710 0x7f4f9019ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:48688/0 (socket says 172.21.15.28:48688) 2024-04-26T01:55:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8f400640 1 -- 172.21.15.28:0/569086274 learned_addr learned my addr 172.21.15.28:0/569086274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f90109490 0x7f4f901a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:45142/0 (socket says 172.21.15.28:45142) 2024-04-26T01:55:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f901043c0 0x7f4f901a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8f400640 1 -- 172.21.15.28:0/569086274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f90109490 msgr2=0x7f4f901a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8f400640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f90109490 0x7f4f901a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8f400640 1 -- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f901043c0 msgr2=0x7f4f901a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8f400640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f901043c0 0x7f4f901a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.330+0000 7f4f8f400640 1 -- 172.21.15.28:0/569086274 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9019b310 con 0x7f4f9019a710 2024-04-26T01:55:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.331+0000 7f4f86000640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f90109490 0x7f4f901a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.331+0000 7f4f8f400640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f9019a710 0x7f4f9019ab90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800e970 tx=0x7f4f8800ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.332+0000 7f4f87e00640 1 -- 172.21.15.28:0/569086274 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8800ccc0 con 0x7f4f9019a710 2024-04-26T01:55:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.332+0000 7f4f87e00640 1 -- 172.21.15.28:0/569086274 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f8800ce60 con 0x7f4f9019a710 2024-04-26T01:55:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.332+0000 7f4f87e00640 1 -- 172.21.15.28:0/569086274 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88010430 con 0x7f4f9019a710 2024-04-26T01:55:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.332+0000 7f4f95a00640 1 -- 172.21.15.28:0/569086274 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f901a73f0 con 0x7f4f9019a710 2024-04-26T01:55:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.333+0000 7f4f95a00640 1 -- 172.21.15.28:0/569086274 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f901a77a0 con 0x7f4f9019a710 2024-04-26T01:55:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.337+0000 7f4f9896a740 1 Processor -- start 2024-04-26T01:55:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.337+0000 7f4f87e00640 1 -- 172.21.15.28:0/569086274 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4f88004f90 con 0x7f4f9019a710 2024-04-26T01:55:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f9896a740 1 -- start start 2024-04-26T01:55:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580db9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dba140 0x557580dc44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 0x557580dc6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f9896a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557580d1f900 con 0x557580dba140 2024-04-26T01:55:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f9896a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557580cc5ef0 con 0x557580dc4a00 2024-04-26T01:55:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f9896a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557580d1b8c0 con 0x557580db9820 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f86000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580db9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f86000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580db9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:48692/0 (socket says 172.21.15.28:48692) 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f86000640 1 -- 172.21.15.28:0/2593997420 learned_addr learned my addr 172.21.15.28:0/2593997420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.338+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dba140 0x557580dc44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.339+0000 7f4f86000640 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 0x557580dc6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.339+0000 7f4f86000640 1 -- 172.21.15.28:0/2593997420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 msgr2=0x557580dc6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.339+0000 7f4f86000640 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 0x557580dc6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.339+0000 7f4f86000640 1 -- 172.21.15.28:0/2593997420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dba140 msgr2=0x557580dc44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.339+0000 7f4f86000640 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dba140 0x557580dc44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.339+0000 7f4f86000640 1 -- 172.21.15.28:0/2593997420 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557580a32210 con 0x557580db9820 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.339+0000 7f4f86000640 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580db9c00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f78013520 tx=0x7f4f78009790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff3c721035db4237 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.340+0000 7f4f7fe00640 1 -- 172.21.15.28:0/2593997420 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78004030 con 0x557580db9820 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.340+0000 7f4f7fe00640 1 -- 172.21.15.28:0/2593997420 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f780035f0 con 0x557580db9820 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.340+0000 7f4f7fe00640 1 -- 172.21.15.28:0/2593997420 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78011760 con 0x557580db9820 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.340+0000 7f4f87e00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4f740a3f20 0x7f4f740a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.340+0000 7f4f87e00640 1 -- 172.21.15.28:0/569086274 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8557+0+0 (secure 0 0 0) 0x7f4f880d0b60 con 0x7f4f9019a710 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.340+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4f740a3f20 0x7f4f740a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 -- 172.21.15.28:0/2593997420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 msgr2=0x557580db9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580db9c00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f78013520 tx=0x7f4f78009790 comp rx=0 tx=0).stop 2024-04-26T01:55:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 -- 172.21.15.28:0/2593997420 shutdown_connections 2024-04-26T01:55:03.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 0x557580dc6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dba140 0x557580dc44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 --2- 172.21.15.28:0/2593997420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580db9c00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 -- 172.21.15.28:0/2593997420 >> 172.21.15.28:0/2593997420 conn(0x557580db5d30 msgr2=0x557580db6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:03.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4f740a3f20 0x7f4f740a63e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4f80006010 tx=0x7f4f80009820 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:03.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 -- 172.21.15.28:0/2593997420 shutdown_connections 2024-04-26T01:55:03.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.341+0000 7f4f9896a740 1 -- 172.21.15.28:0/2593997420 wait complete. 2024-04-26T01:55:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.342+0000 7f4f9896a740 1 -- start start 2024-04-26T01:55:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.342+0000 7f4f9896a740 1 Processor -- start 2024-04-26T01:55:03.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580dc19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dc4a00 0x557580dc2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f9896a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557580dc7630 con 0x557580dc4a00 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f9896a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557580cd9310 con 0x557580dba140 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f9896a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557580dc77b0 con 0x557580db9820 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f9896a740 10 client.0 _ll_register_callbacks cb 0x557580ca7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:45170/0 (socket says 172.21.15.28:45170) 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f8ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580dc19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.343+0000 7f4f86000640 1 -- 172.21.15.28:0/1623780477 learned_addr learned my addr 172.21.15.28:0/1623780477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f86000640 1 -- 172.21.15.28:0/1623780477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 msgr2=0x557580dc19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f86000640 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580dc19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f86000640 1 -- 172.21.15.28:0/1623780477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dc4a00 msgr2=0x557580dc2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f86000640 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dc4a00 0x557580dc2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f86000640 1 -- 172.21.15.28:0/1623780477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557580afe620 con 0x557580dba140 2024-04-26T01:55:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f86000640 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc1f00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4f78003a80 tx=0x7f4f78003ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cc6c13b2655bd48 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:55:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1623780477 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78029060 con 0x557580dba140 2024-04-26T01:55:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1623780477 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8557+0+0 (secure 0 0 0) 0x7f4f78005240 con 0x557580dba140 2024-04-26T01:55:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.344+0000 7f4f9896a740 1 -- 172.21.15.28:0/1623780477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f780035f0 con 0x557580dba140 2024-04-26T01:55:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.345+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1623780477 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f4f78013110 con 0x557580dba140 2024-04-26T01:55:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.345+0000 7f4f9896a740 10 client.26369 fetch_fsmap learned FSMap version 46 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.345+0000 7f4f9896a740 1 -- 172.21.15.28:0/1623780477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x557580dd38c0 con 0x557580dba140 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.345+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1623780477 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f780035f0 con 0x557580dba140 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.345+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1623780477 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f4f7801e070 con 0x557580dba140 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.346+0000 7f4f9896a740 10 client.26369 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.346+0000 7f4f9896a740 10 client.26369 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.346+0000 7f4f9896a740 20 client.26369 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.346+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557580dd4cc0 0x557580dd70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.346+0000 7f4f9896a740 4 client.26369 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.346+0000 7f4f9896a740 1 -- 172.21.15.28:0/1623780477 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557580d338b0 con 0x557580dd4cc0 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.346+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557580dd4cc0 0x557580dd70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.347+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557580dd4cc0 0x557580dd70b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fea118e80a540408 server_cookie=5ea60f30d5c7ded9 in_seq=0 out_seq=0 2024-04-26T01:55:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.347+0000 7f4f7cc00640 10 client.26369 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1623780477 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557580d338b0 con 0x557580dd4cc0 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f7cc00640 10 client.26369 handle_command_reply: tid=0 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 client.26369 shutdown 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 -- 172.21.15.28:0/1623780477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 msgr2=0x557580dc1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc1f00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4f78003a80 tx=0x7f4f78003ab0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 -- 172.21.15.28:0/1623780477 shutdown_connections 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557580dd4cc0 0x557580dd70b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580dc4a00 0x557580dc2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc1f00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1623780477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580db9820 0x557580dc19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 -- 172.21.15.28:0/1623780477 >> 172.21.15.28:0/1623780477 conn(0x557580db5d30 msgr2=0x557580dbbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f7cc00640 0 client.26369 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 -- 172.21.15.28:0/1623780477 shutdown_connections 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 1 -- 172.21.15.28:0/1623780477 wait complete. 2024-04-26T01:55:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.348+0000 7f4f9896a740 20 client.26369 trim_cache size 0 max 16384 2024-04-26T01:55:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f9896a740 1 Processor -- start 2024-04-26T01:55:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f9896a740 1 -- start start 2024-04-26T01:55:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 0x557580dd04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dba140 0x557580db5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 0x557580ddc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f9896a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901b26d0 con 0x557580db9820 2024-04-26T01:55:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f9896a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901aff40 con 0x557580dc4a00 2024-04-26T01:55:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f9896a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901addb0 con 0x557580dba140 2024-04-26T01:55:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f86000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dba140 0x557580db5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f8ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 0x557580dd04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.366+0000 7f4f86000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dba140 0x557580db5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:48700/0 (socket says 172.21.15.28:48700) 2024-04-26T01:55:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f86000640 1 -- 172.21.15.28:0/495582912 learned_addr learned my addr 172.21.15.28:0/495582912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f8f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 0x557580ddc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f86000640 1 -- 172.21.15.28:0/495582912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 msgr2=0x557580ddc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f86000640 1 --2- 172.21.15.28:0/495582912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 0x557580ddc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f86000640 1 -- 172.21.15.28:0/495582912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 msgr2=0x557580dd04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f86000640 1 --2- 172.21.15.28:0/495582912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 0x557580dd04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f86000640 1 -- 172.21.15.28:0/495582912 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557580dbaea0 con 0x557580dba140 2024-04-26T01:55:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f86000640 1 --2- 172.21.15.28:0/495582912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dba140 0x557580db5610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4f78004ce0 tx=0x7f4f78013330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb1736d355c7d6eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.367+0000 7f4f7f400640 1 -- 172.21.15.28:0/495582912 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78029040 con 0x557580dba140 2024-04-26T01:55:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.368+0000 7f4f7f400640 1 -- 172.21.15.28:0/495582912 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f780035f0 con 0x557580dba140 2024-04-26T01:55:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.368+0000 7f4f7f400640 1 -- 172.21.15.28:0/495582912 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78004410 con 0x557580dba140 2024-04-26T01:55:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.368+0000 7f4f9896a740 1 -- 172.21.15.28:0/495582912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dba140 msgr2=0x557580db5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.368+0000 7f4f9896a740 1 --2- 172.21.15.28:0/495582912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dba140 0x557580db5610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4f78004ce0 tx=0x7f4f78013330 comp rx=0 tx=0).stop 2024-04-26T01:55:03.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 -- 172.21.15.28:0/495582912 shutdown_connections 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 --2- 172.21.15.28:0/495582912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dc4a00 0x557580ddc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 --2- 172.21.15.28:0/495582912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dba140 0x557580db5610 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 --2- 172.21.15.28:0/495582912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 0x557580dd04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 -- 172.21.15.28:0/495582912 >> 172.21.15.28:0/495582912 conn(0x557580db5d30 msgr2=0x557580dd6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 -- 172.21.15.28:0/495582912 shutdown_connections 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 -- 172.21.15.28:0/495582912 wait complete. 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 -- start start 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.369+0000 7f4f9896a740 1 Processor -- start 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 0x557580dc99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f9896a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dc4a00 0x557580e39130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f9896a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901ae400 con 0x557580db9820 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f9896a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901af870 con 0x557580dba140 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f9896a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901aead0 con 0x557580dc4a00 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f9896a740 10 client.0 _ll_register_callbacks cb 0x7f4f78004360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:45190/0 (socket says 172.21.15.28:45190) 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f86000640 1 -- 172.21.15.28:0/1585518842 learned_addr learned my addr 172.21.15.28:0/1585518842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f8f400640 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dc4a00 0x557580e39130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.370+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 0x557580dc99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f86000640 1 -- 172.21.15.28:0/1585518842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dc4a00 msgr2=0x557580e39130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f86000640 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dc4a00 0x557580e39130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f86000640 1 -- 172.21.15.28:0/1585518842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 msgr2=0x557580dc99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f86000640 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 0x557580dc99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f86000640 1 -- 172.21.15.28:0/1585518842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557580dca7a0 con 0x557580dba140 2024-04-26T01:55:03.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f86000640 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc9f10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4f78005860 tx=0x7f4f78004f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7674e6c4100a81f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1585518842 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7803d020 con 0x557580dba140 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1585518842 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8557+0+0 (secure 0 0 0) 0x7f4f78004410 con 0x557580dba140 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.371+0000 7f4f9896a740 1 -- 172.21.15.28:0/1585518842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f780035f0 con 0x557580dba140 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1585518842 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f4f78048050 con 0x557580dba140 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f9896a740 10 client.26375 fetch_fsmap learned FSMap version 46 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f9896a740 1 -- 172.21.15.28:0/1585518842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x557580e3a750 con 0x557580dba140 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1585518842 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f780035f0 con 0x557580dba140 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1585518842 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7f4f7801e040 con 0x557580dba140 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f9896a740 10 client.26375 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f9896a740 10 client.26375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f9896a740 20 client.26375 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557580e3ae10 0x557580e3d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f9896a740 4 client.26375 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.372+0000 7f4f9896a740 1 -- 172.21.15.28:0/1585518842 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557580d2d320 con 0x557580e3ae10 2024-04-26T01:55:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.373+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557580e3ae10 0x557580e3d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.373+0000 7f4f8ea00640 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557580e3ae10 0x557580e3d230 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa8ebc1457839791 server_cookie=d1f282d4490ff6c5 in_seq=0 out_seq=0 2024-04-26T01:55:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.373+0000 7f4f7cc00640 10 client.26375 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:03.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.398+0000 7f4f7cc00640 1 -- 172.21.15.28:0/1585518842 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557580d2d320 con 0x557580e3ae10 2024-04-26T01:55:03.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.398+0000 7f4f7cc00640 10 client.26375 handle_command_reply: tid=0 2024-04-26T01:55:03.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.398+0000 7f4f9896a740 1 client.26375 shutdown 2024-04-26T01:55:03.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.398+0000 7f4f9896a740 1 -- 172.21.15.28:0/1585518842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 msgr2=0x557580dc9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.398+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc9f10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4f78005860 tx=0x7f4f78004f90 comp rx=0 tx=0).stop 2024-04-26T01:55:03.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.399+0000 7f4f9896a740 1 -- 172.21.15.28:0/1585518842 shutdown_connections 2024-04-26T01:55:03.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.399+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557580e3ae10 0x557580e3d230 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.399+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557580dc4a00 0x557580e39130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.399+0000 7f4f7cc00640 0 client.26375 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.399+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557580dba140 0x557580dc9f10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.399+0000 7f4f9896a740 1 --2- 172.21.15.28:0/1585518842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557580db9820 0x557580dc99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.399+0000 7f4f9896a740 1 -- 172.21.15.28:0/1585518842 >> 172.21.15.28:0/1585518842 conn(0x557580db5d30 msgr2=0x557580dc1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.400+0000 7f4f9896a740 1 -- 172.21.15.28:0/1585518842 shutdown_connections 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.400+0000 7f4f9896a740 1 -- 172.21.15.28:0/1585518842 wait complete. 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.400+0000 7f4f9896a740 20 client.26375 trim_cache size 0 max 16384 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:55:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:55:03.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:55:03.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:55:03.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 -- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4f740a3f20 msgr2=0x7f4f740a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4f740a3f20 0x7f4f740a63e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4f80006010 tx=0x7f4f80009820 comp rx=0 tx=0).stop 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 -- 172.21.15.28:0/569086274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f9019a710 msgr2=0x7f4f9019ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f9019a710 0x7f4f9019ab90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800e970 tx=0x7f4f8800ed70 comp rx=0 tx=0).stop 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 -- 172.21.15.28:0/569086274 shutdown_connections 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f4f740a3f20 0x7f4f740a63e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f9019a710 0x7f4f9019ab90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f90109490 0x7f4f901a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 --2- 172.21.15.28:0/569086274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f901043c0 0x7f4f901a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 -- 172.21.15.28:0/569086274 >> 172.21.15.28:0/569086274 conn(0x7f4f900ffc80 msgr2=0x7f4f90102000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 -- 172.21.15.28:0/569086274 shutdown_connections 2024-04-26T01:55:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:03.401+0000 7f4f95a00640 1 -- 172.21.15.28:0/569086274 wait complete. 2024-04-26T01:55:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:03 smithi169 ceph-mon[32851]: pgmap v441: 97 pgs: 97 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.6 MiB/s wr, 555 op/s 2024-04-26T01:55:03.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:03 smithi169 ceph-mon[32851]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:55:03.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:03 smithi053 ceph-mon[32593]: pgmap v441: 97 pgs: 97 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.6 MiB/s wr, 555 op/s 2024-04-26T01:55:03.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:03 smithi053 ceph-mon[32593]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:55:04.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:03 smithi028 ceph-mon[28247]: pgmap v441: 97 pgs: 97 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.6 MiB/s wr, 555 op/s 2024-04-26T01:55:04.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:03 smithi028 ceph-mon[28247]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:55:05.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:05 smithi028 ceph-mon[28247]: pgmap v443: 97 pgs: 97 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.6 MiB/s wr, 640 op/s 2024-04-26T01:55:05.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:05 smithi169 ceph-mon[32851]: pgmap v443: 97 pgs: 97 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.6 MiB/s wr, 640 op/s 2024-04-26T01:55:05.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:05 smithi053 ceph-mon[32593]: pgmap v443: 97 pgs: 97 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.6 MiB/s wr, 640 op/s 2024-04-26T01:55:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 790.0 s, 122.6 tps, lat 848.720 ms stddev 492.874 2024-04-26T01:55:07.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:07 smithi169 ceph-mon[32851]: pgmap v444: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:55:07.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:07 smithi053 ceph-mon[32593]: pgmap v444: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:55:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:07 smithi028 ceph-mon[28247]: pgmap v444: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:55:09.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:09 smithi169 ceph-mon[32851]: pgmap v445: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:55:09.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:09 smithi169 ceph-mon[32851]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:55:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:09 smithi053 ceph-mon[32593]: pgmap v445: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:55:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:09 smithi053 ceph-mon[32593]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:55:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:09 smithi028 ceph-mon[28247]: pgmap v445: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-26T01:55:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:09 smithi028 ceph-mon[28247]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:55:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 795.0 s, 765.4 tps, lat 137.626 ms stddev 377.121 2024-04-26T01:55:11.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:11 smithi169 ceph-mon[32851]: pgmap v447: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 36 MiB/s wr, 1.94k op/s 2024-04-26T01:55:11.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:11 smithi053 ceph-mon[32593]: pgmap v447: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 36 MiB/s wr, 1.94k op/s 2024-04-26T01:55:11.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:11 smithi028 ceph-mon[28247]: pgmap v447: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 36 MiB/s wr, 1.94k op/s 2024-04-26T01:55:13.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:13 smithi169 ceph-mon[32851]: pgmap v448: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 30 MiB/s wr, 1.59k op/s 2024-04-26T01:55:13.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:13 smithi053 ceph-mon[32593]: pgmap v448: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 30 MiB/s wr, 1.59k op/s 2024-04-26T01:55:13.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:13 smithi028 ceph-mon[28247]: pgmap v448: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 30 MiB/s wr, 1.59k op/s 2024-04-26T01:55:15.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:15 smithi053 ceph-mon[32593]: pgmap v449: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 29 MiB/s wr, 1.60k op/s 2024-04-26T01:55:15.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:15 smithi028 ceph-mon[28247]: pgmap v449: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 29 MiB/s wr, 1.60k op/s 2024-04-26T01:55:15.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:15 smithi169 ceph-mon[32851]: pgmap v449: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 29 MiB/s wr, 1.60k op/s 2024-04-26T01:55:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 800.0 s, 1929.7 tps, lat 51.542 ms stddev 64.511 2024-04-26T01:55:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 -- 172.21.15.28:0/2750069675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc3c06acc0 msgr2=0x7fcc3c06b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 --2- 172.21.15.28:0/2750069675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc3c06acc0 0x7fcc3c06b140 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009900 tx=0x7fcc2802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 -- 172.21.15.28:0/2750069675 shutdown_connections 2024-04-26T01:55:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 --2- 172.21.15.28:0/2750069675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc3c10e8a0 0x7fcc3c110c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 --2- 172.21.15.28:0/2750069675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc3c06acc0 0x7fcc3c06b140 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 --2- 172.21.15.28:0/2750069675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c102f00 0x7fcc3c06a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 -- 172.21.15.28:0/2750069675 >> 172.21.15.28:0/2750069675 conn(0x7fcc3c076c50 msgr2=0x7fcc3c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 -- 172.21.15.28:0/2750069675 shutdown_connections 2024-04-26T01:55:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.595+0000 7fcc40a00640 1 -- 172.21.15.28:0/2750069675 wait complete. 2024-04-26T01:55:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.597+0000 7fcc40a00640 1 Processor -- start 2024-04-26T01:55:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.597+0000 7fcc40a00640 1 -- start start 2024-04-26T01:55:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.598+0000 7fcc40a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc3c06acc0 0x7fcc3c103480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.598+0000 7fcc40a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c102f00 0x7fcc3c1039c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.598+0000 7fcc40a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc3c10e8a0 0x7fcc3c103f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc39400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c102f00 0x7fcc3c1039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc39400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c102f00 0x7fcc3c1039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35592/0 (socket says 172.21.15.28:35592) 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc39400640 1 -- 172.21.15.28:0/555534282 learned_addr learned my addr 172.21.15.28:0/555534282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c1144e0 con 0x7fcc3c102f00 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c114360 con 0x7fcc3c06acc0 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c114660 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc3a800640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc3c10e8a0 0x7fcc3c103f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc39e00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc3c06acc0 0x7fcc3c103480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc3a800640 1 -- 172.21.15.28:0/555534282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc3c06acc0 msgr2=0x7fcc3c103480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc3a800640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc3c06acc0 0x7fcc3c103480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc3a800640 1 -- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c102f00 msgr2=0x7fcc3c1039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc3a800640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c102f00 0x7fcc3c1039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.599+0000 7fcc3a800640 1 -- 172.21.15.28:0/555534282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3c1aafc0 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.600+0000 7fcc3a800640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc3c10e8a0 0x7fcc3c103f00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400ea80 tx=0x7fcc3400ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.600+0000 7fcc32a00640 1 -- 172.21.15.28:0/555534282 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3400cea0 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.600+0000 7fcc32a00640 1 -- 172.21.15.28:0/555534282 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc34005020 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.600+0000 7fcc32a00640 1 -- 172.21.15.28:0/555534282 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34010690 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.600+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3c1ab2b0 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.600+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc3c1ab780 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.607+0000 7fcc437eb740 1 Processor -- start 2024-04-26T01:55:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.607+0000 7fcc32a00640 1 -- 172.21.15.28:0/555534282 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcc340054d0 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.607+0000 7fcc437eb740 1 -- start start 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf1720 0x55ac08cf1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d0d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc437eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ac08c57900 con 0x55ac08cf2040 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc437eb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ac08bfdef0 con 0x55ac08d0da10 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc437eb740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55ac08c538c0 con 0x55ac08cf1720 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc39e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d0d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc39e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d0d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35608/0 (socket says 172.21.15.28:35608) 2024-04-26T01:55:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc39e00640 1 -- 172.21.15.28:0/4230335114 learned_addr learned my addr 172.21.15.28:0/4230335114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.608+0000 7fcc39400640 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf1720 0x55ac08cf1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.609+0000 7fcc39400640 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.609+0000 7fcc39400640 1 -- 172.21.15.28:0/4230335114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 msgr2=0x55ac08d0fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.609+0000 7fcc39400640 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.609+0000 7fcc39400640 1 -- 172.21.15.28:0/4230335114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 msgr2=0x55ac08d0d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.609+0000 7fcc39400640 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d0d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.609+0000 7fcc39400640 1 -- 172.21.15.28:0/4230335114 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac0896a210 con 0x55ac08cf1720 2024-04-26T01:55:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.610+0000 7fcc39400640 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf1720 0x55ac08cf1b00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcc28000c00 tx=0x7fcc28005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a097b2d60c7469b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.610+0000 7fcc32a00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc200a3ec0 0x7fcc200a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.610+0000 7fcc32a00640 1 -- 172.21.15.28:0/555534282 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7fcc340d0d60 con 0x7fcc3c10e8a0 2024-04-26T01:55:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.610+0000 7fcc39e00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc200a3ec0 0x7fcc200a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.612+0000 7fcc27000640 1 -- 172.21.15.28:0/4230335114 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28035040 con 0x55ac08cf1720 2024-04-26T01:55:16.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.612+0000 7fcc27000640 1 -- 172.21.15.28:0/4230335114 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc28003810 con 0x55ac08cf1720 2024-04-26T01:55:16.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.612+0000 7fcc27000640 1 -- 172.21.15.28:0/4230335114 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28004a00 con 0x55ac08cf1720 2024-04-26T01:55:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.613+0000 7fcc437eb740 1 -- 172.21.15.28:0/4230335114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf1720 msgr2=0x55ac08cf1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.613+0000 7fcc437eb740 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf1720 0x55ac08cf1b00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcc28000c00 tx=0x7fcc28005a20 comp rx=0 tx=0).stop 2024-04-26T01:55:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.614+0000 7fcc39e00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc200a3ec0 0x7fcc200a6380 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55ac08cf4410 tx=0x7fcc2c00b410 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.620+0000 7fcc437eb740 1 -- 172.21.15.28:0/4230335114 shutdown_connections 2024-04-26T01:55:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.620+0000 7fcc437eb740 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.620+0000 7fcc437eb740 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d0d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.620+0000 7fcc437eb740 1 --2- 172.21.15.28:0/4230335114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf1720 0x55ac08cf1b00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.620+0000 7fcc437eb740 1 -- 172.21.15.28:0/4230335114 >> 172.21.15.28:0/4230335114 conn(0x55ac08cedd30 msgr2=0x55ac08cee110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.622+0000 7fcc437eb740 1 -- 172.21.15.28:0/4230335114 shutdown_connections 2024-04-26T01:55:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.622+0000 7fcc437eb740 1 -- 172.21.15.28:0/4230335114 wait complete. 2024-04-26T01:55:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.622+0000 7fcc437eb740 1 -- start start 2024-04-26T01:55:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.623+0000 7fcc437eb740 1 Processor -- start 2024-04-26T01:55:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf1720 0x55ac08d0a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d0ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc437eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ac08d105e0 con 0x55ac08cf1720 2024-04-26T01:55:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc437eb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ac08c11310 con 0x55ac08d0da10 2024-04-26T01:55:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc437eb740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55ac08d10760 con 0x55ac08cf2040 2024-04-26T01:55:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc437eb740 10 client.0 _ll_register_callbacks cb 0x55ac08bdfcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc39400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d0ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc39400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d0ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:57632/0 (socket says 172.21.15.28:57632) 2024-04-26T01:55:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.624+0000 7fcc39400640 1 -- 172.21.15.28:0/2557921836 learned_addr learned my addr 172.21.15.28:0/2557921836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.625+0000 7fcc39e00640 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf1720 0x55ac08d0a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.625+0000 7fcc39e00640 1 -- 172.21.15.28:0/2557921836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 msgr2=0x55ac08d0ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.625+0000 7fcc39e00640 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d0ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.626+0000 7fcc3a800640 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.626+0000 7fcc39e00640 1 -- 172.21.15.28:0/2557921836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 msgr2=0x55ac08d0b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.626+0000 7fcc39e00640 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.626+0000 7fcc39e00640 1 -- 172.21.15.28:0/2557921836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac08a36620 con 0x55ac08cf1720 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.626+0000 7fcc3a800640 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.626+0000 7fcc39e00640 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf1720 0x55ac08d0a900 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c010a70 tx=0x7fcc2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8880200508d9e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.627+0000 7fcc1b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.627+0000 7fcc1b400640 1 -- 172.21.15.28:0/2557921836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c004220 con 0x55ac08cf1720 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.627+0000 7fcc437eb740 1 -- 172.21.15.28:0/2557921836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc28003810 con 0x55ac08cf1720 2024-04-26T01:55:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.627+0000 7fcc1b400640 1 -- 172.21.15.28:0/2557921836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7fcc2c004020 con 0x55ac08cf1720 2024-04-26T01:55:16.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.631+0000 7fcc1b400640 1 -- 172.21.15.28:0/2557921836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fcc2c004220 con 0x55ac08cf1720 2024-04-26T01:55:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.631+0000 7fcc437eb740 10 client.16113 fetch_fsmap learned FSMap version 47 2024-04-26T01:55:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.631+0000 7fcc437eb740 1 -- 172.21.15.28:0/2557921836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ac08d2d930 con 0x55ac08cf1720 2024-04-26T01:55:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.634+0000 7fcc1b400640 1 -- 172.21.15.28:0/2557921836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc28003810 con 0x55ac08cf1720 2024-04-26T01:55:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.634+0000 7fcc1b400640 1 -- 172.21.15.28:0/2557921836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fcc2c024030 con 0x55ac08cf1720 2024-04-26T01:55:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.635+0000 7fcc437eb740 10 client.16113 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:55:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.635+0000 7fcc437eb740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.635+0000 7fcc437eb740 20 client.16113 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.635+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ac08d2eda0 0x55ac08d311d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.635+0000 7fcc437eb740 4 client.16113 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.635+0000 7fcc437eb740 1 -- 172.21.15.28:0/2557921836 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac08c6b8b0 con 0x55ac08d2eda0 2024-04-26T01:55:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.639+0000 7fcc39400640 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ac08d2eda0 0x55ac08d311d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.643+0000 7fcc1b400640 10 client.16113 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.643+0000 7fcc39400640 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ac08d2eda0 0x55ac08d311d0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=932295603a71f1dd server_cookie=de98b2b3fb9938d9 in_seq=0 out_seq=0 2024-04-26T01:55:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.645+0000 7fcc1b400640 1 -- 172.21.15.28:0/2557921836 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac08c6b8b0 con 0x55ac08d2eda0 2024-04-26T01:55:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.645+0000 7fcc1b400640 10 client.16113 handle_command_reply: tid=0 2024-04-26T01:55:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.645+0000 7fcc437eb740 1 client.16113 shutdown 2024-04-26T01:55:16.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 -- 172.21.15.28:0/2557921836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf1720 msgr2=0x55ac08d0a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf1720 0x55ac08d0a900 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c010a70 tx=0x7fcc2c0046c0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 -- 172.21.15.28:0/2557921836 shutdown_connections 2024-04-26T01:55:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ac08d2eda0 0x55ac08d311d0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d0da10 0x55ac08d0b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d0ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2557921836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf1720 0x55ac08d0a900 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 -- 172.21.15.28:0/2557921836 >> 172.21.15.28:0/2557921836 conn(0x55ac08cedd30 msgr2=0x55ac08cf3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc1b400640 0 client.16113 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.647+0000 7fcc437eb740 1 -- 172.21.15.28:0/2557921836 shutdown_connections 2024-04-26T01:55:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.648+0000 7fcc437eb740 1 -- 172.21.15.28:0/2557921836 wait complete. 2024-04-26T01:55:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.648+0000 7fcc437eb740 20 client.16113 trim_cache size 0 max 16384 2024-04-26T01:55:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.684+0000 7fcc437eb740 1 Processor -- start 2024-04-26T01:55:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.685+0000 7fcc437eb740 1 -- start start 2024-04-26T01:55:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.685+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08cf1720 0x55ac08d19530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.686+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d36ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.686+0000 7fcc39e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08cf1720 0x55ac08d19530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.686+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08d0da10 0x55ac08d37020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.686+0000 7fcc437eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c06e190 con 0x55ac08cf2040 2024-04-26T01:55:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.686+0000 7fcc437eb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c06ba60 con 0x55ac08cf1720 2024-04-26T01:55:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.686+0000 7fcc437eb740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c101480 con 0x55ac08d0da10 2024-04-26T01:55:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.686+0000 7fcc39e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08cf1720 0x55ac08d19530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:52080/0 (socket says 172.21.15.28:52080) 2024-04-26T01:55:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.686+0000 7fcc39e00640 1 -- 172.21.15.28:0/3187805737 learned_addr learned my addr 172.21.15.28:0/3187805737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.687+0000 7fcc39e00640 1 -- 172.21.15.28:0/3187805737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08d0da10 msgr2=0x55ac08d37020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.687+0000 7fcc39e00640 1 --2- 172.21.15.28:0/3187805737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08d0da10 0x55ac08d37020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.687+0000 7fcc39400640 1 --2- 172.21.15.28:0/3187805737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d36ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.687+0000 7fcc39e00640 1 -- 172.21.15.28:0/3187805737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 msgr2=0x55ac08d36ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.687+0000 7fcc39e00640 1 --2- 172.21.15.28:0/3187805737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d36ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.687+0000 7fcc39e00640 1 -- 172.21.15.28:0/3187805737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac08d098b0 con 0x55ac08cf1720 2024-04-26T01:55:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.689+0000 7fcc39e00640 1 --2- 172.21.15.28:0/3187805737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08cf1720 0x55ac08d19530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c011010 tx=0x7fcc2c00bb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36431144013b0e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.690+0000 7fcc26600640 1 -- 172.21.15.28:0/3187805737 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c03c070 con 0x55ac08cf1720 2024-04-26T01:55:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.690+0000 7fcc26600640 1 -- 172.21.15.28:0/3187805737 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc2c003e00 con 0x55ac08cf1720 2024-04-26T01:55:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.690+0000 7fcc26600640 1 -- 172.21.15.28:0/3187805737 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c01b670 con 0x55ac08cf1720 2024-04-26T01:55:16.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.694+0000 7fcc437eb740 1 -- 172.21.15.28:0/3187805737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08cf1720 msgr2=0x55ac08d19530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.695+0000 7fcc437eb740 1 --2- 172.21.15.28:0/3187805737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08cf1720 0x55ac08d19530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c011010 tx=0x7fcc2c00bb40 comp rx=0 tx=0).stop 2024-04-26T01:55:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.695+0000 7fcc437eb740 1 -- 172.21.15.28:0/3187805737 shutdown_connections 2024-04-26T01:55:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.695+0000 7fcc437eb740 1 --2- 172.21.15.28:0/3187805737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08d0da10 0x55ac08d37020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.695+0000 7fcc437eb740 1 --2- 172.21.15.28:0/3187805737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08cf2040 0x55ac08d36ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.695+0000 7fcc437eb740 1 --2- 172.21.15.28:0/3187805737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08cf1720 0x55ac08d19530 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.695+0000 7fcc437eb740 1 -- 172.21.15.28:0/3187805737 >> 172.21.15.28:0/3187805737 conn(0x55ac08cedd30 msgr2=0x55ac08cee110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.695+0000 7fcc437eb740 1 -- 172.21.15.28:0/3187805737 shutdown_connections 2024-04-26T01:55:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.695+0000 7fcc437eb740 1 -- 172.21.15.28:0/3187805737 wait complete. 2024-04-26T01:55:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.696+0000 7fcc437eb740 1 -- start start 2024-04-26T01:55:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.696+0000 7fcc437eb740 1 Processor -- start 2024-04-26T01:55:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.699+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d13a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.699+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08d0da10 0x55ac08d13f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc437eb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d92350 0x55ac08d94740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc39e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d13a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc39e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d13a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:57644/0 (socket says 172.21.15.28:57644) 2024-04-26T01:55:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc39e00640 1 -- 172.21.15.28:0/2531418114 learned_addr learned my addr 172.21.15.28:0/2531418114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc39400640 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08d0da10 0x55ac08d13f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc3a800640 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d92350 0x55ac08d94740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c1a1460 con 0x55ac08d0da10 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c1a28d0 con 0x55ac08d92350 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.700+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c1a1b30 con 0x55ac08cf2040 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.701+0000 7fcc437eb740 10 client.0 _ll_register_callbacks cb 0x7fcc2c00ba90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.701+0000 7fcc39400640 1 -- 172.21.15.28:0/2531418114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 msgr2=0x55ac08d13a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.701+0000 7fcc39400640 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d13a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.701+0000 7fcc39400640 1 -- 172.21.15.28:0/2531418114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d92350 msgr2=0x55ac08d94740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.701+0000 7fcc39400640 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d92350 0x55ac08d94740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.701+0000 7fcc39400640 1 -- 172.21.15.28:0/2531418114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac08cf1e70 con 0x55ac08d0da10 2024-04-26T01:55:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.702+0000 7fcc3a800640 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d92350 0x55ac08d94740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:55:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.702+0000 7fcc39400640 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08d0da10 0x55ac08d13f40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55ac08d0a160 tx=0x7fcc2802ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cb4fe260a9ce42 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.702+0000 7fcc1b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:55:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.703+0000 7fcc1b400640 1 -- 172.21.15.28:0/2531418114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28054020 con 0x55ac08d0da10 2024-04-26T01:55:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.703+0000 7fcc1b400640 1 -- 172.21.15.28:0/2531418114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7fcc28003b20 con 0x55ac08d0da10 2024-04-26T01:55:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.703+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ac08cf2d90 con 0x55ac08d0da10 2024-04-26T01:55:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.704+0000 7fcc1b400640 1 -- 172.21.15.28:0/2531418114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fcc28035040 con 0x55ac08d0da10 2024-04-26T01:55:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.704+0000 7fcc437eb740 10 client.16116 fetch_fsmap learned FSMap version 47 2024-04-26T01:55:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.704+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ac08d95cc0 con 0x55ac08d0da10 2024-04-26T01:55:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.704+0000 7fcc1b400640 1 -- 172.21.15.28:0/2531418114 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ac08cf2d90 con 0x55ac08d0da10 2024-04-26T01:55:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.705+0000 7fcc1b400640 1 -- 172.21.15.28:0/2531418114 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fcc280314a0 con 0x55ac08d0da10 2024-04-26T01:55:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.706+0000 7fcc437eb740 10 client.16116 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:55:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.706+0000 7fcc437eb740 10 client.16116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.706+0000 7fcc437eb740 20 client.16116 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.706+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ac08d95ef0 0x55ac08d982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.706+0000 7fcc437eb740 4 client.16116 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.706+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac08c65320 con 0x55ac08d95ef0 2024-04-26T01:55:16.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.706+0000 7fcc39e00640 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ac08d95ef0 0x55ac08d982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.708+0000 7fcc39e00640 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ac08d95ef0 0x55ac08d982e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0b19f2df7cf4fbb server_cookie=814d957593eaa4a4 in_seq=0 out_seq=0 2024-04-26T01:55:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.708+0000 7fcc1b400640 10 client.16116 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.733+0000 7fcc1b400640 1 -- 172.21.15.28:0/2531418114 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac08c65320 con 0x55ac08d95ef0 2024-04-26T01:55:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.733+0000 7fcc1b400640 10 client.16116 handle_command_reply: tid=0 2024-04-26T01:55:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.733+0000 7fcc437eb740 1 client.16116 shutdown 2024-04-26T01:55:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.734+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08d0da10 msgr2=0x55ac08d13f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.734+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08d0da10 0x55ac08d13f40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55ac08d0a160 tx=0x7fcc2802ab90 comp rx=0 tx=0).stop 2024-04-26T01:55:16.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 shutdown_connections 2024-04-26T01:55:16.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55ac08d95ef0 0x55ac08d982e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ac08d92350 0x55ac08d94740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac08d0da10 0x55ac08d13f40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc437eb740 1 --2- 172.21.15.28:0/2531418114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ac08cf2040 0x55ac08d13a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 >> 172.21.15.28:0/2531418114 conn(0x55ac08cedd30 msgr2=0x55ac08cec600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc1b400640 0 client.16116 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 shutdown_connections 2024-04-26T01:55:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.735+0000 7fcc437eb740 1 -- 172.21.15.28:0/2531418114 wait complete. 2024-04-26T01:55:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.736+0000 7fcc437eb740 20 client.16116 trim_cache size 0 max 16384 2024-04-26T01:55:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:55:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:55:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:55:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.736+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc200a3ec0 msgr2=0x7fcc200a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.736+0000 7fcc40a00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc200a3ec0 0x7fcc200a6380 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55ac08cf4410 tx=0x7fcc2c00b410 comp rx=0 tx=0).stop 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.736+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc3c10e8a0 msgr2=0x7fcc3c103f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.736+0000 7fcc40a00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc3c10e8a0 0x7fcc3c103f00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400ea80 tx=0x7fcc3400ef40 comp rx=0 tx=0).stop 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.736+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 shutdown_connections 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.737+0000 7fcc40a00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc200a3ec0 0x7fcc200a6380 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.737+0000 7fcc40a00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc3c10e8a0 0x7fcc3c103f00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.737+0000 7fcc40a00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc3c102f00 0x7fcc3c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.737+0000 7fcc40a00640 1 --2- 172.21.15.28:0/555534282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc3c06acc0 0x7fcc3c103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.737+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 >> 172.21.15.28:0/555534282 conn(0x7fcc3c076c50 msgr2=0x7fcc3c100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.737+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 shutdown_connections 2024-04-26T01:55:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:16.737+0000 7fcc40a00640 1 -- 172.21.15.28:0/555534282 wait complete. 2024-04-26T01:55:17.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:17 smithi053 ceph-mon[32593]: pgmap v450: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-26T01:55:17.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:17 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:17.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:17 smithi028 ceph-mon[28247]: pgmap v450: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-26T01:55:17.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:17 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:17.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:17 smithi169 ceph-mon[32851]: pgmap v450: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-26T01:55:17.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:17 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:19.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:19 smithi053 ceph-mon[32593]: pgmap v451: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-26T01:55:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:19 smithi028 ceph-mon[28247]: pgmap v451: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-26T01:55:19.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:19 smithi169 ceph-mon[32851]: pgmap v451: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-26T01:55:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 805.0 s, 1892.2 tps, lat 52.343 ms stddev 62.493 2024-04-26T01:55:21.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:21 smithi053 ceph-mon[32593]: pgmap v452: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-04-26T01:55:21.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:21 smithi028 ceph-mon[28247]: pgmap v452: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-04-26T01:55:21.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:21 smithi169 ceph-mon[32851]: pgmap v452: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-04-26T01:55:23.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:23 smithi053 ceph-mon[32593]: pgmap v453: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.2 MiB/s wr, 576 op/s 2024-04-26T01:55:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:23 smithi028 ceph-mon[28247]: pgmap v453: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.2 MiB/s wr, 576 op/s 2024-04-26T01:55:23.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:23 smithi169 ceph-mon[32851]: pgmap v453: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.2 MiB/s wr, 576 op/s 2024-04-26T01:55:25.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:25 smithi053 ceph-mon[32593]: pgmap v454: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.2 MiB/s wr, 670 op/s 2024-04-26T01:55:25.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:25 smithi028 ceph-mon[28247]: pgmap v454: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.2 MiB/s wr, 670 op/s 2024-04-26T01:55:25.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:25 smithi169 ceph-mon[32851]: pgmap v454: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.2 MiB/s wr, 670 op/s 2024-04-26T01:55:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 810.0 s, 1191.6 tps, lat 84.866 ms stddev 94.602 2024-04-26T01:55:27.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:27 smithi053 ceph-mon[32593]: pgmap v455: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.9 MiB/s wr, 744 op/s 2024-04-26T01:55:27.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:27 smithi028 ceph-mon[28247]: pgmap v455: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.9 MiB/s wr, 744 op/s 2024-04-26T01:55:27.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:27 smithi169 ceph-mon[32851]: pgmap v455: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.9 MiB/s wr, 744 op/s 2024-04-26T01:55:29.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:29 smithi028 ceph-mon[28247]: pgmap v456: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T01:55:29.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:29 smithi169 ceph-mon[32851]: pgmap v456: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T01:55:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.927+0000 7fc7be200640 1 -- 172.21.15.28:0/3500212819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 msgr2=0x7fc7b810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.927+0000 7fc7be200640 1 --2- 172.21.15.28:0/3500212819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 0x7fc7b810ee40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009900 tx=0x7fc7b002a420 comp rx=0 tx=0).stop 2024-04-26T01:55:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.928+0000 7fc7be200640 1 -- 172.21.15.28:0/3500212819 shutdown_connections 2024-04-26T01:55:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.928+0000 7fc7be200640 1 --2- 172.21.15.28:0/3500212819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 0x7fc7b810ee40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.928+0000 7fc7be200640 1 --2- 172.21.15.28:0/3500212819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b8102b50 0x7fc7b8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.928+0000 7fc7be200640 1 --2- 172.21.15.28:0/3500212819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b81021a0 0x7fc7b8102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.928+0000 7fc7be200640 1 -- 172.21.15.28:0/3500212819 >> 172.21.15.28:0/3500212819 conn(0x7fc7b80fda00 msgr2=0x7fc7b80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.928+0000 7fc7be200640 1 -- 172.21.15.28:0/3500212819 shutdown_connections 2024-04-26T01:55:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.928+0000 7fc7be200640 1 -- 172.21.15.28:0/3500212819 wait complete. 2024-04-26T01:55:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.929+0000 7fc7be200640 1 Processor -- start 2024-04-26T01:55:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.930+0000 7fc7be200640 1 -- start start 2024-04-26T01:55:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.930+0000 7fc7be200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b81021a0 0x7fc7b8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.930+0000 7fc7be200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 0x7fc7b8198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.930+0000 7fc7be200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b819cf30 0x7fc7b819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.930+0000 7fc7be200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b8112210 con 0x7fc7b81021a0 2024-04-26T01:55:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.930+0000 7fc7be200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b8112090 con 0x7fc7b8107270 2024-04-26T01:55:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.930+0000 7fc7be200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b8112390 con 0x7fc7b819cf30 2024-04-26T01:55:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.930+0000 7fc7b7e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b819cf30 0x7fc7b819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.931+0000 7fc7b7e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b819cf30 0x7fc7b819d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:44020/0 (socket says 172.21.15.28:44020) 2024-04-26T01:55:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.931+0000 7fc7b7e00640 1 -- 172.21.15.28:0/3307172013 learned_addr learned my addr 172.21.15.28:0/3307172013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.931+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 0x7fc7b8198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.931+0000 7fc7b7400640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b81021a0 0x7fc7b8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.932+0000 7fc7b6a00640 1 -- 172.21.15.28:0/3307172013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b819cf30 msgr2=0x7fc7b819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:29.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.932+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b819cf30 0x7fc7b819d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.932+0000 7fc7b6a00640 1 -- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b81021a0 msgr2=0x7fc7b8198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:29.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.932+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b81021a0 0x7fc7b8198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.932+0000 7fc7b6a00640 1 -- 172.21.15.28:0/3307172013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b819da20 con 0x7fc7b8107270 2024-04-26T01:55:29.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.932+0000 7fc7b7400640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b81021a0 0x7fc7b8198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.932+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 0x7fc7b8198bb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8005fd0 tx=0x7fc7a800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.932+0000 7fc7afe00640 1 -- 172.21.15.28:0/3307172013 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a800cd80 con 0x7fc7b8107270 2024-04-26T01:55:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.933+0000 7fc7afe00640 1 -- 172.21.15.28:0/3307172013 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7a8011920 con 0x7fc7b8107270 2024-04-26T01:55:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.933+0000 7fc7afe00640 1 -- 172.21.15.28:0/3307172013 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a801a9f0 con 0x7fc7b8107270 2024-04-26T01:55:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.933+0000 7fc7be200640 1 -- 172.21.15.28:0/3307172013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b819dd10 con 0x7fc7b8107270 2024-04-26T01:55:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.933+0000 7fc7be200640 1 -- 172.21.15.28:0/3307172013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b81a54b0 con 0x7fc7b8107270 2024-04-26T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.937+0000 7fc7c0fc1740 1 Processor -- start 2024-04-26T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.937+0000 7fc7c0fc1740 1 -- start start 2024-04-26T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.937+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0841b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.937+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841bc20 0x55fd0841c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.937+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0842f0a0 0x55fd08431490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.938+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fd08381900 con 0x55fd0841cd60 2024-04-26T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.938+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fd08327ef0 con 0x55fd0841bc20 2024-04-26T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.938+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55fd0837d8c0 con 0x55fd0842f0a0 2024-04-26T01:55:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.938+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841bc20 0x55fd0841c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.938+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841bc20 0x55fd0841c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36808/0 (socket says 172.21.15.28:36808) 2024-04-26T01:55:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.938+0000 7fc7b7400640 1 -- 172.21.15.28:0/715814604 learned_addr learned my addr 172.21.15.28:0/715814604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.938+0000 7fc7b7e00640 1 --2- 172.21.15.28:0/715814604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0841b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.939+0000 7fc7b7400640 1 -- 172.21.15.28:0/715814604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0842f0a0 msgr2=0x55fd08431490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:55:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.939+0000 7fc7b7400640 1 --2- 172.21.15.28:0/715814604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0842f0a0 0x55fd08431490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.939+0000 7fc7b7400640 1 -- 172.21.15.28:0/715814604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 msgr2=0x55fd0841b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.939+0000 7fc7b7400640 1 --2- 172.21.15.28:0/715814604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0841b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.939+0000 7fc7b7400640 1 -- 172.21.15.28:0/715814604 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd08094210 con 0x55fd0841bc20 2024-04-26T01:55:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.939+0000 7fc7b7400640 1 --2- 172.21.15.28:0/715814604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841bc20 0x55fd0841c0a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009c50 tx=0x7fc7a000e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5544859fb630de50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.940+0000 7fc7a7e00640 1 -- 172.21.15.28:0/715814604 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0016070 con 0x55fd0841bc20 2024-04-26T01:55:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.940+0000 7fc7a7e00640 1 -- 172.21.15.28:0/715814604 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7a00054d0 con 0x55fd0841bc20 2024-04-26T01:55:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.940+0000 7fc7a7e00640 1 -- 172.21.15.28:0/715814604 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0011470 con 0x55fd0841bc20 2024-04-26T01:55:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.940+0000 7fc7afe00640 1 -- 172.21.15.28:0/3307172013 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc7a8011ad0 con 0x7fc7b8107270 2024-04-26T01:55:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.941+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/715814604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841bc20 msgr2=0x55fd0841c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.941+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/715814604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841bc20 0x55fd0841c0a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009c50 tx=0x7fc7a000e6c0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.942+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/715814604 shutdown_connections 2024-04-26T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.942+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/715814604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0842f0a0 0x55fd08431490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.942+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/715814604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841bc20 0x55fd0841c0a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.942+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/715814604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0841b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.942+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/715814604 >> 172.21.15.28:0/715814604 conn(0x55fd08417d30 msgr2=0x55fd08418110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.942+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/715814604 shutdown_connections 2024-04-26T01:55:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.942+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/715814604 wait complete. 2024-04-26T01:55:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.943+0000 7fc7c0fc1740 1 -- start start 2024-04-26T01:55:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.943+0000 7fc7c0fc1740 1 Processor -- start 2024-04-26T01:55:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.944+0000 7fc7afe00640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc79c0a3d50 0x7fc79c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.944+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd08428ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.944+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841cd60 0x55fd084293e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.944+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0842f0a0 0x55fd08424200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.944+0000 7fc7b7e00640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc79c0a3d50 0x7fc79c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.944+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd08428ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.944+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd08428ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:44022/0 (socket says 172.21.15.28:44022) 2024-04-26T01:55:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7b7400640 1 -- 172.21.15.28:0/1951752561 learned_addr learned my addr 172.21.15.28:0/1951752561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841cd60 0x55fd084293e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fd08422c70 con 0x55fd0842f0a0 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fd0833b310 con 0x55fd0841cd60 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55fd08422df0 con 0x55fd0841bc20 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7c0fc1740 10 client.0 _ll_register_callbacks cb 0x55fd08309cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7b7e00640 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0842f0a0 0x55fd08424200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7afe00640 1 -- 172.21.15.28:0/3307172013 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7fc7a80d0b90 con 0x7fc7b8107270 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7b7400640 1 -- 172.21.15.28:0/1951752561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841cd60 msgr2=0x55fd084293e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7b7400640 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841cd60 0x55fd084293e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.945+0000 7fc7b7400640 1 -- 172.21.15.28:0/1951752561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0842f0a0 msgr2=0x55fd08424200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.946+0000 7fc7b7400640 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0842f0a0 0x55fd08424200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.946+0000 7fc7b7400640 1 -- 172.21.15.28:0/1951752561 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd08160620 con 0x55fd0841bc20 2024-04-26T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.946+0000 7fc7b7400640 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd08428ea0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000ebc0 tx=0x7fc7a0005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63cbe8bcb10b6a99 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.946+0000 7fc7a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.947+0000 7fc7b7e00640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc79c0a3d50 0x7fc79c0a6210 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0005c40 tx=0x7fc7b0004590 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.947+0000 7fc7a4c00640 1 -- 172.21.15.28:0/1951752561 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0016040 con 0x55fd0841bc20 2024-04-26T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.947+0000 7fc7a4c00640 1 -- 172.21.15.28:0/1951752561 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7fc7a0005680 con 0x55fd0841bc20 2024-04-26T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.947+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7a00054d0 con 0x55fd0841bc20 2024-04-26T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.947+0000 7fc7a4c00640 1 -- 172.21.15.28:0/1951752561 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc7a0009760 con 0x55fd0841bc20 2024-04-26T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.947+0000 7fc7c0fc1740 10 client.26413 fetch_fsmap learned FSMap version 47 2024-04-26T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.947+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55fd08446a60 con 0x55fd0841bc20 2024-04-26T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.948+0000 7fc7a4c00640 1 -- 172.21.15.28:0/1951752561 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7a00054d0 con 0x55fd0841bc20 2024-04-26T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.949+0000 7fc7a4c00640 1 -- 172.21.15.28:0/1951752561 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fc7a000f040 con 0x55fd0841bc20 2024-04-26T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.949+0000 7fc7c0fc1740 10 client.26413 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.949+0000 7fc7c0fc1740 10 client.26413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.949+0000 7fc7c0fc1740 20 client.26413 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.949+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fd08447e70 0x55fd0844a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.949+0000 7fc7c0fc1740 4 client.26413 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.949+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd083958b0 con 0x55fd08447e70 2024-04-26T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.949+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fd08447e70 0x55fd0844a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.950+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fd08447e70 0x55fd0844a260 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca5de6e68c8acbb server_cookie=9663bb2f019f8e64 in_seq=0 out_seq=0 2024-04-26T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.950+0000 7fc7a4c00640 10 client.26413 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.951+0000 7fc7a4c00640 1 -- 172.21.15.28:0/1951752561 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd083958b0 con 0x55fd08447e70 2024-04-26T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.951+0000 7fc7a4c00640 10 client.26413 handle_command_reply: tid=0 2024-04-26T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.951+0000 7fc7c0fc1740 1 client.26413 shutdown 2024-04-26T01:55:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.951+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 msgr2=0x55fd08428ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.951+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd08428ea0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000ebc0 tx=0x7fc7a0005eb0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 shutdown_connections 2024-04-26T01:55:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fd08447e70 0x55fd0844a260 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0842f0a0 0x55fd08424200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0841cd60 0x55fd084293e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/1951752561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd08428ea0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 >> 172.21.15.28:0/1951752561 conn(0x55fd08417d30 msgr2=0x55fd0841db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7a4c00640 0 client.26413 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 shutdown_connections 2024-04-26T01:55:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/1951752561 wait complete. 2024-04-26T01:55:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.952+0000 7fc7c0fc1740 20 client.26413 trim_cache size 0 max 16384 2024-04-26T01:55:29.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:29 smithi053 ceph-mon[32593]: pgmap v456: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 577 op/s 2024-04-26T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.978+0000 7fc7c0fc1740 1 Processor -- start 2024-04-26T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.978+0000 7fc7c0fc1740 1 -- start start 2024-04-26T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0844f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0844fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08423a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b81ae260 con 0x55fd0841cd60 2024-04-26T01:55:29.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b81abb30 con 0x55fd0842f0a0 2024-04-26T01:55:29.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b81a99a0 con 0x55fd0841bc20 2024-04-26T01:55:29.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0844f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0844f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:44036/0 (socket says 172.21.15.28:44036) 2024-04-26T01:55:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0844fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0844fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43800/0 (socket says 172.21.15.28:43800) 2024-04-26T01:55:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7b6a00640 1 -- 172.21.15.28:0/2551587683 learned_addr learned my addr 172.21.15.28:0/2551587683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.979+0000 7fc7b7e00640 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08423a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.980+0000 7fc7b7400640 1 -- 172.21.15.28:0/2551587683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 msgr2=0x55fd08423a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.980+0000 7fc7b7400640 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08423a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.980+0000 7fc7b7400640 1 -- 172.21.15.28:0/2551587683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 msgr2=0x55fd0844fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.980+0000 7fc7b7400640 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0844fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.980+0000 7fc7b7400640 1 -- 172.21.15.28:0/2551587683 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd08422e00 con 0x55fd0841bc20 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.980+0000 7fc7b7e00640 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08423a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.980+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0844fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.980+0000 7fc7b7400640 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0844f890 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0011e60 tx=0x7fc7a0004c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9d46e3d8bd4b5cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.981+0000 7fc7a7400640 1 -- 172.21.15.28:0/2551587683 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0016070 con 0x55fd0841bc20 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.981+0000 7fc7a7400640 1 -- 172.21.15.28:0/2551587683 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7a0030020 con 0x55fd0841bc20 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.981+0000 7fc7a7400640 1 -- 172.21.15.28:0/2551587683 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a002b400 con 0x55fd0841bc20 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.982+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/2551587683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 msgr2=0x55fd0844f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.982+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0844f890 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0011e60 tx=0x7fc7a0004c90 comp rx=0 tx=0).stop 2024-04-26T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.982+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/2551587683 shutdown_connections 2024-04-26T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.982+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08423a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.982+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd0844fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.982+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/2551587683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0844f890 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.982+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/2551587683 >> 172.21.15.28:0/2551587683 conn(0x55fd08417d30 msgr2=0x55fd0841e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.982+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/2551587683 shutdown_connections 2024-04-26T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.983+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/2551587683 wait complete. 2024-04-26T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.983+0000 7fc7c0fc1740 1 -- start start 2024-04-26T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.984+0000 7fc7c0fc1740 1 Processor -- start 2024-04-26T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0845b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd08428040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7c0fc1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08428580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b81a9ff0 con 0x55fd0841cd60 2024-04-26T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b81ab460 con 0x55fd0842f0a0 2024-04-26T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7c0fc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b81aa6c0 con 0x55fd0841bc20 2024-04-26T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7c0fc1740 10 client.0 _ll_register_callbacks cb 0x7fc7a0005150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0845b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd08428040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7b7400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0845b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:44050/0 (socket says 172.21.15.28:44050) 2024-04-26T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.985+0000 7fc7b7400640 1 -- 172.21.15.28:0/499092705 learned_addr learned my addr 172.21.15.28:0/499092705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.986+0000 7fc7b7e00640 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08428580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.986+0000 7fc7b6a00640 1 -- 172.21.15.28:0/499092705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 msgr2=0x55fd0845b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.986+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0845b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.986+0000 7fc7b6a00640 1 -- 172.21.15.28:0/499092705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 msgr2=0x55fd08428580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.986+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08428580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.987+0000 7fc7b6a00640 1 -- 172.21.15.28:0/499092705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd08428bc0 con 0x55fd0841cd60 2024-04-26T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.987+0000 7fc7b7e00640 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08428580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.987+0000 7fc7b7400640 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0845b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.987+0000 7fc7b6a00640 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd08428040 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80ce390 tx=0x7fc7a8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9efbf5fbe3b77d77 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.987+0000 7fc7a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.988+0000 7fc7a4c00640 1 -- 172.21.15.28:0/499092705 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a809de60 con 0x55fd0841cd60 2024-04-26T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.988+0000 7fc7a4c00640 1 -- 172.21.15.28:0/499092705 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7fc7a80bd0b0 con 0x55fd0841cd60 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.988+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/499092705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7a0030020 con 0x55fd0841cd60 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.988+0000 7fc7a4c00640 1 -- 172.21.15.28:0/499092705 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc7a809cb30 con 0x55fd0841cd60 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7c0fc1740 10 client.16131 fetch_fsmap learned FSMap version 47 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/499092705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55fd084ada80 con 0x55fd0841cd60 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7a4c00640 1 -- 172.21.15.28:0/499092705 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7a0030020 con 0x55fd0841cd60 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7a4c00640 1 -- 172.21.15.28:0/499092705 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fc7a8022030 con 0x55fd0841cd60 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7c0fc1740 10 client.16131 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7c0fc1740 10 client.16131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7c0fc1740 20 client.16131 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fd084ae1c0 0x55fd084b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7c0fc1740 4 client.16131 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.989+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/499092705 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd0838f320 con 0x55fd084ae1c0 2024-04-26T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.990+0000 7fc7b7400640 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fd084ae1c0 0x55fd084b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.991+0000 7fc7b7400640 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fd084ae1c0 0x55fd084b05e0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b9b0c0051f6179d server_cookie=af0bb5da97742167 in_seq=0 out_seq=0 2024-04-26T01:55:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:29.991+0000 7fc7a4c00640 10 client.16131 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.005+0000 7fc7a4c00640 1 -- 172.21.15.28:0/499092705 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fd0838f320 con 0x55fd084ae1c0 2024-04-26T01:55:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.005+0000 7fc7a4c00640 10 client.16131 handle_command_reply: tid=0 2024-04-26T01:55:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.005+0000 7fc7c0fc1740 1 client.16131 shutdown 2024-04-26T01:55:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.005+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/499092705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 msgr2=0x55fd08428040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.005+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd08428040 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80ce390 tx=0x7fc7a8000f40 comp rx=0 tx=0).stop 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/499092705 shutdown_connections 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55fd084ae1c0 0x55fd084b05e0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fd0842f0a0 0x55fd08428580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd0841cd60 0x55fd08428040 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7c0fc1740 1 --2- 172.21.15.28:0/499092705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fd0841bc20 0x55fd0845b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/499092705 >> 172.21.15.28:0/499092705 conn(0x55fd08417d30 msgr2=0x55fd08423a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7a4c00640 0 client.16131 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/499092705 shutdown_connections 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.006+0000 7fc7c0fc1740 1 -- 172.21.15.28:0/499092705 wait complete. 2024-04-26T01:55:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.007+0000 7fc7c0fc1740 20 client.16131 trim_cache size 0 max 16384 2024-04-26T01:55:30.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:55:30.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:55:30.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:55:30.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 -- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc79c0a3d50 msgr2=0x7fc79c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc79c0a3d50 0x7fc79c0a6210 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0005c40 tx=0x7fc7b0004590 comp rx=0 tx=0).stop 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 -- 172.21.15.28:0/3307172013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 msgr2=0x7fc7b8198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 0x7fc7b8198bb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8005fd0 tx=0x7fc7a800ee80 comp rx=0 tx=0).stop 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 -- 172.21.15.28:0/3307172013 shutdown_connections 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fc79c0a3d50 0x7fc79c0a6210 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b819cf30 0x7fc7b819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b8107270 0x7fc7b8198bb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 --2- 172.21.15.28:0/3307172013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b81021a0 0x7fc7b8198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.008+0000 7fc7be200640 1 -- 172.21.15.28:0/3307172013 >> 172.21.15.28:0/3307172013 conn(0x7fc7b80fda00 msgr2=0x7fc7b80fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.009+0000 7fc7be200640 1 -- 172.21.15.28:0/3307172013 shutdown_connections 2024-04-26T01:55:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:30.009+0000 7fc7be200640 1 -- 172.21.15.28:0/3307172013 wait complete. 2024-04-26T01:55:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 815.0 s, 1857.4 tps, lat 53.711 ms stddev 64.561 2024-04-26T01:55:31.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:31 smithi028 ceph-mon[28247]: pgmap v457: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.0 MiB/s wr, 858 op/s 2024-04-26T01:55:31.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:31 smithi169 ceph-mon[32851]: pgmap v457: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.0 MiB/s wr, 858 op/s 2024-04-26T01:55:31.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:31 smithi053 ceph-mon[32593]: pgmap v457: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.0 MiB/s wr, 858 op/s 2024-04-26T01:55:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:33 smithi028 ceph-mon[28247]: pgmap v458: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-04-26T01:55:33.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:33 smithi169 ceph-mon[32851]: pgmap v458: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-04-26T01:55:33.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:33 smithi053 ceph-mon[32593]: pgmap v458: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-04-26T01:55:35.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:35 smithi028 ceph-mon[28247]: pgmap v459: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.1 MiB/s wr, 649 op/s 2024-04-26T01:55:35.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:35 smithi169 ceph-mon[32851]: pgmap v459: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.1 MiB/s wr, 649 op/s 2024-04-26T01:55:35.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:35 smithi053 ceph-mon[32593]: pgmap v459: 97 pgs: 97 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.1 MiB/s wr, 649 op/s 2024-04-26T01:55:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 820.0 s, 1725.6 tps, lat 58.341 ms stddev 66.871 2024-04-26T01:55:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:37 smithi028 ceph-mon[28247]: pgmap v460: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.9 MiB/s wr, 752 op/s 2024-04-26T01:55:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:37 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:37.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:37 smithi169 ceph-mon[32851]: pgmap v460: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.9 MiB/s wr, 752 op/s 2024-04-26T01:55:37.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:37 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:37.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:37 smithi053 ceph-mon[32593]: pgmap v460: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.9 MiB/s wr, 752 op/s 2024-04-26T01:55:37.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:37 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:39.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:39 smithi028 ceph-mon[28247]: pgmap v461: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.4 MiB/s wr, 584 op/s 2024-04-26T01:55:39.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:39 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:39.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:39 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:39.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:39 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:55:39.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:39 smithi169 ceph-mon[32851]: pgmap v461: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.4 MiB/s wr, 584 op/s 2024-04-26T01:55:39.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:39 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:39.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:39 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:39.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:39 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:55:39.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:39 smithi053 ceph-mon[32593]: pgmap v461: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.4 MiB/s wr, 584 op/s 2024-04-26T01:55:39.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:39 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:39.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:39 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:39.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:39 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:55:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 825.0 s, 1650.8 tps, lat 59.680 ms stddev 69.580 2024-04-26T01:55:41.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:41 smithi169 ceph-mon[32851]: pgmap v462: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.8 MiB/s wr, 845 op/s 2024-04-26T01:55:41.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:41 smithi053 ceph-mon[32593]: pgmap v462: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.8 MiB/s wr, 845 op/s 2024-04-26T01:55:42.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:41 smithi028 ceph-mon[28247]: pgmap v462: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.8 MiB/s wr, 845 op/s 2024-04-26T01:55:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 -- 172.21.15.28:0/2045821778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a101043c0 msgr2=0x7f3a101047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 --2- 172.21.15.28:0/2045821778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a101043c0 0x7f3a101047a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009900 tx=0x7f39fc02a410 comp rx=0 tx=0).stop 2024-04-26T01:55:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 -- 172.21.15.28:0/2045821778 shutdown_connections 2024-04-26T01:55:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 --2- 172.21.15.28:0/2045821778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a10109490 0x7f3a10111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 --2- 172.21.15.28:0/2045821778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10104d70 0x7f3a10108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 --2- 172.21.15.28:0/2045821778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a101043c0 0x7f3a101047a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 -- 172.21.15.28:0/2045821778 >> 172.21.15.28:0/2045821778 conn(0x7f3a100ffc80 msgr2=0x7f3a101020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 -- 172.21.15.28:0/2045821778 shutdown_connections 2024-04-26T01:55:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.230+0000 7f3a15800640 1 -- 172.21.15.28:0/2045821778 wait complete. 2024-04-26T01:55:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.231+0000 7f3a15800640 1 Processor -- start 2024-04-26T01:55:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.232+0000 7f3a15800640 1 -- start start 2024-04-26T01:55:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.232+0000 7f3a15800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a101043c0 0x7f3a10073aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.232+0000 7f3a15800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10109490 0x7f3a1006ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.232+0000 7f3a15800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a1006bfa0 0x7f3a1006c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.232+0000 7f3a15800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a10114450 con 0x7f3a10109490 2024-04-26T01:55:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.232+0000 7f3a15800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101142d0 con 0x7f3a101043c0 2024-04-26T01:55:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.232+0000 7f3a15800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101145d0 con 0x7f3a1006bfa0 2024-04-26T01:55:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a1006bfa0 0x7f3a1006c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a1006bfa0 0x7f3a1006c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:36486/0 (socket says 172.21.15.28:36486) 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0f400640 1 -- 172.21.15.28:0/446880824 learned_addr learned my addr 172.21.15.28:0/446880824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0e000640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10109490 0x7f3a1006ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0f400640 1 -- 172.21.15.28:0/446880824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a101043c0 msgr2=0x7f3a10073aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0f400640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a101043c0 0x7f3a10073aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0f400640 1 -- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10109490 msgr2=0x7f3a1006ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0f400640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10109490 0x7f3a1006ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.233+0000 7f3a0f400640 1 -- 172.21.15.28:0/446880824 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a101af880 con 0x7f3a1006bfa0 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.234+0000 7f3a0f400640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a1006bfa0 0x7f3a1006c450 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000e9b0 tx=0x7f3a0000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.235+0000 7f3a07400640 1 -- 172.21.15.28:0/446880824 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0000cd20 con 0x7f3a1006bfa0 2024-04-26T01:55:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.235+0000 7f3a07400640 1 -- 172.21.15.28:0/446880824 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a000035f0 con 0x7f3a1006bfa0 2024-04-26T01:55:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.235+0000 7f3a07400640 1 -- 172.21.15.28:0/446880824 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a000216a0 con 0x7f3a1006bfa0 2024-04-26T01:55:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.235+0000 7f3a15800640 1 -- 172.21.15.28:0/446880824 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a100763f0 con 0x7f3a1006bfa0 2024-04-26T01:55:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.235+0000 7f3a15800640 1 -- 172.21.15.28:0/446880824 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a100768f0 con 0x7f3a1006bfa0 2024-04-26T01:55:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.241+0000 7f3a18785740 1 Processor -- start 2024-04-26T01:55:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.246+0000 7f3a07400640 1 -- 172.21.15.28:0/446880824 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3a00004020 con 0x7f3a1006bfa0 2024-04-26T01:55:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.246+0000 7f3a18785740 1 -- start start 2024-04-26T01:55:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a18785740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e0820 0x55f56f1e0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a18785740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a18785740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1eba00 0x55f56f1eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a18785740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f56f146900 con 0x55f56f1eba00 2024-04-26T01:55:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a18785740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f56f0ecef0 con 0x55f56f1e0820 2024-04-26T01:55:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a18785740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f56f1428c0 con 0x55f56f1e1140 2024-04-26T01:55:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a0ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a0ea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1eb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:36488/0 (socket says 172.21.15.28:36488) 2024-04-26T01:55:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.247+0000 7f3a0ea00640 1 -- 172.21.15.28:0/4243555213 learned_addr learned my addr 172.21.15.28:0/4243555213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.248+0000 7f3a0e000640 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1eba00 0x55f56f1eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.248+0000 7f3a0e000640 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e0820 0x55f56f1e0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.249+0000 7f3a0ea00640 1 -- 172.21.15.28:0/4243555213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e0820 msgr2=0x55f56f1e0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.249+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e0820 0x55f56f1e0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.249+0000 7f3a0ea00640 1 -- 172.21.15.28:0/4243555213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1eba00 msgr2=0x55f56f1eddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.249+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1eba00 0x55f56f1eddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.249+0000 7f3a0ea00640 1 -- 172.21.15.28:0/4243555213 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56ee59210 con 0x55f56f1e1140 2024-04-26T01:55:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.250+0000 7f3a0e000640 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e0820 0x55f56f1e0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:43.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.253+0000 7f3a07400640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f39f40a3e40 0x7f39f40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.255+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1eb4c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f39fc02a930 tx=0x7f39fc009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e39bd40c88b65a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.255+0000 7f3a07400640 1 -- 172.21.15.28:0/446880824 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7f3a000e1bd0 con 0x7f3a1006bfa0 2024-04-26T01:55:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.256+0000 7f39fbe00640 1 -- 172.21.15.28:0/4243555213 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc040070 con 0x55f56f1e1140 2024-04-26T01:55:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.256+0000 7f3a0e000640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f39f40a3e40 0x7f39f40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.256+0000 7f39fbe00640 1 -- 172.21.15.28:0/4243555213 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39fc032d90 con 0x55f56f1e1140 2024-04-26T01:55:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.256+0000 7f39fbe00640 1 -- 172.21.15.28:0/4243555213 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc040070 con 0x55f56f1e1140 2024-04-26T01:55:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.257+0000 7f3a18785740 1 -- 172.21.15.28:0/4243555213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 msgr2=0x55f56f1eb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.257+0000 7f3a18785740 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1eb4c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f39fc02a930 tx=0x7f39fc009680 comp rx=0 tx=0).stop 2024-04-26T01:55:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.260+0000 7f3a18785740 1 -- 172.21.15.28:0/4243555213 shutdown_connections 2024-04-26T01:55:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.260+0000 7f3a18785740 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1eba00 0x55f56f1eddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.260+0000 7f3a18785740 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1eb4c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.260+0000 7f3a18785740 1 --2- 172.21.15.28:0/4243555213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e0820 0x55f56f1e0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.260+0000 7f3a18785740 1 -- 172.21.15.28:0/4243555213 >> 172.21.15.28:0/4243555213 conn(0x55f56f1dcd30 msgr2=0x55f56f1dd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.260+0000 7f3a0e000640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f39f40a3e40 0x7f39f40a6300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800a010 tx=0x7f3a0800e040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.260+0000 7f3a18785740 1 -- 172.21.15.28:0/4243555213 shutdown_connections 2024-04-26T01:55:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.261+0000 7f3a18785740 1 -- 172.21.15.28:0/4243555213 wait complete. 2024-04-26T01:55:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.261+0000 7f3a18785740 1 -- start start 2024-04-26T01:55:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.262+0000 7f3a18785740 1 Processor -- start 2024-04-26T01:55:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.264+0000 7f3a18785740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1e89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.264+0000 7f3a18785740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.264+0000 7f3a18785740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1e9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.264+0000 7f3a0e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.265+0000 7f3a0f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.265+0000 7f3a0ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1e89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.265+0000 7f3a0f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1e9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:36492/0 (socket says 172.21.15.28:36492) 2024-04-26T01:55:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.265+0000 7f3a0ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1e89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44508/0 (socket says 172.21.15.28:44508) 2024-04-26T01:55:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.265+0000 7f3a0f400640 1 -- 172.21.15.28:0/4088838914 learned_addr learned my addr 172.21.15.28:0/4088838914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.265+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f56f1ee5d0 con 0x55f56f1e0820 2024-04-26T01:55:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.265+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f56f100310 con 0x55f56f1e1140 2024-04-26T01:55:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.265+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f56f1ee750 con 0x55f56f1eba00 2024-04-26T01:55:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.266+0000 7f3a18785740 10 client.0 _ll_register_callbacks cb 0x55f56f0cecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f3a0f400640 1 -- 172.21.15.28:0/4088838914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 msgr2=0x55f56f1e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f3a0f400640 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1e8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f3a0f400640 1 -- 172.21.15.28:0/4088838914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 msgr2=0x55f56f1e89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f3a0f400640 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1e89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f3a0f400640 1 -- 172.21.15.28:0/4088838914 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f56ef25620 con 0x55f56f1eba00 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f3a0f400640 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1e9440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3a000adb00 tx=0x7f3a00005790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3236103aadef5736 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f39f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f39f8c00640 1 -- 172.21.15.28:0/4088838914 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a000b1f20 con 0x55f56f1eba00 2024-04-26T01:55:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f39f8c00640 1 -- 172.21.15.28:0/4088838914 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7f3a000ce0f0 con 0x55f56f1eba00 2024-04-26T01:55:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39fc032d90 con 0x55f56f1eba00 2024-04-26T01:55:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.269+0000 7f39f8c00640 1 -- 172.21.15.28:0/4088838914 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3a000b67e0 con 0x55f56f1eba00 2024-04-26T01:55:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.270+0000 7f3a18785740 10 client.26431 fetch_fsmap learned FSMap version 47 2024-04-26T01:55:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.270+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f56f1fa8b0 con 0x55f56f1eba00 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.270+0000 7f39f8c00640 1 -- 172.21.15.28:0/4088838914 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39fc032d90 con 0x55f56f1eba00 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.272+0000 7f39f8c00640 1 -- 172.21.15.28:0/4088838914 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f3a000c8030 con 0x55f56f1eba00 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.272+0000 7f3a18785740 10 client.26431 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.272+0000 7f3a18785740 10 client.26431 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.272+0000 7f3a18785740 20 client.26431 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.272+0000 7f3a18785740 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f56f1fbd70 0x55f56f1fe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.272+0000 7f3a18785740 4 client.26431 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.272+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f56f15a8b0 con 0x55f56f1fbd70 2024-04-26T01:55:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.273+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f56f1fbd70 0x55f56f1fe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.274+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f56f1fbd70 0x55f56f1fe1c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9da14f361c2c367d server_cookie=49cd1e89cbec828b in_seq=0 out_seq=0 2024-04-26T01:55:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.274+0000 7f39f8c00640 10 client.26431 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.275+0000 7f39f8c00640 1 -- 172.21.15.28:0/4088838914 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f56f15a8b0 con 0x55f56f1fbd70 2024-04-26T01:55:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.275+0000 7f39f8c00640 10 client.26431 handle_command_reply: tid=0 2024-04-26T01:55:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.275+0000 7f3a18785740 1 client.26431 shutdown 2024-04-26T01:55:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.276+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 msgr2=0x55f56f1e9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.276+0000 7f3a18785740 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1e9440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3a000adb00 tx=0x7f3a00005790 comp rx=0 tx=0).stop 2024-04-26T01:55:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.276+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 shutdown_connections 2024-04-26T01:55:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.276+0000 7f3a18785740 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f56f1fbd70 0x55f56f1fe1c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.276+0000 7f3a18785740 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1e9440 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.276+0000 7f3a18785740 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.276+0000 7f3a18785740 1 --2- 172.21.15.28:0/4088838914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1e89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.276+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 >> 172.21.15.28:0/4088838914 conn(0x55f56f1dcd30 msgr2=0x55f56f1e2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.277+0000 7f39f8c00640 0 client.26431 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.278+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 shutdown_connections 2024-04-26T01:55:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.278+0000 7f3a18785740 1 -- 172.21.15.28:0/4088838914 wait complete. 2024-04-26T01:55:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.278+0000 7f3a18785740 20 client.26431 trim_cache size 0 max 16384 2024-04-26T01:55:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.315+0000 7f3a18785740 1 Processor -- start 2024-04-26T01:55:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.315+0000 7f3a18785740 1 -- start start 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.315+0000 7f3a18785740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.315+0000 7f3a18785740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1e9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.315+0000 7f3a18785740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1eba00 0x55f56f203ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.315+0000 7f3a18785740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101b6970 con 0x55f56f1e0820 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.315+0000 7f3a18785740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101b4240 con 0x55f56f1eba00 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.315+0000 7f3a18785740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101b20b0 con 0x55f56f1e1140 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0e000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1e9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1eba00 0x55f56f203ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1eba00 0x55f56f203ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:42132/0 (socket says 172.21.15.28:42132) 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0f400640 1 -- 172.21.15.28:0/2038597407 learned_addr learned my addr 172.21.15.28:0/2038597407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0f400640 1 -- 172.21.15.28:0/2038597407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 msgr2=0x55f56f1e9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0f400640 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1e9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0f400640 1 -- 172.21.15.28:0/2038597407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 msgr2=0x55f56f1f7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0f400640 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0f400640 1 -- 172.21.15.28:0/2038597407 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56f1e0f70 con 0x55f56f1eba00 2024-04-26T01:55:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.316+0000 7f3a0e000640 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1e9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.317+0000 7f3a0f400640 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1eba00 0x55f56f203ad0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3a000aa540 tx=0x7f3a000ac080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f272dd5cdc7e5eb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.317+0000 7f39fb400640 1 -- 172.21.15.28:0/2038597407 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a000a9e10 con 0x55f56f1eba00 2024-04-26T01:55:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.317+0000 7f39fb400640 1 -- 172.21.15.28:0/2038597407 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a000aa1e0 con 0x55f56f1eba00 2024-04-26T01:55:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.317+0000 7f39fb400640 1 -- 172.21.15.28:0/2038597407 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a000ce070 con 0x55f56f1eba00 2024-04-26T01:55:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.318+0000 7f3a18785740 1 -- 172.21.15.28:0/2038597407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1eba00 msgr2=0x55f56f203ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.318+0000 7f3a18785740 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1eba00 0x55f56f203ad0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3a000aa540 tx=0x7f3a000ac080 comp rx=0 tx=0).stop 2024-04-26T01:55:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.319+0000 7f3a18785740 1 -- 172.21.15.28:0/2038597407 shutdown_connections 2024-04-26T01:55:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.319+0000 7f3a18785740 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1eba00 0x55f56f203ad0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.319+0000 7f3a18785740 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1e1140 0x55f56f1e9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.319+0000 7f3a18785740 1 --2- 172.21.15.28:0/2038597407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.319+0000 7f3a18785740 1 -- 172.21.15.28:0/2038597407 >> 172.21.15.28:0/2038597407 conn(0x55f56f1dcd30 msgr2=0x55f56f1e3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.319+0000 7f3a18785740 1 -- 172.21.15.28:0/2038597407 shutdown_connections 2024-04-26T01:55:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.319+0000 7f3a18785740 1 -- 172.21.15.28:0/2038597407 wait complete. 2024-04-26T01:55:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.320+0000 7f3a18785740 1 -- start start 2024-04-26T01:55:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.320+0000 7f3a18785740 1 Processor -- start 2024-04-26T01:55:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.321+0000 7f3a18785740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.322+0000 7f3a18785740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.322+0000 7f3a0ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.323+0000 7f3a18785740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.323+0000 7f3a18785740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101b2700 con 0x55f56f1e0820 2024-04-26T01:55:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.323+0000 7f3a18785740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101b3b70 con 0x55f56f1e1140 2024-04-26T01:55:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.323+0000 7f3a18785740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3a101b2dd0 con 0x55f56f1eba00 2024-04-26T01:55:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.323+0000 7f3a18785740 10 client.0 _ll_register_callbacks cb 0x7f3a000ad740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.323+0000 7f3a0e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.323+0000 7f3a0e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1f0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:42146/0 (socket says 172.21.15.28:42146) 2024-04-26T01:55:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.323+0000 7f3a0e000640 1 -- 172.21.15.28:0/3277915053 learned_addr learned my addr 172.21.15.28:0/3277915053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.324+0000 7f3a0ea00640 1 -- 172.21.15.28:0/3277915053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 msgr2=0x55f56f1f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.324+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.324+0000 7f3a0ea00640 1 -- 172.21.15.28:0/3277915053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 msgr2=0x55f56f1f0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.324+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1f0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.324+0000 7f3a0ea00640 1 -- 172.21.15.28:0/3277915053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f56f1f19f0 con 0x55f56f1e0820 2024-04-26T01:55:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.325+0000 7f3a0ea00640 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f3a40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0092a0 tx=0x7f39fc005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195d03a0edbecb35 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.325+0000 7f39f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:55:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.326+0000 7f39f8c00640 1 -- 172.21.15.28:0/3277915053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc040040 con 0x55f56f1e0820 2024-04-26T01:55:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.326+0000 7f39f8c00640 1 -- 172.21.15.28:0/3277915053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7f39fc031430 con 0x55f56f1e0820 2024-04-26T01:55:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.326+0000 7f3a18785740 1 -- 172.21.15.28:0/3277915053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a000aa1e0 con 0x55f56f1e0820 2024-04-26T01:55:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.326+0000 7f39f8c00640 1 -- 172.21.15.28:0/3277915053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f39fc035040 con 0x55f56f1e0820 2024-04-26T01:55:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.327+0000 7f3a18785740 10 client.16146 fetch_fsmap learned FSMap version 47 2024-04-26T01:55:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.327+0000 7f3a18785740 1 -- 172.21.15.28:0/3277915053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f56f262070 con 0x55f56f1e0820 2024-04-26T01:55:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.327+0000 7f39f8c00640 1 -- 172.21.15.28:0/3277915053 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a000aa1e0 con 0x55f56f1e0820 2024-04-26T01:55:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.328+0000 7f39f8c00640 1 -- 172.21.15.28:0/3277915053 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f39fc04f540 con 0x55f56f1e0820 2024-04-26T01:55:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.328+0000 7f3a18785740 10 client.16146 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:55:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.328+0000 7f3a18785740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.328+0000 7f3a18785740 20 client.16146 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.328+0000 7f3a18785740 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f56f2624c0 0x55f56f2648b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.328+0000 7f3a18785740 4 client.16146 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.328+0000 7f3a18785740 1 -- 172.21.15.28:0/3277915053 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f56f154320 con 0x55f56f2624c0 2024-04-26T01:55:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.328+0000 7f3a0e000640 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f56f2624c0 0x55f56f2648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.329+0000 7f39f8c00640 10 client.16146 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.329+0000 7f3a0e000640 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f56f2624c0 0x55f56f2648b0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=462901624390dae4 server_cookie=69045de2174f7d4b in_seq=0 out_seq=0 2024-04-26T01:55:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.345+0000 7f39f8c00640 1 -- 172.21.15.28:0/3277915053 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f56f154320 con 0x55f56f2624c0 2024-04-26T01:55:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.345+0000 7f39f8c00640 10 client.16146 handle_command_reply: tid=0 2024-04-26T01:55:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.345+0000 7f3a18785740 1 client.16146 shutdown 2024-04-26T01:55:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 -- 172.21.15.28:0/3277915053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 msgr2=0x55f56f1f3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f3a40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0092a0 tx=0x7f39fc005be0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 -- 172.21.15.28:0/3277915053 shutdown_connections 2024-04-26T01:55:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f56f2624c0 0x55f56f2648b0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f56f1eba00 0x55f56f1f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f56f1e1140 0x55f56f1f0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 --2- 172.21.15.28:0/3277915053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f56f1e0820 0x55f56f1f3a40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 -- 172.21.15.28:0/3277915053 >> 172.21.15.28:0/3277915053 conn(0x55f56f1dcd30 msgr2=0x55f56f1e21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f39f8c00640 0 client.16146 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 -- 172.21.15.28:0/3277915053 shutdown_connections 2024-04-26T01:55:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.346+0000 7f3a18785740 1 -- 172.21.15.28:0/3277915053 wait complete. 2024-04-26T01:55:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.347+0000 7f3a18785740 20 client.16146 trim_cache size 0 max 16384 2024-04-26T01:55:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:55:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:55:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:55:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:55:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.348+0000 7f3a15800640 1 -- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f39f40a3e40 msgr2=0x7f39f40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.348+0000 7f3a15800640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f39f40a3e40 0x7f39f40a6300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800a010 tx=0x7f3a0800e040 comp rx=0 tx=0).stop 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.348+0000 7f3a15800640 1 -- 172.21.15.28:0/446880824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a1006bfa0 msgr2=0x7f3a1006c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.348+0000 7f3a15800640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a1006bfa0 0x7f3a1006c450 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000e9b0 tx=0x7f3a0000ee70 comp rx=0 tx=0).stop 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.349+0000 7f3a15800640 1 -- 172.21.15.28:0/446880824 shutdown_connections 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.349+0000 7f3a15800640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f39f40a3e40 0x7f39f40a6300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.349+0000 7f3a15800640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a1006bfa0 0x7f3a1006c450 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.349+0000 7f3a15800640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10109490 0x7f3a1006ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.349+0000 7f3a15800640 1 --2- 172.21.15.28:0/446880824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a101043c0 0x7f3a10073aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.349+0000 7f3a15800640 1 -- 172.21.15.28:0/446880824 >> 172.21.15.28:0/446880824 conn(0x7f3a100ffc80 msgr2=0x7f3a10101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.349+0000 7f3a15800640 1 -- 172.21.15.28:0/446880824 shutdown_connections 2024-04-26T01:55:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:43.349+0000 7f3a15800640 1 -- 172.21.15.28:0/446880824 wait complete. 2024-04-26T01:55:43.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:43 smithi169 ceph-mon[32851]: pgmap v463: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 MiB/s wr, 564 op/s 2024-04-26T01:55:43.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:43 smithi053 ceph-mon[32593]: pgmap v463: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 MiB/s wr, 564 op/s 2024-04-26T01:55:44.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:43 smithi028 ceph-mon[28247]: pgmap v463: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 MiB/s wr, 564 op/s 2024-04-26T01:55:45.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:45 smithi169 ceph-mon[32851]: pgmap v464: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-26T01:55:45.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:45 smithi053 ceph-mon[32593]: pgmap v464: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-26T01:55:46.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:45 smithi028 ceph-mon[28247]: pgmap v464: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-26T01:55:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 830.0 s, 1755.6 tps, lat 57.721 ms stddev 115.949 2024-04-26T01:55:47.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:47 smithi169 ceph-mon[32851]: pgmap v465: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 21 MiB/s wr, 1.77k op/s 2024-04-26T01:55:47.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:47 smithi053 ceph-mon[32593]: pgmap v465: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 21 MiB/s wr, 1.77k op/s 2024-04-26T01:55:48.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:47 smithi028 ceph-mon[28247]: pgmap v465: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 21 MiB/s wr, 1.77k op/s 2024-04-26T01:55:49.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:49 smithi169 ceph-mon[32851]: pgmap v466: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 20 MiB/s wr, 1.57k op/s 2024-04-26T01:55:49.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:49 smithi053 ceph-mon[32593]: pgmap v466: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 20 MiB/s wr, 1.57k op/s 2024-04-26T01:55:50.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:49 smithi028 ceph-mon[28247]: pgmap v466: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 20 MiB/s wr, 1.57k op/s 2024-04-26T01:55:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 835.0 s, 1727.8 tps, lat 57.937 ms stddev 72.414 2024-04-26T01:55:51.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:51 smithi169 ceph-mon[32851]: pgmap v467: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 22 MiB/s wr, 1.86k op/s 2024-04-26T01:55:51.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:51 smithi053 ceph-mon[32593]: pgmap v467: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 22 MiB/s wr, 1.86k op/s 2024-04-26T01:55:52.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:51 smithi028 ceph-mon[28247]: pgmap v467: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 22 MiB/s wr, 1.86k op/s 2024-04-26T01:55:53.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:53 smithi169 ceph-mon[32851]: pgmap v468: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 MiB/s wr, 1.60k op/s 2024-04-26T01:55:53.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:53 smithi053 ceph-mon[32593]: pgmap v468: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 MiB/s wr, 1.60k op/s 2024-04-26T01:55:54.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:53 smithi028 ceph-mon[28247]: pgmap v468: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 MiB/s wr, 1.60k op/s 2024-04-26T01:55:55.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:55 smithi169 ceph-mon[32851]: pgmap v469: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 21 MiB/s wr, 1.74k op/s 2024-04-26T01:55:55.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:55 smithi053 ceph-mon[32593]: pgmap v469: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 21 MiB/s wr, 1.74k op/s 2024-04-26T01:55:56.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:55 smithi028 ceph-mon[28247]: pgmap v469: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 21 MiB/s wr, 1.74k op/s 2024-04-26T01:55:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 840.0 s, 1656.8 tps, lat 59.799 ms stddev 71.847 2024-04-26T01:55:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.544+0000 7f053a200640 1 -- 172.21.15.28:0/2926650863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534104d70 msgr2=0x7f0534108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.544+0000 7f053a200640 1 --2- 172.21.15.28:0/2926650863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534104d70 0x7f0534108db0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0520009900 tx=0x7f052002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.545+0000 7f053a200640 1 -- 172.21.15.28:0/2926650863 shutdown_connections 2024-04-26T01:55:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.545+0000 7f053a200640 1 --2- 172.21.15.28:0/2926650863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0534109490 0x7f0534111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.545+0000 7f053a200640 1 --2- 172.21.15.28:0/2926650863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534104d70 0x7f0534108db0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.545+0000 7f053a200640 1 --2- 172.21.15.28:0/2926650863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05341043c0 0x7f05341047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.545+0000 7f053a200640 1 -- 172.21.15.28:0/2926650863 >> 172.21.15.28:0/2926650863 conn(0x7f05340ffc80 msgr2=0x7f05341020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.545+0000 7f053a200640 1 -- 172.21.15.28:0/2926650863 shutdown_connections 2024-04-26T01:55:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.545+0000 7f053a200640 1 -- 172.21.15.28:0/2926650863 wait complete. 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.546+0000 7f053a200640 1 Processor -- start 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.546+0000 7f053a200640 1 -- start start 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f053a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05341043c0 0x7f05341a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f053a200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534109490 0x7f05341a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f053a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053419a780 0x7f053419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f053a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0534114420 con 0x7f05341043c0 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f053a200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f05341142a0 con 0x7f053419a780 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f053a200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f05341145a0 con 0x7f0534109490 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f0532a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534109490 0x7f05341a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f0533400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05341043c0 0x7f05341a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f0532a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534109490 0x7f05341a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:34496/0 (socket says 172.21.15.28:34496) 2024-04-26T01:55:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f0532a00640 1 -- 172.21.15.28:0/1671335857 learned_addr learned my addr 172.21.15.28:0/1671335857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.547+0000 7f0533e00640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053419a780 0x7f053419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.548+0000 7f0532a00640 1 -- 172.21.15.28:0/1671335857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053419a780 msgr2=0x7f053419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.548+0000 7f0532a00640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053419a780 0x7f053419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.548+0000 7f0532a00640 1 -- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05341043c0 msgr2=0x7f05341a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.548+0000 7f0532a00640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05341043c0 0x7f05341a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.548+0000 7f0532a00640 1 -- 172.21.15.28:0/1671335857 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053419b310 con 0x7f0534109490 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.548+0000 7f0532a00640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534109490 0x7f05341a0b50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0520031440 tx=0x7f0520003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.548+0000 7f052be00640 1 -- 172.21.15.28:0/1671335857 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052003f070 con 0x7f0534109490 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.549+0000 7f052be00640 1 -- 172.21.15.28:0/1671335857 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0520031b50 con 0x7f0534109490 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.549+0000 7f052be00640 1 -- 172.21.15.28:0/1671335857 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0520035030 con 0x7f0534109490 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.549+0000 7f053a200640 1 -- 172.21.15.28:0/1671335857 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053419b5a0 con 0x7f0534109490 2024-04-26T01:55:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.549+0000 7f053a200640 1 -- 172.21.15.28:0/1671335857 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0534076570 con 0x7f0534109490 2024-04-26T01:55:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.554+0000 7f053cfc8740 1 Processor -- start 2024-04-26T01:55:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.554+0000 7f053cfc8740 1 -- start start 2024-04-26T01:55:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.554+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb643820 0x55bacb643c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.554+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 0x55bacb650df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f053cfc8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bacb5a9900 con 0x55bacb643820 2024-04-26T01:55:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f053cfc8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55bacb54fef0 con 0x55bacb644140 2024-04-26T01:55:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f053cfc8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55bacb5a58c0 con 0x55bacb64ea00 2024-04-26T01:55:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb643820 0x55bacb643c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57728/0 (socket says 172.21.15.28:57728) 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533400640 1 -- 172.21.15.28:0/2639007353 learned_addr learned my addr 172.21.15.28:0/2639007353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533400640 1 -- 172.21.15.28:0/2639007353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 msgr2=0x55bacb650df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533400640 1 --2- 172.21.15.28:0/2639007353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 0x55bacb650df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533400640 1 -- 172.21.15.28:0/2639007353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb643820 msgr2=0x55bacb643c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533400640 1 --2- 172.21.15.28:0/2639007353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb643820 0x55bacb643c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.555+0000 7f0533400640 1 -- 172.21.15.28:0/2639007353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bacb2bc210 con 0x55bacb644140 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.556+0000 7f0533400640 1 --2- 172.21.15.28:0/2639007353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64e4c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0524009d20 tx=0x7f0524011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=583d72a5f36f7718 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.556+0000 7f051fe00640 1 -- 172.21.15.28:0/2639007353 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052401e070 con 0x55bacb644140 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.556+0000 7f052be00640 1 -- 172.21.15.28:0/1671335857 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0520042070 con 0x7f0534109490 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.556+0000 7f051fe00640 1 -- 172.21.15.28:0/2639007353 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0524019ae0 con 0x55bacb644140 2024-04-26T01:55:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.556+0000 7f051fe00640 1 -- 172.21.15.28:0/2639007353 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052400ea60 con 0x55bacb644140 2024-04-26T01:55:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.557+0000 7f053cfc8740 1 -- 172.21.15.28:0/2639007353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 msgr2=0x55bacb64e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.557+0000 7f053cfc8740 1 --2- 172.21.15.28:0/2639007353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64e4c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0524009d20 tx=0x7f0524011810 comp rx=0 tx=0).stop 2024-04-26T01:55:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.557+0000 7f053cfc8740 1 -- 172.21.15.28:0/2639007353 shutdown_connections 2024-04-26T01:55:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.557+0000 7f053cfc8740 1 --2- 172.21.15.28:0/2639007353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 0x55bacb650df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.557+0000 7f053cfc8740 1 --2- 172.21.15.28:0/2639007353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64e4c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.557+0000 7f053cfc8740 1 --2- 172.21.15.28:0/2639007353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb643820 0x55bacb643c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.557+0000 7f053cfc8740 1 -- 172.21.15.28:0/2639007353 >> 172.21.15.28:0/2639007353 conn(0x55bacb63fd30 msgr2=0x55bacb640110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.558+0000 7f053cfc8740 1 -- 172.21.15.28:0/2639007353 shutdown_connections 2024-04-26T01:55:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.558+0000 7f053cfc8740 1 -- 172.21.15.28:0/2639007353 wait complete. 2024-04-26T01:55:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.558+0000 7f053cfc8740 1 -- start start 2024-04-26T01:55:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.559+0000 7f052be00640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f05180a3da0 0x7f05180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.559+0000 7f053cfc8740 1 Processor -- start 2024-04-26T01:55:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.559+0000 7f052be00640 1 -- 172.21.15.28:0/1671335857 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7f05200f0880 con 0x7f0534109490 2024-04-26T01:55:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.560+0000 7f0533e00640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f05180a3da0 0x7f05180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.560+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb643820 0x55bacb64ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.560+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb644140 0x55bacb64bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.560+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 0x55bacb64c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.560+0000 7f053cfc8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bacb651630 con 0x55bacb644140 2024-04-26T01:55:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.560+0000 7f053cfc8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55bacb563310 con 0x55bacb643820 2024-04-26T01:55:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.560+0000 7f053cfc8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55bacb6517b0 con 0x55bacb64ea00 2024-04-26T01:55:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.561+0000 7f053cfc8740 10 client.0 _ll_register_callbacks cb 0x55bacb531cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.561+0000 7f0532a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb644140 0x55bacb64bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.561+0000 7f0532a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb644140 0x55bacb64bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51724/0 (socket says 172.21.15.28:51724) 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.561+0000 7f0532a00640 1 -- 172.21.15.28:0/4186282143 learned_addr learned my addr 172.21.15.28:0/4186282143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.561+0000 7f0533e00640 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 0x55bacb64c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.561+0000 7f0533400640 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb643820 0x55bacb64ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.562+0000 7f0533e00640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f05180a3da0 0x7f05180a6260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f052c00ae50 tx=0x7f052c00a4d0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.562+0000 7f0532a00640 1 -- 172.21.15.28:0/4186282143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 msgr2=0x55bacb64c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.562+0000 7f0532a00640 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 0x55bacb64c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.562+0000 7f0532a00640 1 -- 172.21.15.28:0/4186282143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb643820 msgr2=0x55bacb64ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.562+0000 7f0532a00640 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb643820 0x55bacb64ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.562+0000 7f0532a00640 1 -- 172.21.15.28:0/4186282143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bacb388620 con 0x55bacb644140 2024-04-26T01:55:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.562+0000 7f0532a00640 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb644140 0x55bacb64bf50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f05200bc800 tx=0x7f05200bc470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0d9ff64accc4c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.562+0000 7f051cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-26T01:55:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.563+0000 7f051cc00640 1 -- 172.21.15.28:0/4186282143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05200c5430 con 0x55bacb644140 2024-04-26T01:55:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.563+0000 7f051cc00640 1 -- 172.21.15.28:0/4186282143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7f05200b8ee0 con 0x55bacb644140 2024-04-26T01:55:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.563+0000 7f053cfc8740 1 -- 172.21.15.28:0/4186282143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0524019ae0 con 0x55bacb644140 2024-04-26T01:55:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.565+0000 7f051cc00640 1 -- 172.21.15.28:0/4186282143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f05200a70b0 con 0x55bacb644140 2024-04-26T01:55:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.565+0000 7f053cfc8740 10 client.16155 fetch_fsmap learned FSMap version 47 2024-04-26T01:55:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.565+0000 7f053cfc8740 1 -- 172.21.15.28:0/4186282143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55bacb65d920 con 0x55bacb644140 2024-04-26T01:55:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.565+0000 7f051cc00640 1 -- 172.21.15.28:0/4186282143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0524019ae0 con 0x55bacb644140 2024-04-26T01:55:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.566+0000 7f051cc00640 1 -- 172.21.15.28:0/4186282143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f05200c4fa0 con 0x55bacb644140 2024-04-26T01:55:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.566+0000 7f053cfc8740 10 client.16155 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:55:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.566+0000 7f053cfc8740 10 client.16155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.566+0000 7f053cfc8740 20 client.16155 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.566+0000 7f053cfc8740 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55bacb65ed60 0x55bacb6611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.566+0000 7f053cfc8740 4 client.16155 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.566+0000 7f053cfc8740 1 -- 172.21.15.28:0/4186282143 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bacb5bd8b0 con 0x55bacb65ed60 2024-04-26T01:55:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.567+0000 7f0533400640 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55bacb65ed60 0x55bacb6611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.567+0000 7f0533400640 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55bacb65ed60 0x55bacb6611c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e604ecd902c9f8c server_cookie=d17a9523c7c3efc5 in_seq=0 out_seq=0 2024-04-26T01:55:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.567+0000 7f051cc00640 10 client.16155 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.568+0000 7f051cc00640 1 -- 172.21.15.28:0/4186282143 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bacb5bd8b0 con 0x55bacb65ed60 2024-04-26T01:55:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.568+0000 7f051cc00640 10 client.16155 handle_command_reply: tid=0 2024-04-26T01:55:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.568+0000 7f053cfc8740 1 client.16155 shutdown 2024-04-26T01:55:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.568+0000 7f053cfc8740 1 -- 172.21.15.28:0/4186282143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb644140 msgr2=0x55bacb64bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.568+0000 7f053cfc8740 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb644140 0x55bacb64bf50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f05200bc800 tx=0x7f05200bc470 comp rx=0 tx=0).stop 2024-04-26T01:55:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.568+0000 7f053cfc8740 1 -- 172.21.15.28:0/4186282143 shutdown_connections 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.568+0000 7f053cfc8740 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55bacb65ed60 0x55bacb6611c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.569+0000 7f053cfc8740 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb64ea00 0x55bacb64c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.569+0000 7f053cfc8740 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb644140 0x55bacb64bf50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.569+0000 7f053cfc8740 1 --2- 172.21.15.28:0/4186282143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb643820 0x55bacb64ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.569+0000 7f053cfc8740 1 -- 172.21.15.28:0/4186282143 >> 172.21.15.28:0/4186282143 conn(0x55bacb63fd30 msgr2=0x55bacb645ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.569+0000 7f051cc00640 0 client.16155 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.569+0000 7f053cfc8740 1 -- 172.21.15.28:0/4186282143 shutdown_connections 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.569+0000 7f053cfc8740 1 -- 172.21.15.28:0/4186282143 wait complete. 2024-04-26T01:55:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.569+0000 7f053cfc8740 20 client.16155 trim_cache size 0 max 16384 2024-04-26T01:55:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.604+0000 7f053cfc8740 1 Processor -- start 2024-04-26T01:55:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.604+0000 7f053cfc8740 1 -- start start 2024-04-26T01:55:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.605+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb64a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.605+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.605+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb6461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.605+0000 7f053cfc8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05341ae320 con 0x55bacb64ea00 2024-04-26T01:55:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.605+0000 7f053cfc8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f05341abbf0 con 0x55bacb644140 2024-04-26T01:55:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.605+0000 7f053cfc8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f05341a9a60 con 0x55bacb643820 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.605+0000 7f0533400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb64a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0532a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0532a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57750/0 (socket says 172.21.15.28:57750) 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0532a00640 1 -- 172.21.15.28:0/1622861616 learned_addr learned my addr 172.21.15.28:0/1622861616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0533e00640 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb6461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0532a00640 1 -- 172.21.15.28:0/1622861616 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 msgr2=0x55bacb64a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0532a00640 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb64a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0532a00640 1 -- 172.21.15.28:0/1622861616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 msgr2=0x55bacb6461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0532a00640 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb6461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.606+0000 7f0532a00640 1 -- 172.21.15.28:0/1622861616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bacb64aa00 con 0x55bacb644140 2024-04-26T01:55:56.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.607+0000 7f0533400640 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb64a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:55:56.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.607+0000 7f0533e00640 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb6461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:55:56.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.608+0000 7f0532a00640 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64c2d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f05200c4f00 tx=0x7f0520000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ea17265b8360707 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.609+0000 7f051f400640 1 -- 172.21.15.28:0/1622861616 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052003b460 con 0x55bacb644140 2024-04-26T01:55:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.609+0000 7f051f400640 1 -- 172.21.15.28:0/1622861616 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05200b8ee0 con 0x55bacb644140 2024-04-26T01:55:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.609+0000 7f051f400640 1 -- 172.21.15.28:0/1622861616 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05200bbd10 con 0x55bacb644140 2024-04-26T01:55:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.611+0000 7f053cfc8740 1 -- 172.21.15.28:0/1622861616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 msgr2=0x55bacb64c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.611+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64c2d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f05200c4f00 tx=0x7f0520000bc0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.612+0000 7f053cfc8740 1 -- 172.21.15.28:0/1622861616 shutdown_connections 2024-04-26T01:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.612+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb6461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.612+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb64c2d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.612+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1622861616 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb64a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.612+0000 7f053cfc8740 1 -- 172.21.15.28:0/1622861616 >> 172.21.15.28:0/1622861616 conn(0x55bacb63fd30 msgr2=0x55bacb63ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.613+0000 7f053cfc8740 1 -- 172.21.15.28:0/1622861616 shutdown_connections 2024-04-26T01:55:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.613+0000 7f053cfc8740 1 -- 172.21.15.28:0/1622861616 wait complete. 2024-04-26T01:55:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.614+0000 7f053cfc8740 1 -- start start 2024-04-26T01:55:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.614+0000 7f053cfc8740 1 Processor -- start 2024-04-26T01:55:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.615+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb656e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.615+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb6541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.615+0000 7f053cfc8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb654760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.615+0000 7f0533400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb656e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.615+0000 7f0532a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb6541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.616+0000 7f0532a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb6541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57766/0 (socket says 172.21.15.28:57766) 2024-04-26T01:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.616+0000 7f0533400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb656e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:34524/0 (socket says 172.21.15.28:34524) 2024-04-26T01:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.616+0000 7f053cfc8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05341aa0b0 con 0x55bacb64ea00 2024-04-26T01:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.616+0000 7f053cfc8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f05341ab520 con 0x55bacb644140 2024-04-26T01:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.616+0000 7f053cfc8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f05341aa780 con 0x55bacb643820 2024-04-26T01:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.616+0000 7f0532a00640 1 -- 172.21.15.28:0/1507534302 learned_addr learned my addr 172.21.15.28:0/1507534302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.616+0000 7f053cfc8740 10 client.0 _ll_register_callbacks cb 0x7f05200bc640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.617+0000 7f0533e00640 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb654760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.617+0000 7f0532a00640 1 -- 172.21.15.28:0/1507534302 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 msgr2=0x55bacb656e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.617+0000 7f0532a00640 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb656e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.617+0000 7f0532a00640 1 -- 172.21.15.28:0/1507534302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 msgr2=0x55bacb654760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.617+0000 7f0532a00640 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb654760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.617+0000 7f0532a00640 1 -- 172.21.15.28:0/1507534302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bacb654f90 con 0x55bacb644140 2024-04-26T01:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.618+0000 7f0532a00640 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb6541e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0520006010 tx=0x7f0520031f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9adc217768edc88 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.618+0000 7f051cc00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.618+0000 7f051cc00640 1 -- 172.21.15.28:0/1507534302 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052009f070 con 0x55bacb644140 2024-04-26T01:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.618+0000 7f051cc00640 1 -- 172.21.15.28:0/1507534302 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8513+0+0 (secure 0 0 0) 0x7f05200bbf10 con 0x55bacb644140 2024-04-26T01:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.618+0000 7f053cfc8740 1 -- 172.21.15.28:0/1507534302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05200b8ee0 con 0x55bacb644140 2024-04-26T01:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.618+0000 7f051cc00640 1 -- 172.21.15.28:0/1507534302 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f05200c5430 con 0x55bacb644140 2024-04-26T01:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f053cfc8740 10 client.26438 fetch_fsmap learned FSMap version 47 2024-04-26T01:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f053cfc8740 1 -- 172.21.15.28:0/1507534302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55bacb654f90 con 0x55bacb644140 2024-04-26T01:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f051cc00640 1 -- 172.21.15.28:0/1507534302 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05200b8ee0 con 0x55bacb644140 2024-04-26T01:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f051cc00640 1 -- 172.21.15.28:0/1507534302 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f05200a2030 con 0x55bacb644140 2024-04-26T01:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f053cfc8740 10 client.26438 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f053cfc8740 10 client.26438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f053cfc8740 20 client.26438 populate_metadata read hostname 'smithi028' 2024-04-26T01:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55bacb6d6640 0x55bacb6d8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f053cfc8740 4 client.26438 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f053cfc8740 1 -- 172.21.15.28:0/1507534302 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bacb5b7320 con 0x55bacb6d6640 2024-04-26T01:55:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.619+0000 7f0533400640 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55bacb6d6640 0x55bacb6d8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.622+0000 7f0533400640 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55bacb6d6640 0x55bacb6d8a60 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b30a6236115e6f5 server_cookie=4bb783bfcb78681e in_seq=0 out_seq=0 2024-04-26T01:55:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.622+0000 7f051cc00640 10 client.26438 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.645+0000 7f051cc00640 1 -- 172.21.15.28:0/1507534302 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bacb5b7320 con 0x55bacb6d6640 2024-04-26T01:55:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.645+0000 7f051cc00640 10 client.26438 handle_command_reply: tid=0 2024-04-26T01:55:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.645+0000 7f053cfc8740 1 client.26438 shutdown 2024-04-26T01:55:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 -- 172.21.15.28:0/1507534302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 msgr2=0x55bacb6541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb6541e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0520006010 tx=0x7f0520031f70 comp rx=0 tx=0).stop 2024-04-26T01:55:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 -- 172.21.15.28:0/1507534302 shutdown_connections 2024-04-26T01:55:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55bacb6d6640 0x55bacb6d8a60 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacb64ea00 0x55bacb654760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f051cc00640 0 client.26438 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:55:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bacb644140 0x55bacb6541e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 --2- 172.21.15.28:0/1507534302 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bacb643820 0x55bacb656e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 -- 172.21.15.28:0/1507534302 >> 172.21.15.28:0/1507534302 conn(0x55bacb63fd30 msgr2=0x55bacb63ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 -- 172.21.15.28:0/1507534302 shutdown_connections 2024-04-26T01:55:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 1 -- 172.21.15.28:0/1507534302 wait complete. 2024-04-26T01:55:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.646+0000 7f053cfc8740 20 client.26438 trim_cache size 0 max 16384 2024-04-26T01:55:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:55:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:55:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:55:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:55:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.647+0000 7f053a200640 1 -- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f05180a3da0 msgr2=0x7f05180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.647+0000 7f053a200640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f05180a3da0 0x7f05180a6260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f052c00ae50 tx=0x7f052c00a4d0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.647+0000 7f053a200640 1 -- 172.21.15.28:0/1671335857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534109490 msgr2=0x7f05341a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.647+0000 7f053a200640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534109490 0x7f05341a0b50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0520031440 tx=0x7f0520003c00 comp rx=0 tx=0).stop 2024-04-26T01:55:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.647+0000 7f053a200640 1 -- 172.21.15.28:0/1671335857 shutdown_connections 2024-04-26T01:55:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.647+0000 7f053a200640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f05180a3da0 0x7f05180a6260 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.647+0000 7f053a200640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053419a780 0x7f053419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.647+0000 7f053a200640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0534109490 0x7f05341a0b50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.648+0000 7f053a200640 1 --2- 172.21.15.28:0/1671335857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05341043c0 0x7f05341a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.648+0000 7f053a200640 1 -- 172.21.15.28:0/1671335857 >> 172.21.15.28:0/1671335857 conn(0x7f05340ffc80 msgr2=0x7f0534102010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.648+0000 7f053a200640 1 -- 172.21.15.28:0/1671335857 shutdown_connections 2024-04-26T01:55:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:55:56.648+0000 7f053a200640 1 -- 172.21.15.28:0/1671335857 wait complete. 2024-04-26T01:55:57.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:57 smithi169 ceph-mon[32851]: pgmap v470: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 17 MiB/s wr, 1.45k op/s 2024-04-26T01:55:57.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:57 smithi053 ceph-mon[32593]: pgmap v470: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 17 MiB/s wr, 1.45k op/s 2024-04-26T01:55:58.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:57 smithi028 ceph-mon[28247]: pgmap v470: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 17 MiB/s wr, 1.45k op/s 2024-04-26T01:55:59.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:55:59 smithi169 ceph-mon[32851]: pgmap v471: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.4 MiB/s wr, 583 op/s 2024-04-26T01:55:59.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:55:59 smithi053 ceph-mon[32593]: pgmap v471: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.4 MiB/s wr, 583 op/s 2024-04-26T01:56:00.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:55:59 smithi028 ceph-mon[28247]: pgmap v471: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.4 MiB/s wr, 583 op/s 2024-04-26T01:56:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 845.0 s, 1301.3 tps, lat 74.570 ms stddev 82.768 2024-04-26T01:56:01.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:01 smithi169 ceph-mon[32851]: pgmap v472: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.7 MiB/s wr, 843 op/s 2024-04-26T01:56:01.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:01 smithi053 ceph-mon[32593]: pgmap v472: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.7 MiB/s wr, 843 op/s 2024-04-26T01:56:02.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:01 smithi028 ceph-mon[28247]: pgmap v472: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.7 MiB/s wr, 843 op/s 2024-04-26T01:56:03.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:03 smithi169 ceph-mon[32851]: pgmap v473: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.0 MiB/s wr, 551 op/s 2024-04-26T01:56:03.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:03 smithi169 ceph-mon[32851]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:56:03.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:03 smithi053 ceph-mon[32593]: pgmap v473: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.0 MiB/s wr, 551 op/s 2024-04-26T01:56:03.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:03 smithi053 ceph-mon[32593]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:56:04.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:03 smithi028 ceph-mon[28247]: pgmap v473: 97 pgs: 97 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.0 MiB/s wr, 551 op/s 2024-04-26T01:56:04.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:03 smithi028 ceph-mon[28247]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:56:05.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:05 smithi169 ceph-mon[32851]: pgmap v475: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.9 MiB/s wr, 552 op/s 2024-04-26T01:56:05.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:05 smithi053 ceph-mon[32593]: pgmap v475: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.9 MiB/s wr, 552 op/s 2024-04-26T01:56:06.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:05 smithi028 ceph-mon[28247]: pgmap v475: 97 pgs: 97 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.9 MiB/s wr, 552 op/s 2024-04-26T01:56:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 850.0 s, 631.8 tps, lat 163.133 ms stddev 388.544 2024-04-26T01:56:07.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:07 smithi169 ceph-mon[32851]: pgmap v476: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 1.60k op/s 2024-04-26T01:56:07.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:07 smithi053 ceph-mon[32593]: pgmap v476: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 1.60k op/s 2024-04-26T01:56:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:07 smithi028 ceph-mon[28247]: pgmap v476: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 1.60k op/s 2024-04-26T01:56:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:09 smithi053 ceph-mon[32593]: pgmap v477: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 1.60k op/s 2024-04-26T01:56:09.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:09 smithi053 ceph-mon[32593]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:56:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:09 smithi028 ceph-mon[28247]: pgmap v477: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 1.60k op/s 2024-04-26T01:56:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:09 smithi028 ceph-mon[28247]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:56:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.843+0000 7f0665400640 1 -- 172.21.15.28:0/1863542381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 msgr2=0x7f06601047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.843+0000 7f0665400640 1 --2- 172.21.15.28:0/1863542381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 0x7f06601047a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0650009ca0 tx=0x7f065002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.843+0000 7f0665400640 1 -- 172.21.15.28:0/1863542381 shutdown_connections 2024-04-26T01:56:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.843+0000 7f0665400640 1 --2- 172.21.15.28:0/1863542381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0660109490 0x7f0660111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.843+0000 7f0665400640 1 --2- 172.21.15.28:0/1863542381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0660104d70 0x7f0660108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.843+0000 7f0665400640 1 --2- 172.21.15.28:0/1863542381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 0x7f06601047a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.843+0000 7f0665400640 1 -- 172.21.15.28:0/1863542381 >> 172.21.15.28:0/1863542381 conn(0x7f06600ffc80 msgr2=0x7f06601020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.844+0000 7f0665400640 1 -- 172.21.15.28:0/1863542381 shutdown_connections 2024-04-26T01:56:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.844+0000 7f0665400640 1 -- 172.21.15.28:0/1863542381 wait complete. 2024-04-26T01:56:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.845+0000 7f0665400640 1 Processor -- start 2024-04-26T01:56:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.845+0000 7f0665400640 1 -- start start 2024-04-26T01:56:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f0665400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 0x7f0660076460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f0665400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0660109490 0x7f06600769a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f0665400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0660076ee0 0x7f06601aae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f0665400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0660114420 con 0x7f0660076ee0 2024-04-26T01:56:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f0665400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06601142a0 con 0x7f0660109490 2024-04-26T01:56:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f0665400640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f06601145a0 con 0x7f06601043c0 2024-04-26T01:56:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f065e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 0x7f0660076460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f065e800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 0x7f0660076460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:39726/0 (socket says 172.21.15.28:39726) 2024-04-26T01:56:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.846+0000 7f065e800640 1 -- 172.21.15.28:0/2436014283 learned_addr learned my addr 172.21.15.28:0/2436014283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.847+0000 7f065f200640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0660076ee0 0x7f06601aae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.847+0000 7f0655e00640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0660109490 0x7f06600769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.847+0000 7f065e800640 1 -- 172.21.15.28:0/2436014283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0660109490 msgr2=0x7f06600769a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.847+0000 7f065e800640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0660109490 0x7f06600769a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.847+0000 7f065e800640 1 -- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0660076ee0 msgr2=0x7f06601aae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.847+0000 7f065e800640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0660076ee0 0x7f06601aae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.847+0000 7f065e800640 1 -- 172.21.15.28:0/2436014283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066019a5f0 con 0x7f06601043c0 2024-04-26T01:56:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.848+0000 7f065f200640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0660076ee0 0x7f06601aae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.848+0000 7f065e800640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 0x7f0660076460 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06500097b0 tx=0x7f06500319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.848+0000 7f0657e00640 1 -- 172.21.15.28:0/2436014283 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0650003bf0 con 0x7f06601043c0 2024-04-26T01:56:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.848+0000 7f0657e00640 1 -- 172.21.15.28:0/2436014283 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0650003d90 con 0x7f06601043c0 2024-04-26T01:56:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.848+0000 7f0665400640 1 -- 172.21.15.28:0/2436014283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066019a880 con 0x7f06601043c0 2024-04-26T01:56:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.849+0000 7f0657e00640 1 -- 172.21.15.28:0/2436014283 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065002fb20 con 0x7f06601043c0 2024-04-26T01:56:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.849+0000 7f0665400640 1 -- 172.21.15.28:0/2436014283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066019ae20 con 0x7f06601043c0 2024-04-26T01:56:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.854+0000 7f06683d3740 1 Processor -- start 2024-04-26T01:56:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.855+0000 7f06683d3740 1 -- start start 2024-04-26T01:56:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.855+0000 7f06683d3740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb42820 0x55f3cfb42c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.855+0000 7f06683d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb43140 0x55f3cfb4d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.855+0000 7f06683d3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 0x55f3cfb4fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.855+0000 7f06683d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3cfaa8900 con 0x55f3cfb43140 2024-04-26T01:56:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.855+0000 7f06683d3740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f3cfa4eef0 con 0x55f3cfb42820 2024-04-26T01:56:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.855+0000 7f06683d3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f3cfaa48c0 con 0x55f3cfb4da00 2024-04-26T01:56:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.856+0000 7f065f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb42820 0x55f3cfb42c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.856+0000 7f065f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb42820 0x55f3cfb42c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57004/0 (socket says 172.21.15.28:57004) 2024-04-26T01:56:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.856+0000 7f065f200640 1 -- 172.21.15.28:0/1918742644 learned_addr learned my addr 172.21.15.28:0/1918742644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.856+0000 7f0655e00640 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 0x55f3cfb4fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.856+0000 7f0657e00640 1 -- 172.21.15.28:0/2436014283 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0650004610 con 0x7f06601043c0 2024-04-26T01:56:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.856+0000 7f065f200640 1 -- 172.21.15.28:0/1918742644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 msgr2=0x55f3cfb4fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.856+0000 7f065f200640 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 0x55f3cfb4fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.856+0000 7f065f200640 1 -- 172.21.15.28:0/1918742644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb43140 msgr2=0x55f3cfb4d4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:56:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.857+0000 7f065e800640 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb43140 0x55f3cfb4d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.858+0000 7f065f200640 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb43140 0x55f3cfb4d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.858+0000 7f065f200640 1 -- 172.21.15.28:0/1918742644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3cf7bb210 con 0x55f3cfb42820 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.858+0000 7f065e800640 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb43140 0x55f3cfb4d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.858+0000 7f065f200640 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb42820 0x55f3cfb42c00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f065800bca0 tx=0x7f065800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bca21ebf7caf6f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.858+0000 7f0657e00640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06440a3f10 0x7f06440a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.858+0000 7f064f400640 1 -- 172.21.15.28:0/1918742644 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0658015070 con 0x55f3cfb42820 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.859+0000 7f064f400640 1 -- 172.21.15.28:0/1918742644 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06580054d0 con 0x55f3cfb42820 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.859+0000 7f064f400640 1 -- 172.21.15.28:0/1918742644 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0658004e60 con 0x55f3cfb42820 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.859+0000 7f0657e00640 1 -- 172.21.15.28:0/2436014283 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f06500f1cf0 con 0x7f06601043c0 2024-04-26T01:56:09.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.859+0000 7f065e800640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06440a3f10 0x7f06440a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 -- 172.21.15.28:0/1918742644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb42820 msgr2=0x55f3cfb42c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f065e800640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06440a3f10 0x7f06440a63d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55f3cfb43dc0 tx=0x7f065002adc0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb42820 0x55f3cfb42c00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f065800bca0 tx=0x7f065800f730 comp rx=0 tx=0).stop 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 -- 172.21.15.28:0/1918742644 shutdown_connections 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 0x55f3cfb4fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb43140 0x55f3cfb4d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 --2- 172.21.15.28:0/1918742644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb42820 0x55f3cfb42c00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 -- 172.21.15.28:0/1918742644 >> 172.21.15.28:0/1918742644 conn(0x55f3cfb3ed30 msgr2=0x55f3cfb3f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 -- 172.21.15.28:0/1918742644 shutdown_connections 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 -- 172.21.15.28:0/1918742644 wait complete. 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.860+0000 7f06683d3740 1 -- start start 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f06683d3740 1 Processor -- start 2024-04-26T01:56:09.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f06683d3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb4a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f06683d3740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb4af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f06683d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 0x55f3cfb4b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f06683d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3cfb50630 con 0x55f3cfb4da00 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f06683d3740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f3cfa62310 con 0x55f3cfb43140 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f06683d3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f3cfb507b0 con 0x55f3cfb42820 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f06683d3740 10 client.0 _ll_register_callbacks cb 0x55f3cfa30cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f065f200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb4a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f065f200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb4a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:39738/0 (socket says 172.21.15.28:39738) 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f065f200640 1 -- 172.21.15.28:0/2470670122 learned_addr learned my addr 172.21.15.28:0/2470670122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f065f200640 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 0x55f3cfb4b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.861+0000 7f0655e00640 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb4af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.862+0000 7f0655e00640 1 -- 172.21.15.28:0/2470670122 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 msgr2=0x55f3cfb4a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:09.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.862+0000 7f0655e00640 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb4a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.862+0000 7f0655e00640 1 -- 172.21.15.28:0/2470670122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 msgr2=0x55f3cfb4b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.862+0000 7f0655e00640 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 0x55f3cfb4b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.862+0000 7f0655e00640 1 -- 172.21.15.28:0/2470670122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3cf887620 con 0x55f3cfb43140 2024-04-26T01:56:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.862+0000 7f065f200640 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb4a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.862+0000 7f0655e00640 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb4af10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0660077bd0 tx=0x7f0648010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7250830c3e7e828 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.862+0000 7f0643e00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:56:09.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.863+0000 7f0643e00640 1 -- 172.21.15.28:0/2470670122 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064801e060 con 0x55f3cfb43140 2024-04-26T01:56:09.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.863+0000 7f0643e00640 1 -- 172.21.15.28:0/2470670122 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f064800fb60 con 0x55f3cfb43140 2024-04-26T01:56:09.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.863+0000 7f06683d3740 1 -- 172.21.15.28:0/2470670122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06580054d0 con 0x55f3cfb43140 2024-04-26T01:56:09.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.863+0000 7f0643e00640 1 -- 172.21.15.28:0/2470670122 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f0648010e60 con 0x55f3cfb43140 2024-04-26T01:56:09.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.863+0000 7f06683d3740 10 client.26447 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:09.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.863+0000 7f06683d3740 1 -- 172.21.15.28:0/2470670122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f3cfb5c8b0 con 0x55f3cfb43140 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.864+0000 7f0643e00640 1 -- 172.21.15.28:0/2470670122 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06580054d0 con 0x55f3cfb43140 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.864+0000 7f0643e00640 1 -- 172.21.15.28:0/2470670122 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f0648018370 con 0x55f3cfb43140 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.864+0000 7f06683d3740 10 client.26447 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.864+0000 7f06683d3740 10 client.26447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.864+0000 7f06683d3740 20 client.26447 populate_metadata read hostname 'smithi028' 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.864+0000 7f06683d3740 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3cfb5dd90 0x55f3cfb60180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.864+0000 7f06683d3740 4 client.26447 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.864+0000 7f06683d3740 1 -- 172.21.15.28:0/2470670122 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3cfabc8b0 con 0x55f3cfb5dd90 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.865+0000 7f065f200640 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3cfb5dd90 0x55f3cfb60180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.866+0000 7f065f200640 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3cfb5dd90 0x55f3cfb60180 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a96c68ab742d25 server_cookie=b39d7a6f012ef684 in_seq=0 out_seq=0 2024-04-26T01:56:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.866+0000 7f0643e00640 10 client.26447 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.866+0000 7f0643e00640 1 -- 172.21.15.28:0/2470670122 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3cfabc8b0 con 0x55f3cfb5dd90 2024-04-26T01:56:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.866+0000 7f0643e00640 10 client.26447 handle_command_reply: tid=0 2024-04-26T01:56:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.866+0000 7f06683d3740 1 client.26447 shutdown 2024-04-26T01:56:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 -- 172.21.15.28:0/2470670122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 msgr2=0x55f3cfb4af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb4af10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0660077bd0 tx=0x7f0648010c10 comp rx=0 tx=0).stop 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 -- 172.21.15.28:0/2470670122 shutdown_connections 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3cfb5dd90 0x55f3cfb60180 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 0x55f3cfb4b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f0643e00640 0 client.26447 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb4af10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 --2- 172.21.15.28:0/2470670122 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb4a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 -- 172.21.15.28:0/2470670122 >> 172.21.15.28:0/2470670122 conn(0x55f3cfb3ed30 msgr2=0x55f3cfb44ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 -- 172.21.15.28:0/2470670122 shutdown_connections 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 1 -- 172.21.15.28:0/2470670122 wait complete. 2024-04-26T01:56:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.867+0000 7f06683d3740 20 client.26447 trim_cache size 0 max 16384 2024-04-26T01:56:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.893+0000 7f06683d3740 1 Processor -- start 2024-04-26T01:56:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.893+0000 7f06683d3740 1 -- start start 2024-04-26T01:56:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.893+0000 7f06683d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb42820 0x55f3cfb3e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.893+0000 7f06683d3740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb594f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f06683d3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 0x55f3cfb65aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f06683d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06601bf240 con 0x55f3cfb42820 2024-04-26T01:56:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f06683d3740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06601bcb10 con 0x55f3cfb43140 2024-04-26T01:56:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f06683d3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f06601a13d0 con 0x55f3cfb4da00 2024-04-26T01:56:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f0655e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f065f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb42820 0x55f3cfb3e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f0655e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57022/0 (socket says 172.21.15.28:57022) 2024-04-26T01:56:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f0655e00640 1 -- 172.21.15.28:0/2228701747 learned_addr learned my addr 172.21.15.28:0/2228701747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f0655e00640 1 -- 172.21.15.28:0/2228701747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 msgr2=0x55f3cfb65aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f0655e00640 1 --2- 172.21.15.28:0/2228701747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 0x55f3cfb65aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f0655e00640 1 -- 172.21.15.28:0/2228701747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb42820 msgr2=0x55f3cfb3e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f0655e00640 1 --2- 172.21.15.28:0/2228701747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb42820 0x55f3cfb3e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.894+0000 7f0655e00640 1 -- 172.21.15.28:0/2228701747 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3cfb46630 con 0x55f3cfb43140 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.895+0000 7f0655e00640 1 --2- 172.21.15.28:0/2228701747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb594f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0648018900 tx=0x7f06480058f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfa0e475f96de1fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.895+0000 7f064ea00640 1 -- 172.21.15.28:0/2228701747 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648000e60 con 0x55f3cfb43140 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.895+0000 7f064ea00640 1 -- 172.21.15.28:0/2228701747 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0648018370 con 0x55f3cfb43140 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.895+0000 7f064ea00640 1 -- 172.21.15.28:0/2228701747 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648028080 con 0x55f3cfb43140 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 -- 172.21.15.28:0/2228701747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 msgr2=0x55f3cfb594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 --2- 172.21.15.28:0/2228701747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb594f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0648018900 tx=0x7f06480058f0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 -- 172.21.15.28:0/2228701747 shutdown_connections 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 --2- 172.21.15.28:0/2228701747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb4da00 0x55f3cfb65aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 --2- 172.21.15.28:0/2228701747 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb43140 0x55f3cfb594f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 --2- 172.21.15.28:0/2228701747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb42820 0x55f3cfb3e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 -- 172.21.15.28:0/2228701747 >> 172.21.15.28:0/2228701747 conn(0x55f3cfb3ed30 msgr2=0x55f3cfb44a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 -- 172.21.15.28:0/2228701747 shutdown_connections 2024-04-26T01:56:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 -- 172.21.15.28:0/2228701747 wait complete. 2024-04-26T01:56:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.896+0000 7f06683d3740 1 -- start start 2024-04-26T01:56:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.897+0000 7f06683d3740 1 Processor -- start 2024-04-26T01:56:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.897+0000 7f06683d3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb55f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.897+0000 7f06683d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 0x55f3cfb53250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.897+0000 7f06683d3740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb538a0 0x55f3cfb53d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.897+0000 7f06683d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06601a1a20 con 0x55f3cfb4da00 2024-04-26T01:56:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.897+0000 7f06683d3740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06601bc440 con 0x55f3cfb538a0 2024-04-26T01:56:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.897+0000 7f06683d3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f06601a20f0 con 0x55f3cfb42820 2024-04-26T01:56:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.897+0000 7f06683d3740 10 client.0 _ll_register_callbacks cb 0x7f064800ac40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb55f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb55f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:39750/0 (socket says 172.21.15.28:39750) 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 -- 172.21.15.28:0/2758418268 learned_addr learned my addr 172.21.15.28:0/2758418268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb538a0 0x55f3cfb53d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 -- 172.21.15.28:0/2758418268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb538a0 msgr2=0x55f3cfb53d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb538a0 0x55f3cfb53d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 -- 172.21.15.28:0/2758418268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 msgr2=0x55f3cfb53250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 0x55f3cfb53250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.898+0000 7f065f200640 1 -- 172.21.15.28:0/2758418268 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3cfbc35c0 con 0x55f3cfb42820 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.899+0000 7f065f200640 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb55f60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f065800bca0 tx=0x7f0658000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6397d4517bc53a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.899+0000 7f0643e00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.899+0000 7f0643e00640 1 -- 172.21.15.28:0/2758418268 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0658015040 con 0x55f3cfb42820 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.899+0000 7f0643e00640 1 -- 172.21.15.28:0/2758418268 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f065802c040 con 0x55f3cfb42820 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.899+0000 7f06683d3740 1 -- 172.21.15.28:0/2758418268 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0648018370 con 0x55f3cfb42820 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.899+0000 7f0643e00640 1 -- 172.21.15.28:0/2758418268 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f0658022040 con 0x55f3cfb42820 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.899+0000 7f06683d3740 10 client.26461 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.899+0000 7f06683d3740 1 -- 172.21.15.28:0/2758418268 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f3cfbc48b0 con 0x55f3cfb42820 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f0643e00640 1 -- 172.21.15.28:0/2758418268 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0648018370 con 0x55f3cfb42820 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f0643e00640 1 -- 172.21.15.28:0/2758418268 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f0658004df0 con 0x55f3cfb42820 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f06683d3740 10 client.26461 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f06683d3740 10 client.26461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f06683d3740 20 client.26461 populate_metadata read hostname 'smithi028' 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f06683d3740 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3cfbc4d70 0x55f3cfbc7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f06683d3740 4 client.26461 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f06683d3740 1 -- 172.21.15.28:0/2758418268 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f3cfab6320 con 0x55f3cfbc4d70 2024-04-26T01:56:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f0655e00640 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3cfbc4d70 0x55f3cfbc7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f0655e00640 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3cfbc4d70 0x55f3cfbc7160 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cae1a1fe1b666c server_cookie=feaee8fb9fcc2439 in_seq=0 out_seq=0 2024-04-26T01:56:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.900+0000 7f0643e00640 10 client.26461 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:09.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:09 smithi169 ceph-mon[32851]: pgmap v477: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 1.60k op/s 2024-04-26T01:56:09.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:09 smithi169 ceph-mon[32851]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:56:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f0643e00640 1 -- 172.21.15.28:0/2758418268 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f3cfab6320 con 0x55f3cfbc4d70 2024-04-26T01:56:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f0643e00640 10 client.26461 handle_command_reply: tid=0 2024-04-26T01:56:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 client.26461 shutdown 2024-04-26T01:56:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 -- 172.21.15.28:0/2758418268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 msgr2=0x55f3cfb55f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb55f60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f065800bca0 tx=0x7f0658000bc0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 -- 172.21.15.28:0/2758418268 shutdown_connections 2024-04-26T01:56:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f3cfbc4d70 0x55f3cfbc7160 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f0643e00640 0 client.26461 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3cfb538a0 0x55f3cfb53d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cfb4da00 0x55f3cfb53250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 --2- 172.21.15.28:0/2758418268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f3cfb42820 0x55f3cfb55f60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 -- 172.21.15.28:0/2758418268 >> 172.21.15.28:0/2758418268 conn(0x55f3cfb3ed30 msgr2=0x55f3cfb441d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 -- 172.21.15.28:0/2758418268 shutdown_connections 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.919+0000 7f06683d3740 1 -- 172.21.15.28:0/2758418268 wait complete. 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.920+0000 7f06683d3740 20 client.26461 trim_cache size 0 max 16384 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:56:09.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.920+0000 7f0665400640 1 -- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06440a3f10 msgr2=0x7f06440a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.920+0000 7f0665400640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06440a3f10 0x7f06440a63d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55f3cfb43dc0 tx=0x7f065002adc0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.920+0000 7f0665400640 1 -- 172.21.15.28:0/2436014283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 msgr2=0x7f0660076460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.920+0000 7f0665400640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 0x7f0660076460 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06500097b0 tx=0x7f06500319f0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.921+0000 7f0665400640 1 -- 172.21.15.28:0/2436014283 shutdown_connections 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.921+0000 7f0665400640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f06440a3f10 0x7f06440a63d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.921+0000 7f0665400640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0660076ee0 0x7f06601aae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.921+0000 7f0665400640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0660109490 0x7f06600769a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.921+0000 7f0665400640 1 --2- 172.21.15.28:0/2436014283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06601043c0 0x7f0660076460 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:09.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.921+0000 7f0665400640 1 -- 172.21.15.28:0/2436014283 >> 172.21.15.28:0/2436014283 conn(0x7f06600ffc80 msgr2=0x7f0660101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:09.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.922+0000 7f0665400640 1 -- 172.21.15.28:0/2436014283 shutdown_connections 2024-04-26T01:56:09.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:09.922+0000 7f0665400640 1 -- 172.21.15.28:0/2436014283 wait complete. 2024-04-26T01:56:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 855.0 s, 1878.4 tps, lat 53.676 ms stddev 71.825 2024-04-26T01:56:11.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:11 smithi169 ceph-mon[32851]: pgmap v479: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 22 MiB/s wr, 1.99k op/s 2024-04-26T01:56:11.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:11 smithi053 ceph-mon[32593]: pgmap v479: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 22 MiB/s wr, 1.99k op/s 2024-04-26T01:56:11.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:11 smithi028 ceph-mon[28247]: pgmap v479: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 22 MiB/s wr, 1.99k op/s 2024-04-26T01:56:13.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:13 smithi053 ceph-mon[32593]: pgmap v480: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 1.64k op/s 2024-04-26T01:56:13.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:13 smithi028 ceph-mon[28247]: pgmap v480: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 1.64k op/s 2024-04-26T01:56:13.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:13 smithi169 ceph-mon[32851]: pgmap v480: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 1.64k op/s 2024-04-26T01:56:15.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:15 smithi053 ceph-mon[32593]: pgmap v481: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 17 MiB/s wr, 1.54k op/s 2024-04-26T01:56:15.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:15 smithi028 ceph-mon[28247]: pgmap v481: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 17 MiB/s wr, 1.54k op/s 2024-04-26T01:56:15.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:15 smithi169 ceph-mon[32851]: pgmap v481: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 17 MiB/s wr, 1.54k op/s 2024-04-26T01:56:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 860.0 s, 1514.4 tps, lat 65.359 ms stddev 76.669 2024-04-26T01:56:17.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:17 smithi053 ceph-mon[32593]: pgmap v482: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.6 MiB/s wr, 625 op/s 2024-04-26T01:56:17.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:17 smithi028 ceph-mon[28247]: pgmap v482: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.6 MiB/s wr, 625 op/s 2024-04-26T01:56:17.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:17 smithi169 ceph-mon[32851]: pgmap v482: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.6 MiB/s wr, 625 op/s 2024-04-26T01:56:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:19 smithi028 ceph-mon[28247]: pgmap v483: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.6 MiB/s wr, 625 op/s 2024-04-26T01:56:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:19 smithi169 ceph-mon[32851]: pgmap v483: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.6 MiB/s wr, 625 op/s 2024-04-26T01:56:19.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:19 smithi053 ceph-mon[32593]: pgmap v483: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.6 MiB/s wr, 625 op/s 2024-04-26T01:56:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 865.0 s, 1516.2 tps, lat 66.318 ms stddev 76.222 2024-04-26T01:56:21.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:21 smithi028 ceph-mon[28247]: pgmap v484: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-04-26T01:56:21.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:21 smithi169 ceph-mon[32851]: pgmap v484: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-04-26T01:56:21.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:21 smithi053 ceph-mon[32593]: pgmap v484: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-04-26T01:56:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.102+0000 7fb6cbe00640 1 -- 172.21.15.28:0/837315944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c4109470 msgr2=0x7fb6c4111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.102+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/837315944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c4109470 0x7fb6c4111040 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009ca0 tx=0x7fb6bc02a8b0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.102+0000 7fb6cbe00640 1 -- 172.21.15.28:0/837315944 shutdown_connections 2024-04-26T01:56:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.102+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/837315944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c4109470 0x7fb6c4111040 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.102+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/837315944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6c4104d50 0x7fb6c4108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.102+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/837315944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c41043a0 0x7fb6c4104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.102+0000 7fb6cbe00640 1 -- 172.21.15.28:0/837315944 >> 172.21.15.28:0/837315944 conn(0x7fb6c40ffc40 msgr2=0x7fb6c4102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.103+0000 7fb6cbe00640 1 -- 172.21.15.28:0/837315944 shutdown_connections 2024-04-26T01:56:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.103+0000 7fb6cbe00640 1 -- 172.21.15.28:0/837315944 wait complete. 2024-04-26T01:56:23.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6cbe00640 1 Processor -- start 2024-04-26T01:56:23.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6cbe00640 1 -- start start 2024-04-26T01:56:23.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6cbe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6c41043a0 0x7fb6c419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c4109470 0x7fb6c419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6cbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c419f020 0x7fb6c419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6cbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41143c0 con 0x7fb6c4109470 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6cbe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4114240 con 0x7fb6c419f020 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6cbe00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4114540 con 0x7fb6c41043a0 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6c41043a0 0x7fb6c419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6c41043a0 0x7fb6c419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55636/0 (socket says 172.21.15.28:55636) 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.104+0000 7fb6c9400640 1 -- 172.21.15.28:0/191351916 learned_addr learned my addr 172.21.15.28:0/191351916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c4109470 0x7fb6c419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c9e00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c419f020 0x7fb6c419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c9400640 1 -- 172.21.15.28:0/191351916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c419f020 msgr2=0x7fb6c419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c9400640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c419f020 0x7fb6c419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c9400640 1 -- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c4109470 msgr2=0x7fb6c419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c9400640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c4109470 0x7fb6c419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c9400640 1 -- 172.21.15.28:0/191351916 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c419fb10 con 0x7fb6c41043a0 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c4109470 0x7fb6c419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c9e00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c419f020 0x7fb6c419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c9400640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6c41043a0 0x7fb6c419a760 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800c920 tx=0x7fb6b800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c2000640 1 -- 172.21.15.28:0/191351916 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8007c70 con 0x7fb6c41043a0 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6c2000640 1 -- 172.21.15.28:0/191351916 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b8007e10 con 0x7fb6c41043a0 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.105+0000 7fb6cbe00640 1 -- 172.21.15.28:0/191351916 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c419fe00 con 0x7fb6c41043a0 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.106+0000 7fb6c2000640 1 -- 172.21.15.28:0/191351916 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8018820 con 0x7fb6c41043a0 2024-04-26T01:56:23.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.106+0000 7fb6cbe00640 1 -- 172.21.15.28:0/191351916 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c41a75a0 con 0x7fb6c41043a0 2024-04-26T01:56:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.111+0000 7fb6cec2e740 1 Processor -- start 2024-04-26T01:56:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.111+0000 7fb6cec2e740 1 -- start start 2024-04-26T01:56:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.111+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258165b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.111+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f258178cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.111+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.111+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2580cb900 con 0x55f258166080 2024-04-26T01:56:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.111+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f258071ef0 con 0x55f258179210 2024-04-26T01:56:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.111+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f2580c78c0 con 0x55f258165760 2024-04-26T01:56:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c9e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258165b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c9e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258165b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55652/0 (socket says 172.21.15.28:55652) 2024-04-26T01:56:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c9e00640 1 -- 172.21.15.28:0/3978008753 learned_addr learned my addr 172.21.15.28:0/3978008753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c8a00640 1 -- 172.21.15.28:0/3978008753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 msgr2=0x55f258165b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/3978008753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258165b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c8a00640 1 -- 172.21.15.28:0/3978008753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 msgr2=0x55f258178cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:56:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/3978008753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f258178cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.112+0000 7fb6c8a00640 1 -- 172.21.15.28:0/3978008753 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f257dde210 con 0x55f258179210 2024-04-26T01:56:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.113+0000 7fb6c2000640 1 -- 172.21.15.28:0/191351916 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb6b800f450 con 0x7fb6c41043a0 2024-04-26T01:56:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.113+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/3978008753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817b600 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb6c406b6c0 tx=0x7fb6b0008f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=425954fce4511124 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.114+0000 7fb6b6600640 1 -- 172.21.15.28:0/3978008753 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0014070 con 0x55f258179210 2024-04-26T01:56:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.114+0000 7fb6b6600640 1 -- 172.21.15.28:0/3978008753 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b00054d0 con 0x55f258179210 2024-04-26T01:56:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.114+0000 7fb6b6600640 1 -- 172.21.15.28:0/3978008753 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0010690 con 0x55f258179210 2024-04-26T01:56:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6cec2e740 1 -- 172.21.15.28:0/3978008753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 msgr2=0x55f25817b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6c2000640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6ac0a3e90 0x7fb6ac0a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/3978008753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817b600 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb6c406b6c0 tx=0x7fb6b0008f90 comp rx=0 tx=0).stop 2024-04-26T01:56:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6c2000640 1 -- 172.21.15.28:0/191351916 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7fb6b80d0a30 con 0x7fb6c41043a0 2024-04-26T01:56:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6cec2e740 1 -- 172.21.15.28:0/3978008753 shutdown_connections 2024-04-26T01:56:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6c9e00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6ac0a3e90 0x7fb6ac0a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/3978008753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817b600 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/3978008753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f258178cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/3978008753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258165b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.115+0000 7fb6cec2e740 1 -- 172.21.15.28:0/3978008753 >> 172.21.15.28:0/3978008753 conn(0x55f258161d30 msgr2=0x55f258162110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.116+0000 7fb6cec2e740 1 -- 172.21.15.28:0/3978008753 shutdown_connections 2024-04-26T01:56:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.116+0000 7fb6cec2e740 1 -- 172.21.15.28:0/3978008753 wait complete. 2024-04-26T01:56:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.116+0000 7fb6c9e00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6ac0a3e90 0x7fb6ac0a6350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc031040 tx=0x7fb6bc009840 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.117+0000 7fb6cec2e740 1 -- start start 2024-04-26T01:56:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.117+0000 7fb6cec2e740 1 Processor -- start 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258165760 0x55f258176170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f2581766b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 0x55f258176bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f25817bde0 con 0x55f258166080 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f258085310 con 0x55f258165760 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55f25817bf60 con 0x55f258179210 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258165760 0x55f258176170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258165760 0x55f258176170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38526/0 (socket says 172.21.15.28:38526) 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6c9400640 1 -- 172.21.15.28:0/4030394342 learned_addr learned my addr 172.21.15.28:0/4030394342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6c9e00640 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 0x55f258176bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6cec2e740 10 client.0 _ll_register_callbacks cb 0x55f258053cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f2581766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6c9400640 1 -- 172.21.15.28:0/4030394342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 msgr2=0x55f258176bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6c9400640 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 0x55f258176bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.118+0000 7fb6c9400640 1 -- 172.21.15.28:0/4030394342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 msgr2=0x55f2581766b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6c9400640 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f2581766b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6c9400640 1 -- 172.21.15.28:0/4030394342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f257eaa620 con 0x55f258165760 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f2581766b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6c9400640 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258165760 0x55f258176170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8007590 tx=0x7fb6b809d130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e28b5a89cba042c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6aaa00640 1 -- 172.21.15.28:0/4030394342 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80a55e0 con 0x55f258165760 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6aaa00640 1 -- 172.21.15.28:0/4030394342 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7fb6b8098fe0 con 0x55f258165760 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6cec2e740 1 -- 172.21.15.28:0/4030394342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b00054d0 con 0x55f258165760 2024-04-26T01:56:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.119+0000 7fb6aaa00640 1 -- 172.21.15.28:0/4030394342 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb6b80ce3a0 con 0x55f258165760 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.120+0000 7fb6cec2e740 10 client.26465 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.120+0000 7fb6cec2e740 1 -- 172.21.15.28:0/4030394342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f258188120 con 0x55f258165760 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.120+0000 7fb6aaa00640 1 -- 172.21.15.28:0/4030394342 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b00054d0 con 0x55f258165760 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.120+0000 7fb6aaa00640 1 -- 172.21.15.28:0/4030394342 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fb6b80870b0 con 0x55f258165760 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.120+0000 7fb6cec2e740 10 client.26465 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.120+0000 7fb6cec2e740 10 client.26465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.121+0000 7fb6cec2e740 20 client.26465 populate_metadata read hostname 'smithi028' 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.121+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f258189630 0x55f25818baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.121+0000 7fb6cec2e740 4 client.26465 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.121+0000 7fb6cec2e740 1 -- 172.21.15.28:0/4030394342 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2580df8b0 con 0x55f258189630 2024-04-26T01:56:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.122+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f258189630 0x55f25818baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.123+0000 7fb6aaa00640 10 client.26465 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.123+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f258189630 0x55f25818baf0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6e3bec2c8766ab1 server_cookie=8753a8ff6dabeeb6 in_seq=0 out_seq=0 2024-04-26T01:56:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.124+0000 7fb6aaa00640 1 -- 172.21.15.28:0/4030394342 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2580df8b0 con 0x55f258189630 2024-04-26T01:56:23.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.124+0000 7fb6aaa00640 10 client.26465 handle_command_reply: tid=0 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 client.26465 shutdown 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 -- 172.21.15.28:0/4030394342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258165760 msgr2=0x55f258176170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258165760 0x55f258176170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8007590 tx=0x7fb6b809d130 comp rx=0 tx=0).stop 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 -- 172.21.15.28:0/4030394342 shutdown_connections 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f258189630 0x55f25818baf0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 0x55f258176bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f2581766b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/4030394342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258165760 0x55f258176170 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 -- 172.21.15.28:0/4030394342 >> 172.21.15.28:0/4030394342 conn(0x55f258161d30 msgr2=0x55f258167ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:23.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6aaa00640 0 client.26465 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:23.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.125+0000 7fb6cec2e740 1 -- 172.21.15.28:0/4030394342 shutdown_connections 2024-04-26T01:56:23.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.126+0000 7fb6cec2e740 1 -- 172.21.15.28:0/4030394342 wait complete. 2024-04-26T01:56:23.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.126+0000 7fb6cec2e740 20 client.26465 trim_cache size 0 max 16384 2024-04-26T01:56:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.157+0000 7fb6cec2e740 1 Processor -- start 2024-04-26T01:56:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.157+0000 7fb6cec2e740 1 -- start start 2024-04-26T01:56:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258165760 0x55f258184d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258166080 0x55f258166860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 0x55f258160d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41ae100 con 0x55f258165760 2024-04-26T01:56:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41ab9d0 con 0x55f258166080 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41a9880 con 0x55f258179210 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258166080 0x55f258166860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258166080 0x55f258166860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:38538/0 (socket says 172.21.15.28:38538) 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c8a00640 1 -- 172.21.15.28:0/2872318772 learned_addr learned my addr 172.21.15.28:0/2872318772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c9400640 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258165760 0x55f258184d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c9e00640 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 0x55f258160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c8a00640 1 -- 172.21.15.28:0/2872318772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 msgr2=0x55f258160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 0x55f258160d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c8a00640 1 -- 172.21.15.28:0/2872318772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258165760 msgr2=0x55f258184d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258165760 0x55f258184d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.158+0000 7fb6c8a00640 1 -- 172.21.15.28:0/2872318772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f258175160 con 0x55f258166080 2024-04-26T01:56:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.159+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258166080 0x55f258166860 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00057c0 tx=0x7fb6b0004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dda18b46bafb0b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.159+0000 7fb6b5c00640 1 -- 172.21.15.28:0/2872318772 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0014040 con 0x55f258166080 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.159+0000 7fb6b5c00640 1 -- 172.21.15.28:0/2872318772 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b002b070 con 0x55f258166080 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.159+0000 7fb6b5c00640 1 -- 172.21.15.28:0/2872318772 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0026580 con 0x55f258166080 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.160+0000 7fb6cec2e740 1 -- 172.21.15.28:0/2872318772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258166080 msgr2=0x55f258166860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.160+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258166080 0x55f258166860 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00057c0 tx=0x7fb6b0004c80 comp rx=0 tx=0).stop 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.160+0000 7fb6cec2e740 1 -- 172.21.15.28:0/2872318772 shutdown_connections 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.160+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258179210 0x55f258160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.160+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258166080 0x55f258166860 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.160+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/2872318772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258165760 0x55f258184d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.160+0000 7fb6cec2e740 1 -- 172.21.15.28:0/2872318772 >> 172.21.15.28:0/2872318772 conn(0x55f258161d30 msgr2=0x55f258160600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.161+0000 7fb6cec2e740 1 -- 172.21.15.28:0/2872318772 shutdown_connections 2024-04-26T01:56:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.161+0000 7fb6cec2e740 1 -- 172.21.15.28:0/2872318772 wait complete. 2024-04-26T01:56:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.161+0000 7fb6cec2e740 1 -- start start 2024-04-26T01:56:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6cec2e740 1 Processor -- start 2024-04-26T01:56:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258181730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f25817eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6cec2e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41a9ed0 con 0x55f258166080 2024-04-26T01:56:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41ab300 con 0x55f258179210 2024-04-26T01:56:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6cec2e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c41aa560 con 0x55f258165760 2024-04-26T01:56:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6cec2e740 10 client.0 _ll_register_callbacks cb 0x7fb6b809c560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258181730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6c8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f25817eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258181730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:55688/0 (socket says 172.21.15.28:55688) 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.162+0000 7fb6c9400640 1 -- 172.21.15.28:0/1782029942 learned_addr learned my addr 172.21.15.28:0/1782029942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6c9e00640 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6c9400640 1 -- 172.21.15.28:0/1782029942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 msgr2=0x55f25817f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6c9400640 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6c9400640 1 -- 172.21.15.28:0/1782029942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 msgr2=0x55f25817eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6c9400640 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f25817eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6c9400640 1 -- 172.21.15.28:0/1782029942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f258165eb0 con 0x55f258165760 2024-04-26T01:56:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f25817eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6c9400640 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258181730 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb6b809c260 tx=0x7fb6b80a0ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79b9fd765c78acdc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.163+0000 7fb6aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6aaa00640 1 -- 172.21.15.28:0/1782029942 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80a55e0 con 0x55f258165760 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6aaa00640 1 -- 172.21.15.28:0/1782029942 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7fb6b8098fe0 con 0x55f258165760 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6cec2e740 1 -- 172.21.15.28:0/1782029942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b002b070 con 0x55f258165760 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6aaa00640 1 -- 172.21.15.28:0/1782029942 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb6b8067330 con 0x55f258165760 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6cec2e740 10 client.26476 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6cec2e740 1 -- 172.21.15.28:0/1782029942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f2581efc00 con 0x55f258165760 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6aaa00640 1 -- 172.21.15.28:0/1782029942 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b002b070 con 0x55f258165760 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6aaa00640 1 -- 172.21.15.28:0/1782029942 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fb6b8098850 con 0x55f258165760 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.164+0000 7fb6cec2e740 10 client.26476 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.165+0000 7fb6cec2e740 10 client.26476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.165+0000 7fb6cec2e740 20 client.26476 populate_metadata read hostname 'smithi028' 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.165+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f2581f0380 0x55f2581f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.165+0000 7fb6cec2e740 4 client.26476 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.165+0000 7fb6cec2e740 1 -- 172.21.15.28:0/1782029942 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2580d9320 con 0x55f2581f0380 2024-04-26T01:56:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.166+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f2581f0380 0x55f2581f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.169+0000 7fb6c8a00640 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f2581f0380 0x55f2581f27a0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b22c039606ac3ac server_cookie=82de36a30a3385e6 in_seq=0 out_seq=0 2024-04-26T01:56:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.169+0000 7fb6aaa00640 10 client.26476 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.263+0000 7fb6aaa00640 1 -- 172.21.15.28:0/1782029942 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f2580d9320 con 0x55f2581f0380 2024-04-26T01:56:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.263+0000 7fb6aaa00640 10 client.26476 handle_command_reply: tid=0 2024-04-26T01:56:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.263+0000 7fb6cec2e740 1 client.26476 shutdown 2024-04-26T01:56:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 -- 172.21.15.28:0/1782029942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 msgr2=0x55f258181730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258181730 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb6b809c260 tx=0x7fb6b80a0ea0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 -- 172.21.15.28:0/1782029942 shutdown_connections 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55f2581f0380 0x55f2581f27a0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f258179210 0x55f25817f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258166080 0x55f25817eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 --2- 172.21.15.28:0/1782029942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f258165760 0x55f258181730 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 -- 172.21.15.28:0/1782029942 >> 172.21.15.28:0/1782029942 conn(0x55f258161d30 msgr2=0x55f258160e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6aaa00640 0 client.26476 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 -- 172.21.15.28:0/1782029942 shutdown_connections 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 1 -- 172.21.15.28:0/1782029942 wait complete. 2024-04-26T01:56:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.264+0000 7fb6cec2e740 20 client.26476 trim_cache size 0 max 16384 2024-04-26T01:56:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:56:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:56:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:56:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:56:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 -- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6ac0a3e90 msgr2=0x7fb6ac0a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6ac0a3e90 0x7fb6ac0a6350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc031040 tx=0x7fb6bc009840 comp rx=0 tx=0).stop 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 -- 172.21.15.28:0/191351916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6c41043a0 msgr2=0x7fb6c419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6c41043a0 0x7fb6c419a760 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800c920 tx=0x7fb6b800cde0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 -- 172.21.15.28:0/191351916 shutdown_connections 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb6ac0a3e90 0x7fb6ac0a6350 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6c419f020 0x7fb6c419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6c4109470 0x7fb6c419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 --2- 172.21.15.28:0/191351916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6c41043a0 0x7fb6c419a760 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 -- 172.21.15.28:0/191351916 >> 172.21.15.28:0/191351916 conn(0x7fb6c40ffc40 msgr2=0x7fb6c41005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 -- 172.21.15.28:0/191351916 shutdown_connections 2024-04-26T01:56:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:23.266+0000 7fb6cbe00640 1 -- 172.21.15.28:0/191351916 wait complete. 2024-04-26T01:56:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:23 smithi028 ceph-mon[28247]: pgmap v485: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.2 MiB/s wr, 533 op/s 2024-04-26T01:56:23.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:23 smithi169 ceph-mon[32851]: pgmap v485: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.2 MiB/s wr, 533 op/s 2024-04-26T01:56:23.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:23 smithi053 ceph-mon[32593]: pgmap v485: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.2 MiB/s wr, 533 op/s 2024-04-26T01:56:25.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:25 smithi028 ceph-mon[28247]: pgmap v486: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.3 MiB/s wr, 546 op/s 2024-04-26T01:56:25.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:25 smithi169 ceph-mon[32851]: pgmap v486: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.3 MiB/s wr, 546 op/s 2024-04-26T01:56:25.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:25 smithi053 ceph-mon[32593]: pgmap v486: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.3 MiB/s wr, 546 op/s 2024-04-26T01:56:26.226 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 870.0 s, 1452.8 tps, lat 68.732 ms stddev 74.914 2024-04-26T01:56:27.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:27 smithi028 ceph-mon[28247]: pgmap v487: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.8 MiB/s wr, 822 op/s 2024-04-26T01:56:27.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:27 smithi169 ceph-mon[32851]: pgmap v487: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.8 MiB/s wr, 822 op/s 2024-04-26T01:56:27.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:27 smithi053 ceph-mon[32593]: pgmap v487: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.8 MiB/s wr, 822 op/s 2024-04-26T01:56:29.772 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:29 smithi053 ceph-mon[32593]: pgmap v488: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.4 MiB/s wr, 560 op/s 2024-04-26T01:56:29.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:29 smithi028 ceph-mon[28247]: pgmap v488: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.4 MiB/s wr, 560 op/s 2024-04-26T01:56:29.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:29 smithi169 ceph-mon[32851]: pgmap v488: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.4 MiB/s wr, 560 op/s 2024-04-26T01:56:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 875.0 s, 1527.0 tps, lat 65.713 ms stddev 76.462 2024-04-26T01:56:31.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:31 smithi028 ceph-mon[28247]: pgmap v489: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.6 MiB/s wr, 813 op/s 2024-04-26T01:56:31.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:31 smithi169 ceph-mon[32851]: pgmap v489: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.6 MiB/s wr, 813 op/s 2024-04-26T01:56:31.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:31 smithi053 ceph-mon[32593]: pgmap v489: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.6 MiB/s wr, 813 op/s 2024-04-26T01:56:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:33 smithi028 ceph-mon[28247]: pgmap v490: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.9 MiB/s wr, 551 op/s 2024-04-26T01:56:33.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:33 smithi169 ceph-mon[32851]: pgmap v490: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.9 MiB/s wr, 551 op/s 2024-04-26T01:56:33.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:33 smithi053 ceph-mon[32593]: pgmap v490: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.9 MiB/s wr, 551 op/s 2024-04-26T01:56:35.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:35 smithi028 ceph-mon[28247]: pgmap v491: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.0 MiB/s wr, 561 op/s 2024-04-26T01:56:35.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:35 smithi169 ceph-mon[32851]: pgmap v491: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.0 MiB/s wr, 561 op/s 2024-04-26T01:56:35.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:35 smithi053 ceph-mon[32593]: pgmap v491: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.0 MiB/s wr, 561 op/s 2024-04-26T01:56:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 880.0 s, 1712.4 tps, lat 57.980 ms stddev 65.670 2024-04-26T01:56:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.466+0000 7f67dc800640 1 -- 172.21.15.28:0/1067527654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d4074320 msgr2=0x7f67d4074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.466+0000 7f67dc800640 1 --2- 172.21.15.28:0/1067527654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d4074320 0x7f67d4074700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009900 tx=0x7f67c802a4b0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.467+0000 7f67dc800640 1 -- 172.21.15.28:0/1067527654 shutdown_connections 2024-04-26T01:56:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.467+0000 7f67dc800640 1 --2- 172.21.15.28:0/1067527654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67d410e7d0 0x7f67d4110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.467+0000 7f67dc800640 1 --2- 172.21.15.28:0/1067527654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4074c40 0x7f67d410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.467+0000 7f67dc800640 1 --2- 172.21.15.28:0/1067527654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d4074320 0x7f67d4074700 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.467+0000 7f67dc800640 1 -- 172.21.15.28:0/1067527654 >> 172.21.15.28:0/1067527654 conn(0x7f67d40fda00 msgr2=0x7f67d40ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.467+0000 7f67dc800640 1 -- 172.21.15.28:0/1067527654 shutdown_connections 2024-04-26T01:56:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.467+0000 7f67dc800640 1 -- 172.21.15.28:0/1067527654 wait complete. 2024-04-26T01:56:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.469+0000 7f67dc800640 1 Processor -- start 2024-04-26T01:56:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.469+0000 7f67dc800640 1 -- start start 2024-04-26T01:56:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.469+0000 7f67dc800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67d4074320 0x7f67d41a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67dc800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4074c40 0x7f67d41a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67dc800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d410e7d0 0x7f67d419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67dc800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67d4114490 con 0x7f67d4074c40 2024-04-26T01:56:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67dc800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f67d4114310 con 0x7f67d4074320 2024-04-26T01:56:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67dc800640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f67d4114610 con 0x7f67d410e7d0 2024-04-26T01:56:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67da800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d410e7d0 0x7f67d419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4074c40 0x7f67d41a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67da800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d410e7d0 0x7f67d419a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54572/0 (socket says 172.21.15.28:54572) 2024-04-26T01:56:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67da800640 1 -- 172.21.15.28:0/2570908647 learned_addr learned my addr 172.21.15.28:0/2570908647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.470+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67d4074320 0x7f67d41a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.471+0000 7f67d9400640 1 -- 172.21.15.28:0/2570908647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d410e7d0 msgr2=0x7f67d419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.471+0000 7f67d9400640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d410e7d0 0x7f67d419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.471+0000 7f67d9400640 1 -- 172.21.15.28:0/2570908647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67d4074320 msgr2=0x7f67d41a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.471+0000 7f67d9400640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67d4074320 0x7f67d41a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.471+0000 7f67d9400640 1 -- 172.21.15.28:0/2570908647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d419b020 con 0x7f67d4074c40 2024-04-26T01:56:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.471+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67d4074320 0x7f67d41a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.474+0000 7f67d9400640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4074c40 0x7f67d41a0bd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f67c400c380 tx=0x7f67c400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.475+0000 7f67cea00640 1 -- 172.21.15.28:0/2570908647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4023070 con 0x7f67d4074c40 2024-04-26T01:56:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.475+0000 7f67cea00640 1 -- 172.21.15.28:0/2570908647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67c4015e20 con 0x7f67d4074c40 2024-04-26T01:56:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.475+0000 7f67dc800640 1 -- 172.21.15.28:0/2570908647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d419b310 con 0x7f67d4074c40 2024-04-26T01:56:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.475+0000 7f67cea00640 1 -- 172.21.15.28:0/2570908647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4016790 con 0x7f67d4074c40 2024-04-26T01:56:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.475+0000 7f67dc800640 1 -- 172.21.15.28:0/2570908647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d4101e20 con 0x7f67d4074c40 2024-04-26T01:56:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.484+0000 7f67df774740 1 Processor -- start 2024-04-26T01:56:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.484+0000 7f67df774740 1 -- start start 2024-04-26T01:56:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.484+0000 7f67df774740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800a700 0x56285800aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.485+0000 7f67df774740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800b020 0x5628580264d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.485+0000 7f67df774740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562858026a10 0x562858028e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.485+0000 7f67df774740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562857f70900 con 0x562858026a10 2024-04-26T01:56:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.485+0000 7f67df774740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562857f16ef0 con 0x56285800b020 2024-04-26T01:56:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.485+0000 7f67df774740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x562857f6c8c0 con 0x56285800a700 2024-04-26T01:56:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.486+0000 7f67d9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800b020 0x5628580264d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.486+0000 7f67d9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800b020 0x5628580264d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46502/0 (socket says 172.21.15.28:46502) 2024-04-26T01:56:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.486+0000 7f67d9e00640 1 -- 172.21.15.28:0/2821937271 learned_addr learned my addr 172.21.15.28:0/2821937271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.488+0000 7f67da800640 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800a700 0x56285800aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.488+0000 7f67d9400640 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562858026a10 0x562858028e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.489+0000 7f67da800640 1 -- 172.21.15.28:0/2821937271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800b020 msgr2=0x5628580264d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.489+0000 7f67da800640 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800b020 0x5628580264d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.489+0000 7f67da800640 1 -- 172.21.15.28:0/2821937271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562858026a10 msgr2=0x562858028e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.489+0000 7f67da800640 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562858026a10 0x562858028e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.489+0000 7f67da800640 1 -- 172.21.15.28:0/2821937271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562857c83210 con 0x56285800a700 2024-04-26T01:56:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.489+0000 7f67d9400640 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562858026a10 0x562858028e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.491+0000 7f67da800640 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800a700 0x56285800aae0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009f50 tx=0x7f67d00096b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95d46b6b02613329 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.491+0000 7f67c3000640 1 -- 172.21.15.28:0/2821937271 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0018020 con 0x56285800a700 2024-04-26T01:56:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.491+0000 7f67c3000640 1 -- 172.21.15.28:0/2821937271 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67d0014b30 con 0x56285800a700 2024-04-26T01:56:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.491+0000 7f67c3000640 1 -- 172.21.15.28:0/2821937271 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0013ae0 con 0x56285800a700 2024-04-26T01:56:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.492+0000 7f67df774740 1 -- 172.21.15.28:0/2821937271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800a700 msgr2=0x56285800aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.492+0000 7f67df774740 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800a700 0x56285800aae0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009f50 tx=0x7f67d00096b0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.494+0000 7f67df774740 1 -- 172.21.15.28:0/2821937271 shutdown_connections 2024-04-26T01:56:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.494+0000 7f67df774740 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562858026a10 0x562858028e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.494+0000 7f67df774740 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800b020 0x5628580264d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.494+0000 7f67df774740 1 --2- 172.21.15.28:0/2821937271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800a700 0x56285800aae0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.494+0000 7f67df774740 1 -- 172.21.15.28:0/2821937271 >> 172.21.15.28:0/2821937271 conn(0x562858006d30 msgr2=0x562858007110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.496+0000 7f67cea00640 1 -- 172.21.15.28:0/2570908647 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f67c4016930 con 0x7f67d4074c40 2024-04-26T01:56:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.496+0000 7f67df774740 1 -- 172.21.15.28:0/2821937271 shutdown_connections 2024-04-26T01:56:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.497+0000 7f67df774740 1 -- 172.21.15.28:0/2821937271 wait complete. 2024-04-26T01:56:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.498+0000 7f67df774740 1 -- start start 2024-04-26T01:56:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.499+0000 7f67df774740 1 Processor -- start 2024-04-26T01:56:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.500+0000 7f67df774740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x562858017d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.500+0000 7f67df774740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x5628580182d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67cea00640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67bc0a3dd0 0x7f67bc0a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67df774740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628580130f0 0x562858013570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67cea00640 1 -- 172.21.15.28:0/2570908647 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f67c40dbd20 con 0x7f67d4074c40 2024-04-26T01:56:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67d9e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x5628580182d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67d9e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x5628580182d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54592/0 (socket says 172.21.15.28:54592) 2024-04-26T01:56:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67da800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x562858017d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67da800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x562858017d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:46504/0 (socket says 172.21.15.28:46504) 2024-04-26T01:56:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67da800640 1 -- 172.21.15.28:0/2769509350 learned_addr learned my addr 172.21.15.28:0/2769509350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56285800d6a0 con 0x5628580130f0 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562857f2a310 con 0x56285800a700 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.501+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x56285800d820 con 0x56285800b020 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.502+0000 7f67df774740 10 client.0 _ll_register_callbacks cb 0x562857ef8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.502+0000 7f67d9400640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67bc0a3dd0 0x7f67bc0a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.502+0000 7f67d9e00640 1 -- 172.21.15.28:0/2769509350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 msgr2=0x562858017d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.502+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x562858017d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.502+0000 7f67d9e00640 1 -- 172.21.15.28:0/2769509350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628580130f0 msgr2=0x562858013570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.502+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628580130f0 0x562858013570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.502+0000 7f67d9e00640 1 -- 172.21.15.28:0/2769509350 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562857d4f620 con 0x56285800b020 2024-04-26T01:56:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.503+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x5628580182d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000c00 tx=0x7f67c8005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52979724722ab95e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.503+0000 7f67b7400640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:56:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.503+0000 7f67d9400640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67bc0a3dd0 0x7f67bc0a6290 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x56285800bb70 tx=0x7f67c4016230 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.503+0000 7f67b7400640 1 -- 172.21.15.28:0/2769509350 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8035040 con 0x56285800b020 2024-04-26T01:56:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.504+0000 7f67b7400640 1 -- 172.21.15.28:0/2769509350 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f67c8004950 con 0x56285800b020 2024-04-26T01:56:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.504+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67d0014b30 con 0x56285800b020 2024-04-26T01:56:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.505+0000 7f67b7400640 1 -- 172.21.15.28:0/2769509350 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f67c8004b50 con 0x56285800b020 2024-04-26T01:56:36.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.505+0000 7f67df774740 10 client.26488 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:36.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.505+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562858013f00 con 0x56285800b020 2024-04-26T01:56:36.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.505+0000 7f67b7400640 1 -- 172.21.15.28:0/2769509350 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67d0014b30 con 0x56285800b020 2024-04-26T01:56:36.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.506+0000 7f67b7400640 1 -- 172.21.15.28:0/2769509350 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f67c8066020 con 0x56285800b020 2024-04-26T01:56:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.507+0000 7f67df774740 10 client.26488 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.507+0000 7f67df774740 10 client.26488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.507+0000 7f67df774740 20 client.26488 populate_metadata read hostname 'smithi028' 2024-04-26T01:56:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.507+0000 7f67df774740 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562858023920 0x562858025d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.507+0000 7f67df774740 4 client.26488 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.507+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562857f848b0 con 0x562858023920 2024-04-26T01:56:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.507+0000 7f67da800640 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562858023920 0x562858025d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.508+0000 7f67da800640 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562858023920 0x562858025d40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd0e85765481aafe server_cookie=6c3710c1f57fd553 in_seq=0 out_seq=0 2024-04-26T01:56:36.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.508+0000 7f67b7400640 10 client.26488 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.508+0000 7f67b7400640 1 -- 172.21.15.28:0/2769509350 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562857f848b0 con 0x562858023920 2024-04-26T01:56:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.508+0000 7f67b7400640 10 client.26488 handle_command_reply: tid=0 2024-04-26T01:56:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 client.26488 shutdown 2024-04-26T01:56:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 msgr2=0x5628580182d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x5628580182d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000c00 tx=0x7f67c8005270 comp rx=0 tx=0).stop 2024-04-26T01:56:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 shutdown_connections 2024-04-26T01:56:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562858023920 0x562858025d40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628580130f0 0x562858013570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x5628580182d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 --2- 172.21.15.28:0/2769509350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x562858017d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 >> 172.21.15.28:0/2769509350 conn(0x562858006d30 msgr2=0x56285800cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.509+0000 7f67b7400640 0 client.26488 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.510+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 shutdown_connections 2024-04-26T01:56:36.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.510+0000 7f67df774740 1 -- 172.21.15.28:0/2769509350 wait complete. 2024-04-26T01:56:36.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.510+0000 7f67df774740 20 client.26488 trim_cache size 0 max 16384 2024-04-26T01:56:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.543+0000 7f67df774740 1 Processor -- start 2024-04-26T01:56:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.544+0000 7f67df774740 1 -- start start 2024-04-26T01:56:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.544+0000 7f67df774740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800a700 0x56285802e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.544+0000 7f67df774740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x56285802e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.544+0000 7f67df774740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628580130f0 0x562858017940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.544+0000 7f67df774740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67d4072670 con 0x56285800a700 2024-04-26T01:56:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.544+0000 7f67df774740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f67d406ff40 con 0x5628580130f0 2024-04-26T01:56:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.544+0000 7f67df774740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f67d406ddb0 con 0x56285800b020 2024-04-26T01:56:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.545+0000 7f67da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800a700 0x56285802e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.545+0000 7f67da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800a700 0x56285802e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47286/0 (socket says 172.21.15.28:47286) 2024-04-26T01:56:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.545+0000 7f67da800640 1 -- 172.21.15.28:0/3568351570 learned_addr learned my addr 172.21.15.28:0/3568351570 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.545+0000 7f67d9e00640 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x56285802e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.545+0000 7f67da800640 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628580130f0 0x562858017940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.546+0000 7f67da800640 1 -- 172.21.15.28:0/3568351570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 msgr2=0x56285802e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.546+0000 7f67da800640 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x56285802e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.546+0000 7f67da800640 1 -- 172.21.15.28:0/3568351570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628580130f0 msgr2=0x562858017940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.546+0000 7f67da800640 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628580130f0 0x562858017940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.546+0000 7f67da800640 1 -- 172.21.15.28:0/3568351570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628580120a0 con 0x56285800a700 2024-04-26T01:56:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.546+0000 7f67da800640 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800a700 0x56285802e430 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f67d0013730 tx=0x7f67d00096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a496f0d2d3559a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.547+0000 7f67c2600640 1 -- 172.21.15.28:0/3568351570 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0028040 con 0x56285800a700 2024-04-26T01:56:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.547+0000 7f67c2600640 1 -- 172.21.15.28:0/3568351570 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67d0014a80 con 0x56285800a700 2024-04-26T01:56:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.547+0000 7f67c2600640 1 -- 172.21.15.28:0/3568351570 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0018040 con 0x56285800a700 2024-04-26T01:56:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.547+0000 7f67df774740 1 -- 172.21.15.28:0/3568351570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800a700 msgr2=0x56285802e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.547+0000 7f67df774740 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800a700 0x56285802e430 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f67d0013730 tx=0x7f67d00096b0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.548+0000 7f67df774740 1 -- 172.21.15.28:0/3568351570 shutdown_connections 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.548+0000 7f67df774740 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628580130f0 0x562858017940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.548+0000 7f67df774740 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56285800b020 0x56285802e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.548+0000 7f67df774740 1 --2- 172.21.15.28:0/3568351570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800a700 0x56285802e430 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.548+0000 7f67df774740 1 -- 172.21.15.28:0/3568351570 >> 172.21.15.28:0/3568351570 conn(0x562858006d30 msgr2=0x562858007110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.548+0000 7f67df774740 1 -- 172.21.15.28:0/3568351570 shutdown_connections 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.548+0000 7f67df774740 1 -- 172.21.15.28:0/3568351570 wait complete. 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.548+0000 7f67df774740 1 -- start start 2024-04-26T01:56:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.549+0000 7f67df774740 1 Processor -- start 2024-04-26T01:56:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.550+0000 7f67df774740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x56285801a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.550+0000 7f67df774740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800b020 0x56285801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.550+0000 7f67df774740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5628580130f0 0x56285801b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.550+0000 7f67df774740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67d406e400 con 0x56285800b020 2024-04-26T01:56:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.550+0000 7f67df774740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f67d406f870 con 0x56285800a700 2024-04-26T01:56:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.550+0000 7f67df774740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f67d406ead0 con 0x5628580130f0 2024-04-26T01:56:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.550+0000 7f67df774740 10 client.0 _ll_register_callbacks cb 0x7f67c8004530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.550+0000 7f67da800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5628580130f0 0x56285801b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.551+0000 7f67da800640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5628580130f0 0x56285801b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:54612/0 (socket says 172.21.15.28:54612) 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.551+0000 7f67da800640 1 -- 172.21.15.28:0/2526574980 learned_addr learned my addr 172.21.15.28:0/2526574980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.551+0000 7f67da800640 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x56285801a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.551+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800b020 0x56285801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.551+0000 7f67da800640 1 -- 172.21.15.28:0/2526574980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 msgr2=0x56285801a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.551+0000 7f67da800640 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x56285801a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.551+0000 7f67da800640 1 -- 172.21.15.28:0/2526574980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800b020 msgr2=0x56285801ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.552+0000 7f67da800640 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800b020 0x56285801ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.552+0000 7f67da800640 1 -- 172.21.15.28:0/2526574980 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56285808bca0 con 0x5628580130f0 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.552+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800b020 0x56285801ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.552+0000 7f67da800640 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5628580130f0 0x56285801b350 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009a70 tx=0x7f67d0020040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65cf2f247304feba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.552+0000 7f67b7400640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.553+0000 7f67b7400640 1 -- 172.21.15.28:0/2526574980 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d003b060 con 0x5628580130f0 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.553+0000 7f67b7400640 1 -- 172.21.15.28:0/2526574980 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f67d0014550 con 0x5628580130f0 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.553+0000 7f67df774740 1 -- 172.21.15.28:0/2526574980 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67d0014a80 con 0x5628580130f0 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.553+0000 7f67b7400640 1 -- 172.21.15.28:0/2526574980 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f67d0018040 con 0x5628580130f0 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.553+0000 7f67df774740 10 client.26494 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.553+0000 7f67df774740 1 -- 172.21.15.28:0/2526574980 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56285808cec0 con 0x5628580130f0 2024-04-26T01:56:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.554+0000 7f67b7400640 1 -- 172.21.15.28:0/2526574980 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67d0014a80 con 0x5628580130f0 2024-04-26T01:56:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.554+0000 7f67b7400640 1 -- 172.21.15.28:0/2526574980 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f67d0014ce0 con 0x5628580130f0 2024-04-26T01:56:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.554+0000 7f67df774740 10 client.26494 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.554+0000 7f67df774740 10 client.26494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.554+0000 7f67df774740 20 client.26494 populate_metadata read hostname 'smithi028' 2024-04-26T01:56:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.554+0000 7f67df774740 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56285808d3d0 0x56285808f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.554+0000 7f67df774740 4 client.26494 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.554+0000 7f67df774740 1 -- 172.21.15.28:0/2526574980 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562857f7e320 con 0x56285808d3d0 2024-04-26T01:56:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.555+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56285808d3d0 0x56285808f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.557+0000 7f67d9e00640 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56285808d3d0 0x56285808f7f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0217465883e57b2 server_cookie=f236c4638a5779d9 in_seq=0 out_seq=0 2024-04-26T01:56:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.557+0000 7f67b7400640 10 client.26494 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.570+0000 7f67b7400640 1 -- 172.21.15.28:0/2526574980 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562857f7e320 con 0x56285808d3d0 2024-04-26T01:56:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.570+0000 7f67b7400640 10 client.26494 handle_command_reply: tid=0 2024-04-26T01:56:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 client.26494 shutdown 2024-04-26T01:56:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 -- 172.21.15.28:0/2526574980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5628580130f0 msgr2=0x56285801b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5628580130f0 0x56285801b350 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009a70 tx=0x7f67d0020040 comp rx=0 tx=0).stop 2024-04-26T01:56:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 -- 172.21.15.28:0/2526574980 shutdown_connections 2024-04-26T01:56:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x56285808d3d0 0x56285808f7f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5628580130f0 0x56285801b350 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56285800b020 0x56285801ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 --2- 172.21.15.28:0/2526574980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56285800a700 0x56285801a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67b7400640 0 client.26494 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 -- 172.21.15.28:0/2526574980 >> 172.21.15.28:0/2526574980 conn(0x562858006d30 msgr2=0x56285800c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 -- 172.21.15.28:0/2526574980 shutdown_connections 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 1 -- 172.21.15.28:0/2526574980 wait complete. 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.571+0000 7f67df774740 20 client.26494 trim_cache size 0 max 16384 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:56:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:56:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:56:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:56:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 -- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67bc0a3dd0 msgr2=0x7f67bc0a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67bc0a3dd0 0x7f67bc0a6290 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x56285800bb70 tx=0x7f67c4016230 comp rx=0 tx=0).stop 2024-04-26T01:56:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 -- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4074c40 msgr2=0x7f67d41a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4074c40 0x7f67d41a0bd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f67c400c380 tx=0x7f67c400c840 comp rx=0 tx=0).stop 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 -- 172.21.15.28:0/2570908647 shutdown_connections 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f67bc0a3dd0 0x7f67bc0a6290 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d410e7d0 0x7f67d419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4074c40 0x7f67d41a0bd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 --2- 172.21.15.28:0/2570908647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67d4074320 0x7f67d41a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.572+0000 7f67dc800640 1 -- 172.21.15.28:0/2570908647 >> 172.21.15.28:0/2570908647 conn(0x7f67d40fda00 msgr2=0x7f67d40fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.573+0000 7f67dc800640 1 -- 172.21.15.28:0/2570908647 shutdown_connections 2024-04-26T01:56:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:36.573+0000 7f67dc800640 1 -- 172.21.15.28:0/2570908647 wait complete. 2024-04-26T01:56:37.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:37 smithi028 ceph-mon[28247]: pgmap v492: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.7 MiB/s wr, 849 op/s 2024-04-26T01:56:37.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:37 smithi169 ceph-mon[32851]: pgmap v492: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.7 MiB/s wr, 849 op/s 2024-04-26T01:56:37.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:37 smithi053 ceph-mon[32593]: pgmap v492: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.7 MiB/s wr, 849 op/s 2024-04-26T01:56:39.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:39 smithi169 ceph-mon[32851]: pgmap v493: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 MiB/s wr, 564 op/s 2024-04-26T01:56:39.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:39 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:39.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:39 smithi053 ceph-mon[32593]: pgmap v493: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 MiB/s wr, 564 op/s 2024-04-26T01:56:39.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:39 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:40.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:39 smithi028 ceph-mon[28247]: pgmap v493: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 MiB/s wr, 564 op/s 2024-04-26T01:56:40.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:39 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 885.0 s, 1535.7 tps, lat 64.555 ms stddev 73.933 2024-04-26T01:56:41.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:41 smithi169 ceph-mon[32851]: pgmap v494: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.2 MiB/s wr, 793 op/s 2024-04-26T01:56:41.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:41 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:41.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:41 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:41.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:41 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:56:41.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:41 smithi053 ceph-mon[32593]: pgmap v494: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.2 MiB/s wr, 793 op/s 2024-04-26T01:56:41.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:41 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:41.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:41 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:41.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:41 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:56:42.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:41 smithi028 ceph-mon[28247]: pgmap v494: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.2 MiB/s wr, 793 op/s 2024-04-26T01:56:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:41 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:41 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:41 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:56:43.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:43 smithi169 ceph-mon[32851]: pgmap v495: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 572 op/s 2024-04-26T01:56:43.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:43 smithi053 ceph-mon[32593]: pgmap v495: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 572 op/s 2024-04-26T01:56:44.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:43 smithi028 ceph-mon[28247]: pgmap v495: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 572 op/s 2024-04-26T01:56:45.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:45 smithi169 ceph-mon[32851]: pgmap v496: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 631 op/s 2024-04-26T01:56:45.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:45 smithi053 ceph-mon[32593]: pgmap v496: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 631 op/s 2024-04-26T01:56:46.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:45 smithi028 ceph-mon[28247]: pgmap v496: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 631 op/s 2024-04-26T01:56:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 890.0 s, 982.6 tps, lat 102.400 ms stddev 185.346 2024-04-26T01:56:47.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:47 smithi169 ceph-mon[32851]: pgmap v497: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.88k op/s 2024-04-26T01:56:47.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:47 smithi053 ceph-mon[32593]: pgmap v497: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.88k op/s 2024-04-26T01:56:48.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:47 smithi028 ceph-mon[28247]: pgmap v497: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.88k op/s 2024-04-26T01:56:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.767+0000 7f004aa00640 1 -- 172.21.15.28:0/4122390757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0044104d70 msgr2=0x7f0044108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.767+0000 7f004aa00640 1 --2- 172.21.15.28:0/4122390757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0044104d70 0x7f0044108db0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f002c009900 tx=0x7f002c02a400 comp rx=0 tx=0).stop 2024-04-26T01:56:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.767+0000 7f004aa00640 1 -- 172.21.15.28:0/4122390757 shutdown_connections 2024-04-26T01:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.767+0000 7f004aa00640 1 --2- 172.21.15.28:0/4122390757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0044109490 0x7f0044111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.767+0000 7f004aa00640 1 --2- 172.21.15.28:0/4122390757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0044104d70 0x7f0044108db0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.767+0000 7f004aa00640 1 --2- 172.21.15.28:0/4122390757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00441043c0 0x7f00441047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.767+0000 7f004aa00640 1 -- 172.21.15.28:0/4122390757 >> 172.21.15.28:0/4122390757 conn(0x7f00440ffc60 msgr2=0x7f0044102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.768+0000 7f004aa00640 1 -- 172.21.15.28:0/4122390757 shutdown_connections 2024-04-26T01:56:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.768+0000 7f004aa00640 1 -- 172.21.15.28:0/4122390757 wait complete. 2024-04-26T01:56:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.770+0000 7f004aa00640 1 Processor -- start 2024-04-26T01:56:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.771+0000 7f004aa00640 1 -- start start 2024-04-26T01:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.771+0000 7f004aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00441043c0 0x7f00441a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.771+0000 7f004aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0044109490 0x7f00441a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.771+0000 7f004aa00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f004419a660 0x7f004419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.771+0000 7f004aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0044114400 con 0x7f00441043c0 2024-04-26T01:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.771+0000 7f004aa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0044114280 con 0x7f0044109490 2024-04-26T01:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.771+0000 7f004aa00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0044114580 con 0x7f004419a660 2024-04-26T01:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.772+0000 7f0043e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00441043c0 0x7f00441a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.772+0000 7f0048a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f004419a660 0x7f004419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.772+0000 7f003b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0044109490 0x7f00441a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.772+0000 7f0048a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f004419a660 0x7f004419aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59308/0 (socket says 172.21.15.28:59308) 2024-04-26T01:56:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.772+0000 7f0048a00640 1 -- 172.21.15.28:0/3914287373 learned_addr learned my addr 172.21.15.28:0/3914287373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.772+0000 7f003b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0044109490 0x7f00441a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:33798/0 (socket says 172.21.15.28:33798) 2024-04-26T01:56:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.773+0000 7f0043e00640 1 -- 172.21.15.28:0/3914287373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f004419a660 msgr2=0x7f004419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.773+0000 7f0043e00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f004419a660 0x7f004419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.773+0000 7f0043e00640 1 -- 172.21.15.28:0/3914287373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0044109490 msgr2=0x7f00441a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.773+0000 7f0043e00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0044109490 0x7f00441a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.773+0000 7f0043e00640 1 -- 172.21.15.28:0/3914287373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004419b260 con 0x7f00441043c0 2024-04-26T01:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.773+0000 7f003b400640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0044109490 0x7f00441a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.774+0000 7f0043e00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00441043c0 0x7f00441a05a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f003400dab0 tx=0x7f003400deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.774+0000 7f0041600640 1 -- 172.21.15.28:0/3914287373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003400bd70 con 0x7f00441043c0 2024-04-26T01:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.774+0000 7f0041600640 1 -- 172.21.15.28:0/3914287373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0034003e20 con 0x7f00441043c0 2024-04-26T01:56:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.774+0000 7f0041600640 1 -- 172.21.15.28:0/3914287373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00340106c0 con 0x7f00441043c0 2024-04-26T01:56:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.775+0000 7f004aa00640 1 -- 172.21.15.28:0/3914287373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00441a7340 con 0x7f00441043c0 2024-04-26T01:56:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.775+0000 7f004aa00640 1 -- 172.21.15.28:0/3914287373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00441a76f0 con 0x7f00441043c0 2024-04-26T01:56:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.780+0000 7f0041600640 1 -- 172.21.15.28:0/3914287373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0034010860 con 0x7f00441043c0 2024-04-26T01:56:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.780+0000 7f004d7f0740 1 Processor -- start 2024-04-26T01:56:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.781+0000 7f004d7f0740 1 -- start start 2024-04-26T01:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.782+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e030 0x557f2815e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.782+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f2815e950 0x557f28168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.782+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f28169210 0x557f2816b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.782+0000 7f004d7f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f280bb900 con 0x557f2815e030 2024-04-26T01:56:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.782+0000 7f004d7f0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557f28061ef0 con 0x557f28169210 2024-04-26T01:56:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.782+0000 7f004d7f0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557f280b78c0 con 0x557f2815e950 2024-04-26T01:56:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.782+0000 7f0048a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e030 0x557f2815e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.782+0000 7f003b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f28169210 0x557f2816b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.783+0000 7f003b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f28169210 0x557f2816b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:33810/0 (socket says 172.21.15.28:33810) 2024-04-26T01:56:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.783+0000 7f003b400640 1 -- 172.21.15.28:0/984503967 learned_addr learned my addr 172.21.15.28:0/984503967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.783+0000 7f0043e00640 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f2815e950 0x557f28168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.783+0000 7f003b400640 1 -- 172.21.15.28:0/984503967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f2815e950 msgr2=0x557f28168cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.783+0000 7f003b400640 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f2815e950 0x557f28168cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.783+0000 7f003b400640 1 -- 172.21.15.28:0/984503967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e030 msgr2=0x557f2815e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.783+0000 7f003b400640 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e030 0x557f2815e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.783+0000 7f003b400640 1 -- 172.21.15.28:0/984503967 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f27dce210 con 0x557f28169210 2024-04-26T01:56:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.784+0000 7f0043e00640 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f2815e950 0x557f28168cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.784+0000 7f0048a00640 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e030 0x557f2815e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.784+0000 7f0041600640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f00280a3e40 0x7f00280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.784+0000 7f003b400640 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f28169210 0x557f2816b600 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f004406b6a0 tx=0x7f002c004380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5edd1fd8936d7b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.785+0000 7f0041600640 1 -- 172.21.15.28:0/3914287373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f00340d0be0 con 0x7f00441043c0 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.785+0000 7f0048a00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f00280a3e40 0x7f00280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.786+0000 7f0039200640 1 -- 172.21.15.28:0/984503967 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c035030 con 0x557f28169210 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.786+0000 7f0039200640 1 -- 172.21.15.28:0/984503967 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f002c0046c0 con 0x557f28169210 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.786+0000 7f0039200640 1 -- 172.21.15.28:0/984503967 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c035030 con 0x557f28169210 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.787+0000 7f004d7f0740 1 -- 172.21.15.28:0/984503967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f28169210 msgr2=0x557f2816b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.787+0000 7f004d7f0740 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f28169210 0x557f2816b600 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f004406b6a0 tx=0x7f002c004380 comp rx=0 tx=0).stop 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.787+0000 7f0048a00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f00280a3e40 0x7f00280a6300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f003c00acf0 tx=0x7f003c00a300 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.787+0000 7f004d7f0740 1 -- 172.21.15.28:0/984503967 shutdown_connections 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.787+0000 7f004d7f0740 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f28169210 0x557f2816b600 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.787+0000 7f004d7f0740 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f2815e950 0x557f28168cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.788+0000 7f004d7f0740 1 --2- 172.21.15.28:0/984503967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e030 0x557f2815e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.788+0000 7f004d7f0740 1 -- 172.21.15.28:0/984503967 >> 172.21.15.28:0/984503967 conn(0x557f2815a540 msgr2=0x557f2815a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.789+0000 7f004d7f0740 1 -- 172.21.15.28:0/984503967 shutdown_connections 2024-04-26T01:56:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.789+0000 7f004d7f0740 1 -- 172.21.15.28:0/984503967 wait complete. 2024-04-26T01:56:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.789+0000 7f004d7f0740 1 -- start start 2024-04-26T01:56:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.790+0000 7f004d7f0740 1 Processor -- start 2024-04-26T01:56:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.790+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e030 0x557f28166180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.790+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e950 0x557f281666c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.790+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f28166c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.790+0000 7f004d7f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f2816bde0 con 0x557f2815e950 2024-04-26T01:56:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.790+0000 7f004d7f0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557f28075310 con 0x557f2815e030 2024-04-26T01:56:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.790+0000 7f004d7f0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x557f2816bf60 con 0x557f28169210 2024-04-26T01:56:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.790+0000 7f004d7f0740 10 client.0 _ll_register_callbacks cb 0x557f28043cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.791+0000 7f0043e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e030 0x557f28166180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.791+0000 7f0043e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e030 0x557f28166180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:33812/0 (socket says 172.21.15.28:33812) 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.791+0000 7f0043e00640 1 -- 172.21.15.28:0/1355642304 learned_addr learned my addr 172.21.15.28:0/1355642304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.791+0000 7f003b400640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e950 0x557f281666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.791+0000 7f0048a00640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f28166c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.791+0000 7f0043e00640 1 -- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 msgr2=0x557f28166c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.791+0000 7f0043e00640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f28166c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.792+0000 7f0043e00640 1 -- 172.21.15.28:0/1355642304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e950 msgr2=0x557f281666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.792+0000 7f0043e00640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e950 0x557f281666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.792+0000 7f0043e00640 1 -- 172.21.15.28:0/1355642304 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f27e9a620 con 0x557f2815e030 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.792+0000 7f0048a00640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f28166c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.792+0000 7f003b400640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e950 0x557f281666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.794+0000 7f0043e00640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e030 0x557f28166180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f00340ce390 tx=0x7f003409d340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1f6b959522d762c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.795+0000 7f0031600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:56:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.795+0000 7f0031600640 1 -- 172.21.15.28:0/1355642304 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00340a0f30 con 0x557f2815e030 2024-04-26T01:56:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.795+0000 7f0031600640 1 -- 172.21.15.28:0/1355642304 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f00340bd0b0 con 0x557f2815e030 2024-04-26T01:56:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.795+0000 7f004d7f0740 1 -- 172.21.15.28:0/1355642304 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f002c0046c0 con 0x557f2815e030 2024-04-26T01:56:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.796+0000 7f0031600640 1 -- 172.21.15.28:0/1355642304 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f00340a57f0 con 0x557f2815e030 2024-04-26T01:56:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.796+0000 7f004d7f0740 10 client.26495 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.796+0000 7f004d7f0740 1 -- 172.21.15.28:0/1355642304 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557f28154530 con 0x557f2815e030 2024-04-26T01:56:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.797+0000 7f0031600640 1 -- 172.21.15.28:0/1355642304 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f002c0046c0 con 0x557f2815e030 2024-04-26T01:56:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.799+0000 7f0031600640 1 -- 172.21.15.28:0/1355642304 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f0034024060 con 0x557f2815e030 2024-04-26T01:56:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.799+0000 7f004d7f0740 10 client.26495 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.799+0000 7f004d7f0740 10 client.26495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.799+0000 7f004d7f0740 20 client.26495 populate_metadata read hostname 'smithi028' 2024-04-26T01:56:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.799+0000 7f004d7f0740 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557f28155ea0 0x557f281582f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.799+0000 7f004d7f0740 4 client.26495 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.799+0000 7f004d7f0740 1 -- 172.21.15.28:0/1355642304 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f280cf8b0 con 0x557f28155ea0 2024-04-26T01:56:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.800+0000 7f003b400640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557f28155ea0 0x557f281582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.802+0000 7f0031600640 10 client.26495 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.802+0000 7f003b400640 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557f28155ea0 0x557f281582f0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c788adab9e64533c server_cookie=5a76adaf6130669f in_seq=0 out_seq=0 2024-04-26T01:56:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.804+0000 7f0031600640 1 -- 172.21.15.28:0/1355642304 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f280cf8b0 con 0x557f28155ea0 2024-04-26T01:56:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.804+0000 7f0031600640 10 client.26495 handle_command_reply: tid=0 2024-04-26T01:56:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.804+0000 7f004d7f0740 1 client.26495 shutdown 2024-04-26T01:56:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f004d7f0740 1 -- 172.21.15.28:0/1355642304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e030 msgr2=0x557f28166180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f004d7f0740 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e030 0x557f28166180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f00340ce390 tx=0x7f003409d340 comp rx=0 tx=0).stop 2024-04-26T01:56:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f004d7f0740 1 -- 172.21.15.28:0/1355642304 shutdown_connections 2024-04-26T01:56:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f004d7f0740 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557f28155ea0 0x557f281582f0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f004d7f0740 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f28166c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f004d7f0740 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f2815e950 0x557f281666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f004d7f0740 1 --2- 172.21.15.28:0/1355642304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e030 0x557f28166180 secure :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f00340ce390 tx=0x7f003409d340 comp rx=0 tx=0).stop 2024-04-26T01:56:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f0031600640 0 client.26495 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.811+0000 7f004d7f0740 1 -- 172.21.15.28:0/1355642304 >> 172.21.15.28:0/1355642304 conn(0x557f2815a540 msgr2=0x557f281602c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.813+0000 7f004d7f0740 1 -- 172.21.15.28:0/1355642304 shutdown_connections 2024-04-26T01:56:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.813+0000 7f004d7f0740 1 -- 172.21.15.28:0/1355642304 wait complete. 2024-04-26T01:56:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.813+0000 7f004d7f0740 20 client.26495 trim_cache size 0 max 16384 2024-04-26T01:56:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.856+0000 7f004d7f0740 1 Processor -- start 2024-04-26T01:56:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.856+0000 7f004d7f0740 1 -- start start 2024-04-26T01:56:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.856+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f28174ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.856+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f28169210 0x557f28150e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.856+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f281609f0 0x557f28166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.856+0000 7f004d7f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00441ae100 con 0x557f28169210 2024-04-26T01:56:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.856+0000 7f004d7f0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f00441ab9d0 con 0x557f2815e950 2024-04-26T01:56:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.856+0000 7f004d7f0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f00441a9840 con 0x557f281609f0 2024-04-26T01:56:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0043e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f28174ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0048a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f281609f0 0x557f28166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0043e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f28174ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:33822/0 (socket says 172.21.15.28:33822) 2024-04-26T01:56:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0043e00640 1 -- 172.21.15.28:0/2923810272 learned_addr learned my addr 172.21.15.28:0/2923810272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0048a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f281609f0 0x557f28166a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59346/0 (socket says 172.21.15.28:59346) 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f003b400640 1 --2- 172.21.15.28:0/2923810272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f28169210 0x557f28150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0043e00640 1 -- 172.21.15.28:0/2923810272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f281609f0 msgr2=0x557f28166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0043e00640 1 --2- 172.21.15.28:0/2923810272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f281609f0 0x557f28166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0043e00640 1 -- 172.21.15.28:0/2923810272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f28169210 msgr2=0x557f28150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0043e00640 1 --2- 172.21.15.28:0/2923810272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f28169210 0x557f28150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.857+0000 7f0043e00640 1 -- 172.21.15.28:0/2923810272 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f28161da0 con 0x557f2815e950 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.858+0000 7f0043e00640 1 --2- 172.21.15.28:0/2923810272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f28174ca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f00340049d0 tx=0x7f0034005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8813105796e1d6ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.858+0000 7f0033e00640 1 -- 172.21.15.28:0/2923810272 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00340a0f30 con 0x557f2815e950 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.858+0000 7f0033e00640 1 -- 172.21.15.28:0/2923810272 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0034023da0 con 0x557f2815e950 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.858+0000 7f0033e00640 1 -- 172.21.15.28:0/2923810272 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034099390 con 0x557f2815e950 2024-04-26T01:56:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 -- 172.21.15.28:0/2923810272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 msgr2=0x557f28174ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 --2- 172.21.15.28:0/2923810272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f28174ca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f00340049d0 tx=0x7f0034005f90 comp rx=0 tx=0).stop 2024-04-26T01:56:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 -- 172.21.15.28:0/2923810272 shutdown_connections 2024-04-26T01:56:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 --2- 172.21.15.28:0/2923810272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f281609f0 0x557f28166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 --2- 172.21.15.28:0/2923810272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f28169210 0x557f28150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 --2- 172.21.15.28:0/2923810272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f28174ca0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 -- 172.21.15.28:0/2923810272 >> 172.21.15.28:0/2923810272 conn(0x557f2815a540 msgr2=0x557f281679b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 -- 172.21.15.28:0/2923810272 shutdown_connections 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 -- 172.21.15.28:0/2923810272 wait complete. 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 -- start start 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.859+0000 7f004d7f0740 1 Processor -- start 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.860+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f281711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.860+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f281609f0 0x557f281716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.860+0000 7f004d7f0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f2816e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f004d7f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00441a9e90 con 0x557f281609f0 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f004d7f0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f00441ab300 con 0x557f2815e950 2024-04-26T01:56:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f004d7f0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f00441aa560 con 0x557f28169210 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0043e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f281711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0048a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f2816e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0048a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f2816e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:59350/0 (socket says 172.21.15.28:59350) 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0048a00640 1 -- 172.21.15.28:0/3268518617 learned_addr learned my addr 172.21.15.28:0/3268518617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f003b400640 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f281609f0 0x557f281716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f004d7f0740 10 client.0 _ll_register_callbacks cb 0x7f003409c4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0048a00640 1 -- 172.21.15.28:0/3268518617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 msgr2=0x557f281711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0048a00640 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f281711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0048a00640 1 -- 172.21.15.28:0/3268518617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f281609f0 msgr2=0x557f281716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0048a00640 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f281609f0 0x557f281716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.861+0000 7f0048a00640 1 -- 172.21.15.28:0/3268518617 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f2816eaa0 con 0x557f28169210 2024-04-26T01:56:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.862+0000 7f0048a00640 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f2816e270 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f003c01de00 tx=0x7f003c014040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fcde5c28e189c3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.862+0000 7f0031600640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:56:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.862+0000 7f0031600640 1 -- 172.21.15.28:0/3268518617 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c02d060 con 0x557f28169210 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.862+0000 7f0031600640 1 -- 172.21.15.28:0/3268518617 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f003c028640 con 0x557f28169210 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.862+0000 7f004d7f0740 1 -- 172.21.15.28:0/3268518617 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0034023da0 con 0x557f28169210 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.862+0000 7f0031600640 1 -- 172.21.15.28:0/3268518617 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f003c028880 con 0x557f28169210 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.862+0000 7f004d7f0740 10 client.26506 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.862+0000 7f004d7f0740 1 -- 172.21.15.28:0/3268518617 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557f281e8730 con 0x557f28169210 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.863+0000 7f0031600640 1 -- 172.21.15.28:0/3268518617 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0034023da0 con 0x557f28169210 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.863+0000 7f0031600640 1 -- 172.21.15.28:0/3268518617 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f003c01c400 con 0x557f28169210 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.864+0000 7f004d7f0740 10 client.26506 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.864+0000 7f004d7f0740 10 client.26506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.864+0000 7f004d7f0740 20 client.26506 populate_metadata read hostname 'smithi028' 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.864+0000 7f004d7f0740 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557f281e8f10 0x557f281eb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.864+0000 7f004d7f0740 4 client.26506 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.864+0000 7f004d7f0740 1 -- 172.21.15.28:0/3268518617 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f280c9320 con 0x557f281e8f10 2024-04-26T01:56:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.865+0000 7f0043e00640 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557f281e8f10 0x557f281eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.866+0000 7f0031600640 10 client.26506 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.866+0000 7f0043e00640 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557f281e8f10 0x557f281eb330 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ebfba223fad3fbb server_cookie=a989cf8b070cda46 in_seq=0 out_seq=0 2024-04-26T01:56:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.880+0000 7f0031600640 1 -- 172.21.15.28:0/3268518617 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557f280c9320 con 0x557f281e8f10 2024-04-26T01:56:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.881+0000 7f0031600640 10 client.26506 handle_command_reply: tid=0 2024-04-26T01:56:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.881+0000 7f004d7f0740 1 client.26506 shutdown 2024-04-26T01:56:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.881+0000 7f004d7f0740 1 -- 172.21.15.28:0/3268518617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 msgr2=0x557f2816e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.881+0000 7f004d7f0740 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f2816e270 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f003c01de00 tx=0x7f003c014040 comp rx=0 tx=0).stop 2024-04-26T01:56:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.881+0000 7f004d7f0740 1 -- 172.21.15.28:0/3268518617 shutdown_connections 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f004d7f0740 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x557f281e8f10 0x557f281eb330 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f004d7f0740 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557f28169210 0x557f2816e270 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f004d7f0740 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f281609f0 0x557f281716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f004d7f0740 1 --2- 172.21.15.28:0/3268518617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557f2815e950 0x557f281711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f004d7f0740 1 -- 172.21.15.28:0/3268518617 >> 172.21.15.28:0/3268518617 conn(0x557f2815a540 msgr2=0x557f28160270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f0031600640 0 client.26506 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f004d7f0740 1 -- 172.21.15.28:0/3268518617 shutdown_connections 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f004d7f0740 1 -- 172.21.15.28:0/3268518617 wait complete. 2024-04-26T01:56:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.882+0000 7f004d7f0740 20 client.26506 trim_cache size 0 max 16384 2024-04-26T01:56:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:56:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:56:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:56:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:56:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 -- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f00280a3e40 msgr2=0x7f00280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f00280a3e40 0x7f00280a6300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f003c00acf0 tx=0x7f003c00a300 comp rx=0 tx=0).stop 2024-04-26T01:56:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 -- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00441043c0 msgr2=0x7f00441a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00441043c0 0x7f00441a05a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f003400dab0 tx=0x7f003400deb0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 -- 172.21.15.28:0/3914287373 shutdown_connections 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f00280a3e40 0x7f00280a6300 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f004419a660 0x7f004419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0044109490 0x7f00441a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 --2- 172.21.15.28:0/3914287373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00441043c0 0x7f00441a05a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 -- 172.21.15.28:0/3914287373 >> 172.21.15.28:0/3914287373 conn(0x7f00440ffc60 msgr2=0x7f0044101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 -- 172.21.15.28:0/3914287373 shutdown_connections 2024-04-26T01:56:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:56:49.883+0000 7f004aa00640 1 -- 172.21.15.28:0/3914287373 wait complete. 2024-04-26T01:56:49.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:49 smithi169 ceph-mon[32851]: pgmap v498: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 21 MiB/s wr, 1.58k op/s 2024-04-26T01:56:49.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:49 smithi053 ceph-mon[32593]: pgmap v498: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 21 MiB/s wr, 1.58k op/s 2024-04-26T01:56:50.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:49 smithi028 ceph-mon[28247]: pgmap v498: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 21 MiB/s wr, 1.58k op/s 2024-04-26T01:56:51.224 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 895.0 s, 1370.2 tps, lat 73.133 ms stddev 86.489 2024-04-26T01:56:51.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:51 smithi169 ceph-mon[32851]: pgmap v499: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-26T01:56:51.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:51 smithi053 ceph-mon[32593]: pgmap v499: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-26T01:56:52.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:51 smithi028 ceph-mon[28247]: pgmap v499: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-26T01:56:53.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:53 smithi169 ceph-mon[32851]: pgmap v500: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 26 MiB/s wr, 1.66k op/s 2024-04-26T01:56:53.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:53 smithi053 ceph-mon[32593]: pgmap v500: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 26 MiB/s wr, 1.66k op/s 2024-04-26T01:56:54.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:53 smithi028 ceph-mon[28247]: pgmap v500: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 26 MiB/s wr, 1.66k op/s 2024-04-26T01:56:55.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:55 smithi169 ceph-mon[32851]: pgmap v501: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-26T01:56:55.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:55 smithi053 ceph-mon[32593]: pgmap v501: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-26T01:56:56.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:55 smithi028 ceph-mon[28247]: pgmap v501: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-26T01:56:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:progress: 900.0 s, 1660.4 tps, lat 60.484 ms stddev 69.447 2024-04-26T01:56:56.422 INFO:teuthology.orchestra.run.smithi028.stdout:transaction type: 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:scaling factor: 32 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:query mode: simple 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:number of clients: 100 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:number of threads: 4 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:duration: 900 s 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:number of transactions actually processed: 1403547 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:latency average = 64.097 ms 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:latency stddev = 821.624 ms 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:tps = 1559.152951 (including connections establishing) 2024-04-26T01:56:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:tps = 1559.164757 (excluding connections establishing) 2024-04-26T01:56:56.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-26T01:56:57.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:57 smithi169 ceph-mon[32851]: pgmap v502: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 31 MiB/s wr, 1.88k op/s 2024-04-26T01:56:57.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:57 smithi053 ceph-mon[32593]: pgmap v502: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 31 MiB/s wr, 1.88k op/s 2024-04-26T01:56:58.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:57 smithi028 ceph-mon[28247]: pgmap v502: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 31 MiB/s wr, 1.88k op/s 2024-04-26T01:56:59.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:56:59 smithi169 ceph-mon[32851]: pgmap v503: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 13 MiB/s wr, 614 op/s 2024-04-26T01:56:59.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:56:59 smithi053 ceph-mon[32593]: pgmap v503: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 13 MiB/s wr, 614 op/s 2024-04-26T01:57:00.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:56:59 smithi028 ceph-mon[28247]: pgmap v503: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 13 MiB/s wr, 614 op/s 2024-04-26T01:57:01.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:01 smithi169 ceph-mon[32851]: pgmap v504: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 21 MiB/s wr, 1.24k op/s 2024-04-26T01:57:01.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:01 smithi053 ceph-mon[32593]: pgmap v504: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 21 MiB/s wr, 1.24k op/s 2024-04-26T01:57:02.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:01 smithi028 ceph-mon[28247]: pgmap v504: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 21 MiB/s wr, 1.24k op/s 2024-04-26T01:57:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.090+0000 7f5ad7600640 1 -- 172.21.15.28:0/1338321770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad0074b70 msgr2=0x7f5ad010e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.090+0000 7f5ad7600640 1 --2- 172.21.15.28:0/1338321770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad0074b70 0x7f5ad010e300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009900 tx=0x7f5abc02a490 comp rx=0 tx=0).stop 2024-04-26T01:57:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.091+0000 7f5ad7600640 1 -- 172.21.15.28:0/1338321770 shutdown_connections 2024-04-26T01:57:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.091+0000 7f5ad7600640 1 --2- 172.21.15.28:0/1338321770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad010e840 0x7f5ad0110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.091+0000 7f5ad7600640 1 --2- 172.21.15.28:0/1338321770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad0074b70 0x7f5ad010e300 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.091+0000 7f5ad7600640 1 --2- 172.21.15.28:0/1338321770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ad0074250 0x7f5ad0074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.091+0000 7f5ad7600640 1 -- 172.21.15.28:0/1338321770 >> 172.21.15.28:0/1338321770 conn(0x7f5ad00793e0 msgr2=0x7f5ad00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.091+0000 7f5ad7600640 1 -- 172.21.15.28:0/1338321770 shutdown_connections 2024-04-26T01:57:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.091+0000 7f5ad7600640 1 -- 172.21.15.28:0/1338321770 wait complete. 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.092+0000 7f5ad7600640 1 Processor -- start 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.092+0000 7f5ad7600640 1 -- start start 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.093+0000 7f5ad7600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad0074250 0x7f5ad010cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.093+0000 7f5ad7600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad0074b70 0x7f5ad010d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.093+0000 7f5ad7600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ad010e840 0x7f5ad01aa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.093+0000 7f5ad7600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0102f20 con 0x7f5ad010e840 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.093+0000 7f5ad7600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0102da0 con 0x7f5ad0074250 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.093+0000 7f5ad7600640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad01030a0 con 0x7f5ad0074b70 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.093+0000 7f5ad5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ad010e840 0x7f5ad01aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5ad5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ad010e840 0x7f5ad01aa920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44162/0 (socket says 172.21.15.28:44162) 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5ad5600640 1 -- 172.21.15.28:0/3672769364 learned_addr learned my addr 172.21.15.28:0/3672769364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad0074250 0x7f5ad010cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad0074b70 0x7f5ad010d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5ad5600640 1 -- 172.21.15.28:0/3672769364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad0074b70 msgr2=0x7f5ad010d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5ad5600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad0074b70 0x7f5ad010d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5ad5600640 1 -- 172.21.15.28:0/3672769364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad0074250 msgr2=0x7f5ad010cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5ad5600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad0074250 0x7f5ad010cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.094+0000 7f5ad5600640 1 -- 172.21.15.28:0/3672769364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad0113490 con 0x7f5ad010e840 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.095+0000 7f5ad5600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ad010e840 0x7f5ad01aa920 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800ea60 tx=0x7f5ac800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.095+0000 7f5acd600640 1 -- 172.21.15.28:0/3672769364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800ce40 con 0x7f5ad010e840 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.095+0000 7f5ad7600640 1 -- 172.21.15.28:0/3672769364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad0113780 con 0x7f5ad010e840 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.095+0000 7f5acd600640 1 -- 172.21.15.28:0/3672769364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac8005020 con 0x7f5ad010e840 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.095+0000 7f5ad7600640 1 -- 172.21.15.28:0/3672769364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad0113ca0 con 0x7f5ad010e840 2024-04-26T01:57:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.096+0000 7f5acd600640 1 -- 172.21.15.28:0/3672769364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8010690 con 0x7f5ad010e840 2024-04-26T01:57:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.100+0000 7f5ada46e740 1 Processor -- start 2024-04-26T01:57:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.101+0000 7f5acd600640 1 -- 172.21.15.28:0/3672769364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5ac8004a40 con 0x7f5ad010e840 2024-04-26T01:57:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.101+0000 7f5ada46e740 1 -- start start 2024-04-26T01:57:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.102+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777c820 0x56208777cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.102+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777d140 0x5620877874c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.102+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 0x562087789df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.102+0000 7f5ada46e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620876e2900 con 0x56208777d140 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.103+0000 7f5ada46e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562087688ef0 con 0x562087787a00 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.103+0000 7f5ada46e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x5620876de8c0 con 0x56208777c820 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.103+0000 7f5acd600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5ab80a3e40 0x7f5ab80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.103+0000 7f5acfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 0x562087789df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.103+0000 7f5acfe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777c820 0x56208777cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.103+0000 7f5acfe00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777c820 0x56208777cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37258/0 (socket says 172.21.15.28:37258) 2024-04-26T01:57:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.103+0000 7f5acfe00640 1 -- 172.21.15.28:0/3304309146 learned_addr learned my addr 172.21.15.28:0/3304309146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.103+0000 7f5acd600640 1 -- 172.21.15.28:0/3672769364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f5ac80d0cf0 con 0x7f5ad010e840 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.104+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/3304309146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777d140 0x5620877874c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.104+0000 7f5ad5600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5ab80a3e40 0x7f5ab80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.104+0000 7f5acfe00640 1 -- 172.21.15.28:0/3304309146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 msgr2=0x562087789df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.104+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3304309146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 0x562087789df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.104+0000 7f5acfe00640 1 -- 172.21.15.28:0/3304309146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777d140 msgr2=0x5620877874c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.104+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3304309146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777d140 0x5620877874c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.104+0000 7f5acfe00640 1 -- 172.21.15.28:0/3304309146 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620873f5210 con 0x56208777c820 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.104+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3304309146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777c820 0x56208777cc00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5abc005ab0 tx=0x7f5abc031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=377872472b4c5b5e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.105+0000 7f5ac1c00640 1 -- 172.21.15.28:0/3304309146 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc035040 con 0x56208777c820 2024-04-26T01:57:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.105+0000 7f5ad5600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5ab80a3e40 0x7f5ab80a6300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800c760 tx=0x7f5ac80096c0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.105+0000 7f5ac1c00640 1 -- 172.21.15.28:0/3304309146 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5abc003ca0 con 0x56208777c820 2024-04-26T01:57:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.105+0000 7f5ac1c00640 1 -- 172.21.15.28:0/3304309146 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc031b40 con 0x56208777c820 2024-04-26T01:57:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.106+0000 7f5ada46e740 1 -- 172.21.15.28:0/3304309146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777c820 msgr2=0x56208777cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.106+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3304309146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777c820 0x56208777cc00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5abc005ab0 tx=0x7f5abc031400 comp rx=0 tx=0).stop 2024-04-26T01:57:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.107+0000 7f5ada46e740 1 -- 172.21.15.28:0/3304309146 shutdown_connections 2024-04-26T01:57:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.107+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3304309146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 0x562087789df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.107+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3304309146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777d140 0x5620877874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.107+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3304309146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777c820 0x56208777cc00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.107+0000 7f5ada46e740 1 -- 172.21.15.28:0/3304309146 >> 172.21.15.28:0/3304309146 conn(0x562087778d30 msgr2=0x562087779110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.107+0000 7f5ada46e740 1 -- 172.21.15.28:0/3304309146 shutdown_connections 2024-04-26T01:57:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.107+0000 7f5ada46e740 1 -- 172.21.15.28:0/3304309146 wait complete. 2024-04-26T01:57:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.108+0000 7f5ada46e740 1 -- start start 2024-04-26T01:57:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.108+0000 7f5ada46e740 1 Processor -- start 2024-04-26T01:57:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.109+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 0x562087784920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.109+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x562087784e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.109+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 0x5620877853a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.109+0000 7f5ada46e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56208778a5d0 con 0x56208777c820 2024-04-26T01:57:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.109+0000 7f5ada46e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56208769c310 con 0x562087787a00 2024-04-26T01:57:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.109+0000 7f5ada46e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x56208778a750 con 0x56208777d140 2024-04-26T01:57:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.109+0000 7f5ada46e740 10 client.0 _ll_register_callbacks cb 0x56208766acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.110+0000 7f5acfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 0x562087784920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.110+0000 7f5ad4c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x562087784e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.110+0000 7f5ad4c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x562087784e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37270/0 (socket says 172.21.15.28:37270) 2024-04-26T01:57:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.111+0000 7f5ad4c00640 1 -- 172.21.15.28:0/3877582237 learned_addr learned my addr 172.21.15.28:0/3877582237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.111+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 0x5620877853a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.111+0000 7f5ad4c00640 1 -- 172.21.15.28:0/3877582237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 msgr2=0x5620877853a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.111+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 0x5620877853a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.111+0000 7f5ad4c00640 1 -- 172.21.15.28:0/3877582237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 msgr2=0x562087784920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.111+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 0x562087784920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.111+0000 7f5ad4c00640 1 -- 172.21.15.28:0/3877582237 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620874c1620 con 0x56208777d140 2024-04-26T01:57:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.112+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x562087784e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400ce00 tx=0x7f5ac40039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91d80c72b6d491bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.112+0000 7f5ab6000640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:57:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.113+0000 7f5ab6000640 1 -- 172.21.15.28:0/3877582237 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4011910 con 0x56208777d140 2024-04-26T01:57:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.113+0000 7f5ab6000640 1 -- 172.21.15.28:0/3877582237 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f5ac4011ab0 con 0x56208777d140 2024-04-26T01:57:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.113+0000 7f5ada46e740 1 -- 172.21.15.28:0/3877582237 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5abc003ca0 con 0x56208777d140 2024-04-26T01:57:03.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.117+0000 7f5ab6000640 1 -- 172.21.15.28:0/3877582237 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5ac4011cb0 con 0x56208777d140 2024-04-26T01:57:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.117+0000 7f5ada46e740 10 client.26518 fetch_fsmap learned FSMap version 47 2024-04-26T01:57:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.117+0000 7f5ada46e740 1 -- 172.21.15.28:0/3877582237 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562087796900 con 0x56208777d140 2024-04-26T01:57:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.118+0000 7f5ab6000640 1 -- 172.21.15.28:0/3877582237 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5abc003ca0 con 0x56208777d140 2024-04-26T01:57:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.118+0000 7f5ab6000640 1 -- 172.21.15.28:0/3877582237 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f5ac4018070 con 0x56208777d140 2024-04-26T01:57:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.119+0000 7f5ada46e740 10 client.26518 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:57:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.119+0000 7f5ada46e740 10 client.26518 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.119+0000 7f5ada46e740 20 client.26518 populate_metadata read hostname 'smithi028' 2024-04-26T01:57:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.119+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562087797e30 0x56208779a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.119+0000 7f5ada46e740 4 client.26518 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.119+0000 7f5ada46e740 1 -- 172.21.15.28:0/3877582237 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620876f68b0 con 0x562087797e30 2024-04-26T01:57:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.120+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562087797e30 0x56208779a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.122+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562087797e30 0x56208779a260 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd9f0440af274d7c server_cookie=8c98a7d64205619e in_seq=0 out_seq=0 2024-04-26T01:57:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.122+0000 7f5ab6000640 10 client.26518 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:57:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.123+0000 7f5ab6000640 1 -- 172.21.15.28:0/3877582237 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620876f68b0 con 0x562087797e30 2024-04-26T01:57:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.123+0000 7f5ab6000640 10 client.26518 handle_command_reply: tid=0 2024-04-26T01:57:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.123+0000 7f5ada46e740 1 client.26518 shutdown 2024-04-26T01:57:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.124+0000 7f5ada46e740 1 -- 172.21.15.28:0/3877582237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 msgr2=0x562087784e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.124+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x562087784e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400ce00 tx=0x7f5ac40039c0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.124+0000 7f5ada46e740 1 -- 172.21.15.28:0/3877582237 shutdown_connections 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.124+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x562087797e30 0x56208779a260 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.124+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562087787a00 0x5620877853a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.124+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x562087784e60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.124+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3877582237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 0x562087784920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.124+0000 7f5ada46e740 1 -- 172.21.15.28:0/3877582237 >> 172.21.15.28:0/3877582237 conn(0x562087778d30 msgr2=0x56208777eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.125+0000 7f5ab6000640 0 client.26518 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.125+0000 7f5ada46e740 1 -- 172.21.15.28:0/3877582237 shutdown_connections 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.125+0000 7f5ada46e740 1 -- 172.21.15.28:0/3877582237 wait complete. 2024-04-26T01:57:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.125+0000 7f5ada46e740 20 client.26518 trim_cache size 0 max 16384 2024-04-26T01:57:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.173+0000 7f5ada46e740 1 Processor -- start 2024-04-26T01:57:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.173+0000 7f5ada46e740 1 -- start start 2024-04-26T01:57:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.173+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208777c820 0x5620877934f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.173+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x56208779fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.173+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562087787a00 0x5620877a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.173+0000 7f5ada46e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab00110d0 con 0x562087787a00 2024-04-26T01:57:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.173+0000 7f5ada46e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab000e9a0 con 0x56208777c820 2024-04-26T01:57:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.173+0000 7f5ada46e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab000c810 con 0x56208777d140 2024-04-26T01:57:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.174+0000 7f5ad4c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x56208779fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.174+0000 7f5ad4c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x56208779fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37284/0 (socket says 172.21.15.28:37284) 2024-04-26T01:57:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.174+0000 7f5ad4c00640 1 -- 172.21.15.28:0/1033541185 learned_addr learned my addr 172.21.15.28:0/1033541185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.174+0000 7f5acfe00640 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208777c820 0x5620877934f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.174+0000 7f5acfe00640 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562087787a00 0x5620877a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.175+0000 7f5ad4c00640 1 -- 172.21.15.28:0/1033541185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208777c820 msgr2=0x5620877934f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.175+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208777c820 0x5620877934f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.175+0000 7f5ad4c00640 1 -- 172.21.15.28:0/1033541185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562087787a00 msgr2=0x5620877a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.175+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562087787a00 0x5620877a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.175+0000 7f5ad4c00640 1 -- 172.21.15.28:0/1033541185 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562087774690 con 0x56208777d140 2024-04-26T01:57:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.175+0000 7f5acfe00640 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562087787a00 0x5620877a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.176+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x56208779fb70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5ac401a730 tx=0x7f5ac40041f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e194a50e457b169 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.178+0000 7f5ac1200640 1 -- 172.21.15.28:0/1033541185 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4011710 con 0x56208777d140 2024-04-26T01:57:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.178+0000 7f5ac1200640 1 -- 172.21.15.28:0/1033541185 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac401ac60 con 0x56208777d140 2024-04-26T01:57:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.178+0000 7f5ac1200640 1 -- 172.21.15.28:0/1033541185 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40246d0 con 0x56208777d140 2024-04-26T01:57:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 -- 172.21.15.28:0/1033541185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 msgr2=0x56208779fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x56208779fb70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5ac401a730 tx=0x7f5ac40041f0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 -- 172.21.15.28:0/1033541185 shutdown_connections 2024-04-26T01:57:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562087787a00 0x5620877a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56208777d140 0x56208779fb70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 --2- 172.21.15.28:0/1033541185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208777c820 0x5620877934f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 -- 172.21.15.28:0/1033541185 >> 172.21.15.28:0/1033541185 conn(0x562087778d30 msgr2=0x562087779110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 -- 172.21.15.28:0/1033541185 shutdown_connections 2024-04-26T01:57:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.179+0000 7f5ada46e740 1 -- 172.21.15.28:0/1033541185 wait complete. 2024-04-26T01:57:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.180+0000 7f5ada46e740 1 -- start start 2024-04-26T01:57:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.180+0000 7f5ada46e740 1 Processor -- start 2024-04-26T01:57:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.181+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 0x56208778d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.181+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562087787a00 0x56208778dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.182+0000 7f5ada46e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208778dff0 0x5620877fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.182+0000 7f5ada46e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab000ce60 con 0x56208777c820 2024-04-26T01:57:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.182+0000 7f5ada46e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab000e2d0 con 0x56208778dff0 2024-04-26T01:57:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.182+0000 7f5ada46e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab000d530 con 0x562087787a00 2024-04-26T01:57:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.182+0000 7f5ada46e740 10 client.0 _ll_register_callbacks cb 0x7f5ac400ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.182+0000 7f5ad4c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562087787a00 0x56208778dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.182+0000 7f5ad4c00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562087787a00 0x56208778dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37288/0 (socket says 172.21.15.28:37288) 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.182+0000 7f5ad4c00640 1 -- 172.21.15.28:0/3827036844 learned_addr learned my addr 172.21.15.28:0/3827036844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.183+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208778dff0 0x5620877fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.183+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 0x56208778d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.184+0000 7f5ad4c00640 1 -- 172.21.15.28:0/3827036844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208778dff0 msgr2=0x5620877fd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.184+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208778dff0 0x5620877fd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.184+0000 7f5ad4c00640 1 -- 172.21.15.28:0/3827036844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 msgr2=0x56208778d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.184+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 0x56208778d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.184+0000 7f5ad4c00640 1 -- 172.21.15.28:0/3827036844 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620877fd9b0 con 0x562087787a00 2024-04-26T01:57:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.186+0000 7f5ad4c00640 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562087787a00 0x56208778dab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4004280 tx=0x7f5ac400c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7236bb6005b60726 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.186+0000 7f5ab6000640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:57:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.187+0000 7f5ab6000640 1 -- 172.21.15.28:0/3827036844 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac401a970 con 0x562087787a00 2024-04-26T01:57:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.188+0000 7f5ab6000640 1 -- 172.21.15.28:0/3827036844 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8513+0+0 (secure 0 0 0) 0x7f5ac4005040 con 0x562087787a00 2024-04-26T01:57:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.188+0000 7f5ada46e740 1 -- 172.21.15.28:0/3827036844 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ac401ac60 con 0x562087787a00 2024-04-26T01:57:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.189+0000 7f5ab6000640 1 -- 172.21.15.28:0/3827036844 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5ac4005240 con 0x562087787a00 2024-04-26T01:57:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.189+0000 7f5ada46e740 10 client.26524 fetch_fsmap learned FSMap version 47 2024-04-26T01:57:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.189+0000 7f5ada46e740 1 -- 172.21.15.28:0/3827036844 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5620877feca0 con 0x562087787a00 2024-04-26T01:57:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.189+0000 7f5ab6000640 1 -- 172.21.15.28:0/3827036844 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac401ac60 con 0x562087787a00 2024-04-26T01:57:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.190+0000 7f5ab6000640 1 -- 172.21.15.28:0/3827036844 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f5ac4018040 con 0x562087787a00 2024-04-26T01:57:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.191+0000 7f5ada46e740 10 client.26524 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:57:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.191+0000 7f5ada46e740 10 client.26524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.191+0000 7f5ada46e740 20 client.26524 populate_metadata read hostname 'smithi028' 2024-04-26T01:57:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.191+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620877ff340 0x562087801760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.191+0000 7f5ada46e740 4 client.26524 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:03.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.191+0000 7f5ada46e740 1 -- 172.21.15.28:0/3827036844 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5620876f0320 con 0x5620877ff340 2024-04-26T01:57:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.192+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620877ff340 0x562087801760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.193+0000 7f5acfe00640 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620877ff340 0x562087801760 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8085ccef9eb8acc server_cookie=f4e3275586cdec8b in_seq=0 out_seq=0 2024-04-26T01:57:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:03.193+0000 7f5ab6000640 10 client.26524 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:57:03.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:03 smithi169 ceph-mon[32851]: pgmap v505: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-04-26T01:57:03.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:03 smithi053 ceph-mon[32593]: pgmap v505: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-04-26T01:57:04.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:03 smithi028 ceph-mon[28247]: pgmap v505: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-04-26T01:57:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.698+0000 7f5ab6000640 1 -- 172.21.15.28:0/3827036844 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5620876f0320 con 0x5620877ff340 2024-04-26T01:57:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.698+0000 7f5ab6000640 10 client.26524 handle_command_reply: tid=0 2024-04-26T01:57:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.698+0000 7f5ada46e740 1 client.26524 shutdown 2024-04-26T01:57:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.699+0000 7f5ada46e740 1 -- 172.21.15.28:0/3827036844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562087787a00 msgr2=0x56208778dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.699+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562087787a00 0x56208778dab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4004280 tx=0x7f5ac400c500 comp rx=0 tx=0).stop 2024-04-26T01:57:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.699+0000 7f5ada46e740 1 -- 172.21.15.28:0/3827036844 shutdown_connections 2024-04-26T01:57:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.699+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x5620877ff340 0x562087801760 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.699+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56208778dff0 0x5620877fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.700+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562087787a00 0x56208778dab0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.700+0000 7f5ada46e740 1 --2- 172.21.15.28:0/3827036844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56208777c820 0x56208778d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.700+0000 7f5ada46e740 1 -- 172.21.15.28:0/3827036844 >> 172.21.15.28:0/3827036844 conn(0x562087778d30 msgr2=0x562087777600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.700+0000 7f5ab6000640 0 client.26524 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:57:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.700+0000 7f5ada46e740 1 -- 172.21.15.28:0/3827036844 shutdown_connections 2024-04-26T01:57:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.700+0000 7f5ada46e740 1 -- 172.21.15.28:0/3827036844 wait complete. 2024-04-26T01:57:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.701+0000 7f5ada46e740 20 client.26524 trim_cache size 0 max 16384 2024-04-26T01:57:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-26T01:57:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-26T01:57:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-26T01:57:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-26T01:57:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 -- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5ab80a3e40 msgr2=0x7f5ab80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5ab80a3e40 0x7f5ab80a6300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800c760 tx=0x7f5ac80096c0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 -- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ad010e840 msgr2=0x7f5ad01aa920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ad010e840 0x7f5ad01aa920 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800ea60 tx=0x7f5ac800ef20 comp rx=0 tx=0).stop 2024-04-26T01:57:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 -- 172.21.15.28:0/3672769364 shutdown_connections 2024-04-26T01:57:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5ab80a3e40 0x7f5ab80a6300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ad010e840 0x7f5ad01aa920 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad0074b70 0x7f5ad010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 --2- 172.21.15.28:0/3672769364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad0074250 0x7f5ad010cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 -- 172.21.15.28:0/3672769364 >> 172.21.15.28:0/3672769364 conn(0x7f5ad00793e0 msgr2=0x7f5ad0100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.702+0000 7f5ad7600640 1 -- 172.21.15.28:0/3672769364 shutdown_connections 2024-04-26T01:57:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-26T01:57:04.703+0000 7f5ad7600640 1 -- 172.21.15.28:0/3672769364 wait complete. 2024-04-26T01:57:05.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:05 smithi169 ceph-mon[32851]: pgmap v506: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-04-26T01:57:05.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:05 smithi053 ceph-mon[32593]: pgmap v506: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-04-26T01:57:06.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:05 smithi028 ceph-mon[28247]: pgmap v506: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-04-26T01:57:06.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-26T01:57:06.467 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/: 2024-04-26T01:57:06.467 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.467 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:57:06.467 INFO:teuthology.orchestra.run.smithi028.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 .. 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 data 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data: 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:total 36 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 . 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 .. 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 01:41 base 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Apr 26 01:41 current_logfiles 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 26 01:41 global 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 log 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_commit_ts 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_dynshmem 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 26 01:41 pg_hba.conf 2024-04-26T01:57:06.468 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 26 01:41 pg_ident.conf 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:57 pg_logical 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 pg_multixact 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_notify 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_replslot 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_serial 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_snapshots 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:57 pg_stat 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:57 pg_stat_tmp 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:56 pg_subtrans 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_tblspc 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pg_twophase 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 PG_VERSION 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 26 01:57 pg_wal 2024-04-26T01:57:06.469 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:56 pg_xact 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 26 01:41 postgresql.auto.conf 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 26 01:41 postgresql.conf 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Apr 26 01:41 postmaster.opts 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base: 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 01:41 . 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 1 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 13493 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 26 01:48 13494 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 pgsql_tmp 2024-04-26T01:57:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:total 7777 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 . 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 01:41 .. 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 112 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 113 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 1247 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1247_fsm 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1247_vm 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 01:41 1249 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1249_fsm 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1249_vm 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 01:41 1255 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1255_fsm 2024-04-26T01:57:06.471 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1255_vm 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 01:41 1259 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1259_fsm 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1259_vm 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 01:41 13344 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13344_fsm 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13344_vm 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13346 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13348 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13349_fsm 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349_vm 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13351 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13353 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354 2024-04-26T01:57:06.472 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13354_fsm 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354_vm 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13356 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13358 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13359_fsm 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359_vm 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13361 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13363 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1417 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1418 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 174 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 175 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2187 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2224 2024-04-26T01:57:06.473 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2328 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2336 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2337 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2579 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2600 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2600_fsm 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2600_vm 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2601_fsm 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601_vm 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2602 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2602_fsm 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2602_vm 2024-04-26T01:57:06.474 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2603 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2603_fsm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2603_vm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2604 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2605 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2605_fsm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2605_vm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2606_fsm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606_vm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2607 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2607_fsm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2607_vm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 01:41 2608 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2608_fsm 2024-04-26T01:57:06.475 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2608_vm 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 01:41 2609 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2609_fsm 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2609_vm 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2610 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2610_fsm 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2610_vm 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2611 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2612_fsm 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612_vm 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2613 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2615_fsm 2024-04-26T01:57:06.476 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615_vm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2616 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2616_fsm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2616_vm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 01:41 2617 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2617_fsm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2617_vm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 01:41 2618 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2618_fsm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2618_vm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 01:41 2619 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2619_fsm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2619_vm 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2620 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2650 2024-04-26T01:57:06.477 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2651 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2652 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2653 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2654 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2655 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2656 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2657 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 01:41 2658 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 2659 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2660 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2661 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2662 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2663 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2664 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2665 2024-04-26T01:57:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2666 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2667 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2668 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2669 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2670 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 01:41 2673 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 01:41 2674 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 01:41 2675 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2678 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2679 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2680 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2681 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2682 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2683 2024-04-26T01:57:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2684 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2685 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2686 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2687 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2688 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2689 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 01:41 2690 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 01:41 2691 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2692 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2693 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2696 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2699 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2701 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2702 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2703 2024-04-26T01:57:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2704 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2753 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2753_fsm 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2753_vm 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2754 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2755 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2756 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2757 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2830 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2831 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2832 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2833 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2834 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2835 2024-04-26T01:57:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2836 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2837 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 01:41 2838 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2838_fsm 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2838_vm 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2839 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840_fsm 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2840_vm 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2841 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2995 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2996 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3079_fsm 2024-04-26T01:57:06.482 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079_vm 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3080 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3081 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3085 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3118 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3119 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 01:41 3164 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3256 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3257 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3258 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3350 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3351 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3379 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3380 2024-04-26T01:57:06.483 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3381 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3394 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3394_fsm 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3394_vm 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3395 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3429 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3430 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3431 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3433 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3439 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3440 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3455 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 01:41 3456 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3456_fsm 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3456_vm 2024-04-26T01:57:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3466 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3467 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3468 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3501 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3502 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3503 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3534 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3541_fsm 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541_vm 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3542 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3574 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3575 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3576 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3596 2024-04-26T01:57:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3597 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3598 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3599 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3600_fsm 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600_vm 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3601_fsm 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601_vm 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3602_fsm 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602_vm 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603_fsm 2024-04-26T01:57:06.486 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3603_vm 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3604 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3605 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3606 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3607 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3608 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 3609 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3712 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3764_fsm 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764_vm 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3766 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3767 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3997 2024-04-26T01:57:06.487 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4143 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4144 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4145 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4146 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4147 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4148 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4149 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4150 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4151 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4152 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4153 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4154 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4155 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4156 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4157 2024-04-26T01:57:06.488 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4158 2024-04-26T01:57:06.489 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4159 2024-04-26T01:57:06.489 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4160 2024-04-26T01:57:06.489 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4161 2024-04-26T01:57:06.489 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4162 2024-04-26T01:57:06.510 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4163 2024-04-26T01:57:06.510 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4164 2024-04-26T01:57:06.510 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4165 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4166 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4167 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4168 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4169 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4170 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4171 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4172 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4173 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4174 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 5002 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 548 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 549 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6102 2024-04-26T01:57:06.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6104 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6106 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6110 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6111 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6112 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6113 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6117 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 826 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 827 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 828 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 01:41 pg_filenode.map 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 PG_VERSION 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-26T01:57:06.512 INFO:teuthology.orchestra.run.smithi028.stdout:total 7777 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 01:41 . 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 01:41 .. 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 112 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 113 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 1247 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1247_fsm 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1247_vm 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 01:41 1249 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1249_fsm 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1249_vm 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 01:41 1255 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1255_fsm 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1255_vm 2024-04-26T01:57:06.513 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 01:41 1259 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1259_fsm 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1259_vm 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 01:41 13344 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13344_fsm 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13344_vm 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13346 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13348 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13349_fsm 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349_vm 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13351 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13353 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354 2024-04-26T01:57:06.514 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13354_fsm 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354_vm 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13356 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13358 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13359_fsm 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359_vm 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13361 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13363 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1417 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1418 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 174 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 175 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2187 2024-04-26T01:57:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2224 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2328 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2336 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2337 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2579 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2600 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2600_fsm 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2600_vm 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2601_fsm 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601_vm 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2602 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2602_fsm 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2602_vm 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2603 2024-04-26T01:57:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2603_fsm 2024-04-26T01:57:06.517 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2603_vm 2024-04-26T01:57:06.517 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2604 2024-04-26T01:57:06.517 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2605 2024-04-26T01:57:06.517 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2605_fsm 2024-04-26T01:57:06.517 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2605_vm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2606_fsm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606_vm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2607 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2607_fsm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2607_vm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 01:41 2608 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2608_fsm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2608_vm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 01:41 2609 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2609_fsm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2609_vm 2024-04-26T01:57:06.518 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2610 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2610_fsm 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2610_vm 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2611 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2612_fsm 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612_vm 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2613 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2615_fsm 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615_vm 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2616 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2616_fsm 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2616_vm 2024-04-26T01:57:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 01:41 2617 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2617_fsm 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2617_vm 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 01:41 2618 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2618_fsm 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2618_vm 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 01:41 2619 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2619_fsm 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2619_vm 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2620 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2650 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2651 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2652 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2653 2024-04-26T01:57:06.520 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2654 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2655 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2656 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2657 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 01:41 2658 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:41 2659 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2660 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2661 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2662 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2663 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2664 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2665 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2666 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2667 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2668 2024-04-26T01:57:06.521 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2669 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2670 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 01:41 2673 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 01:41 2674 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 01:41 2675 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2678 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2679 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2680 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2681 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2682 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2683 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2684 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2685 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2686 2024-04-26T01:57:06.522 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2687 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2688 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2689 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 01:41 2690 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 01:41 2691 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2692 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2693 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2696 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2699 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2701 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2702 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2703 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2704 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2753 2024-04-26T01:57:06.523 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2753_fsm 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2753_vm 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2754 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2755 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2756 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2757 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2830 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2831 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2832 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2833 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2834 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2835 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2836 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2837 2024-04-26T01:57:06.524 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 01:41 2838 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2838_fsm 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2838_vm 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2839 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840_fsm 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2840_vm 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2841 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2995 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2996 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3079_fsm 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079_vm 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3080 2024-04-26T01:57:06.525 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3081 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3085 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3118 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3119 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 01:41 3164 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3256 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3257 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3258 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3350 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3351 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3379 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3380 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3381 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3394 2024-04-26T01:57:06.526 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3394_fsm 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3394_vm 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3395 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3429 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3430 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3431 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3433 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3439 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3440 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3455 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 01:41 3456 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3456_fsm 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3456_vm 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3466 2024-04-26T01:57:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3467 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3468 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3501 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3502 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3503 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3534 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3541_fsm 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541_vm 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3542 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3574 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3575 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3576 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3596 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3597 2024-04-26T01:57:06.528 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3598 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3599 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3600_fsm 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600_vm 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3601_fsm 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601_vm 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3602_fsm 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602_vm 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603_fsm 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3603_vm 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3604 2024-04-26T01:57:06.529 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3605 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3606 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3607 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3608 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 3609 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3712 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3764_fsm 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764_vm 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3766 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3767 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3997 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4143 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4144 2024-04-26T01:57:06.530 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4145 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4146 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4147 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4148 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4149 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4150 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4151 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4152 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4153 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4154 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4155 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4156 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4157 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4158 2024-04-26T01:57:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4159 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4160 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4161 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4162 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4163 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4164 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4165 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4166 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4167 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4168 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4169 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4170 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4171 2024-04-26T01:57:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4172 2024-04-26T01:57:06.589 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4173 2024-04-26T01:57:06.589 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4174 2024-04-26T01:57:06.589 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 5002 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 548 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 549 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6102 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6104 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6106 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6110 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6111 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6112 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6113 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6117 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 826 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 827 2024-04-26T01:57:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 828 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 01:41 pg_filenode.map 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 PG_VERSION 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:total 584022 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 26 01:48 . 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 01:41 .. 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 112 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 113 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:42 1247 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1247_fsm 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1247_vm 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 01:42 1249 2024-04-26T01:57:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1249_fsm 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1249_vm 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 01:41 1255 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1255_fsm 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1255_vm 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 01:56 1259 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1259_fsm 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1259_vm 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 01:41 13344 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13344_fsm 2024-04-26T01:57:06.592 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13344_vm 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13346 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13348 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13349_fsm 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13349_vm 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13351 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13353 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13354_fsm 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13354_vm 2024-04-26T01:57:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13356 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13358 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 13359_fsm 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13359_vm 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 13361 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 13363 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1417 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 1418 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 442130432 Apr 26 01:56 16396 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 26 01:56 16396_fsm 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:50 16396_vm 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 925696 Apr 26 01:56 16397 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:56 16397_fsm 2024-04-26T01:57:06.594 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:56 16397_vm 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1081344 Apr 26 01:56 16399 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:56 16399_fsm 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:56 16399_vm 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:56 16400 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:56 16402 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Apr 26 01:56 16404 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 73523200 Apr 26 01:56 16406 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:56 16406_fsm 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:56 16406_vm 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 174 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 175 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2187 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2224 2024-04-26T01:57:06.595 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2328 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2336 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2337 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:42 2579 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2600 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2600_fsm 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2600_vm 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2601_fsm 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2601_vm 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2602 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2602_fsm 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2602_vm 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2603 2024-04-26T01:57:06.596 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2603_fsm 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2603_vm 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2604 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2605 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2605_fsm 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2605_vm 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:42 2606 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2606_fsm 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2606_vm 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2607 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2607_fsm 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2607_vm 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 01:42 2608 2024-04-26T01:57:06.597 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2608_fsm 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2608_vm 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 01:41 2609 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2609_fsm 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2609_vm 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:42 2610 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2610_fsm 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2610_vm 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2611 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2612_fsm 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2612_vm 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2613 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615 2024-04-26T01:57:06.598 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2615_fsm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2615_vm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2616 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2616_fsm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2616_vm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 01:41 2617 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2617_fsm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2617_vm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 01:41 2618 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2618_fsm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2618_vm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 172032 Apr 26 01:56 2619 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:45 2619_fsm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2619_vm 2024-04-26T01:57:06.599 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2620 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2650 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2651 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2652 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2653 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2654 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2655 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2656 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2657 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 01:42 2658 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 01:42 2659 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2660 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2661 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:42 2662 2024-04-26T01:57:06.600 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:42 2663 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:42 2664 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:42 2665 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:42 2666 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:42 2667 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2668 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2669 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2670 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 01:42 2673 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 01:42 2674 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 01:41 2675 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:42 2678 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:42 2679 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2680 2024-04-26T01:57:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2681 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2682 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2683 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2684 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2685 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2686 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2687 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 2688 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 01:41 2689 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 01:41 2690 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 01:41 2691 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2692 2024-04-26T01:57:06.602 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2693 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:47 2696 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2699 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2701 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2702 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:42 2703 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:42 2704 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2753 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2753_fsm 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2753_vm 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2754 2024-04-26T01:57:06.603 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2755 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2756 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 2757 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2830 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2831 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2832 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2833 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2834 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2835 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2836 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2837 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 01:41 2838 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2838_fsm 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2838_vm 2024-04-26T01:57:06.604 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2839 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2840_fsm 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2840_vm 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2841 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2995 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2996 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3079_fsm 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3079_vm 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3080 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3081 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:41 3085 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3118 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3119 2024-04-26T01:57:06.605 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 01:41 3164 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3256 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3257 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3258 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3350 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3351 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3379 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3380 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3381 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3394 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3394_fsm 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3394_vm 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3395 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3429 2024-04-26T01:57:06.606 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3430 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3431 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3433 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3439 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3440 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 01:42 3455 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 01:41 3456 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3456_fsm 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3456_vm 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3466 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3467 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3468 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3501 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3502 2024-04-26T01:57:06.607 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3503 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3534 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3541_fsm 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3541_vm 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3542 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3574 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3575 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3576 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3596 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3597 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3598 2024-04-26T01:57:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3599 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3600_fsm 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3600_vm 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3601_fsm 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3601_vm 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3602_fsm 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3602_vm 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3603_fsm 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3603_vm 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3604 2024-04-26T01:57:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3605 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3606 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3607 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3608 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 01:41 3609 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3712 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 3764_fsm 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3764_vm 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3766 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 3767 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3997 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4143 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4144 2024-04-26T01:57:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4145 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4146 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4147 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4148 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4149 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4150 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4151 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4152 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4153 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4154 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4155 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4156 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4157 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4158 2024-04-26T01:57:06.611 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4159 2024-04-26T01:57:06.612 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4160 2024-04-26T01:57:06.612 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4161 2024-04-26T01:57:06.612 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4162 2024-04-26T01:57:06.612 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4163 2024-04-26T01:57:06.612 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4164 2024-04-26T01:57:06.612 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4165 2024-04-26T01:57:06.612 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4166 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4167 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4168 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4169 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4170 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4171 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4172 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4173 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4174 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 5002 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 548 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 549 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6102 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6104 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6106 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6110 2024-04-26T01:57:06.613 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6111 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6112 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6113 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6117 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 826 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 827 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 828 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 01:41 pg_filenode.map 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Apr 26 01:41 pg_internal.init 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:41 PG_VERSION 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 01:41 .. 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/global: 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:total 560 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 26 01:41 . 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1213 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1213_fsm 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1213_vm 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1214 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1214_fsm 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1214_vm 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 1232 2024-04-26T01:57:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 1233 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1260 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1260_fsm 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1260_vm 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1261 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1261_fsm 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1261_vm 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1262 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 1262_fsm 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 1262_vm 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2396 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 01:41 2396_fsm 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2396_vm 2024-04-26T01:57:06.616 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2397 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2671 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2672 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2676 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2677 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2694 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2695 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2697 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 01:41 2698 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2846 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2847 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2964 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2965 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 2966 2024-04-26T01:57:06.617 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 2967 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 3592 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 3593 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4060 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4061 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4175 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4176 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4177 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4178 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4181 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4182 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4183 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4184 2024-04-26T01:57:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 4185 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 4186 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6000 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6001 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6002 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 6100 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6114 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 6115 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:57 pg_control 2024-04-26T01:57:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 01:41 pg_filenode.map 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Apr 26 01:41 pg_internal.init 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/log: 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:total 2 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1522 Apr 26 01:57 postgresql-Fri.log 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:total 1 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:57 . 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 mappings 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 26 01:57 replorigin_checkpoint 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 snapshots 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.621 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:57 .. 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:57 .. 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 . 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 members 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 offsets 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:total 8 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 .. 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:41 0000 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:total 8 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:41 . 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:41 .. 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 01:46 0000 2024-04-26T01:57:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.624 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-26T01:57:06.624 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.624 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.624 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.624 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-26T01:57:06.624 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.624 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.634 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:total 15 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 01:57 . 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Apr 26 01:57 db_0.stat 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Apr 26 01:57 db_13494.stat 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Apr 26 01:57 global.stat 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:57 . 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:total 112 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 01:56 . 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 01:56 0015 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.636 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:total 1048576 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 26 01:57 . 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:57 000000010000000000000082 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:50 000000010000000000000083 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:50 000000010000000000000084 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:46 000000010000000000000085 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 000000010000000000000086 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:48 000000010000000000000087 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:49 000000010000000000000088 2024-04-26T01:57:06.637 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:46 000000010000000000000089 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:49 00000001000000000000008A 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:48 00000001000000000000008B 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:49 00000001000000000000008C 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:48 00000001000000000000008D 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:46 00000001000000000000008E 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:46 00000001000000000000008F 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 000000010000000000000090 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 000000010000000000000091 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 000000010000000000000092 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:54 000000010000000000000093 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 000000010000000000000094 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:55 000000010000000000000095 2024-04-26T01:57:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 000000010000000000000096 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:55 000000010000000000000097 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 000000010000000000000098 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 000000010000000000000099 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 00000001000000000000009A 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 00000001000000000000009B 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 00000001000000000000009C 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 00000001000000000000009D 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 00000001000000000000009E 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 00000001000000000000009F 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:54 0000000100000000000000A0 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 0000000100000000000000A1 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 0000000100000000000000A2 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 0000000100000000000000A3 2024-04-26T01:57:06.639 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 0000000100000000000000A4 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 0000000100000000000000A5 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:54 0000000100000000000000A6 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 0000000100000000000000A7 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:53 0000000100000000000000A8 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 0000000100000000000000A9 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:53 0000000100000000000000AA 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 0000000100000000000000AB 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 0000000100000000000000AC 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:53 0000000100000000000000AD 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 0000000100000000000000AE 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:53 0000000100000000000000AF 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 0000000100000000000000B0 2024-04-26T01:57:06.640 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 0000000100000000000000B1 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:53 0000000100000000000000B2 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 0000000100000000000000B3 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 0000000100000000000000B4 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:51 0000000100000000000000B5 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:54 0000000100000000000000B6 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:55 0000000100000000000000B7 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:52 0000000100000000000000B8 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000B9 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000BA 2024-04-26T01:57:06.641 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000BB 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000BC 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000BD 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000BE 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000BF 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000C0 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 01:56 0000000100000000000000C1 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 archive_status 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:total 0 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 01:41 . 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 26 01:57 .. 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-26T01:57:06.643 INFO:teuthology.orchestra.run.smithi028.stdout:total 344 2024-04-26T01:57:06.643 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 01:56 . 2024-04-26T01:57:06.643 INFO:teuthology.orchestra.run.smithi028.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 01:57 .. 2024-04-26T01:57:06.643 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Apr 26 01:56 0000 2024-04-26T01:57:06.643 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 01:56 0001 2024-04-26T01:57:06.648 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-26T01:57:06.660 INFO:tasks.quiescer:joining Quiescers 2024-04-26T01:57:06.660 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-26T01:57:06.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:57:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:07.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:07 smithi028 ceph-mon[28247]: pgmap v507: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-26T01:57:07.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:07 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:07.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:07 smithi053 ceph-mon[32593]: pgmap v507: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-26T01:57:07.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:07 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:08.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:07 smithi169 ceph-mon[32851]: pgmap v507: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-26T01:57:08.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:07 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:08 smithi053 ceph-mon[32593]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:57:09.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:08 smithi028 ceph-mon[28247]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:57:09.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:08 smithi169 ceph-mon[32851]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:57:09.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:09 smithi028 ceph-mon[28247]: pgmap v509: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 19 MiB/s wr, 1.79k op/s 2024-04-26T01:57:09.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:09 smithi053 ceph-mon[32593]: pgmap v509: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 19 MiB/s wr, 1.79k op/s 2024-04-26T01:57:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 -- 172.21.15.28:0/2409066376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb150103b80 msgr2=0x7fb150103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 --2- 172.21.15.28:0/2409066376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb150103b80 0x7fb150103f60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb140009990 tx=0x7fb140009ca0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 -- 172.21.15.28:0/2409066376 shutdown_connections 2024-04-26T01:57:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 --2- 172.21.15.28:0/2409066376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15010cfb0 0x7fb15010f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 --2- 172.21.15.28:0/2409066376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb150103b80 0x7fb150103f60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 --2- 172.21.15.28:0/2409066376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1501044a0 0x7fb15010c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 -- 172.21.15.28:0/2409066376 >> 172.21.15.28:0/2409066376 conn(0x7fb1500fd3d0 msgr2=0x7fb1500ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 -- 172.21.15.28:0/2409066376 shutdown_connections 2024-04-26T01:57:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.150+0000 7fb14f200700 1 -- 172.21.15.28:0/2409066376 wait complete. 2024-04-26T01:57:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.152+0000 7fb14f200700 1 Processor -- start 2024-04-26T01:57:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.153+0000 7fb14f200700 1 -- start start 2024-04-26T01:57:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.153+0000 7fb14f200700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb150103b80 0x7fb1501017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.153+0000 7fb14f200700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1501044a0 0x7fb150101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.153+0000 7fb14f200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15010cfb0 0x7fb150102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.153+0000 7fb14f200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb150112be0 con 0x7fb15010cfb0 2024-04-26T01:57:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.153+0000 7fb14f200700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb150112a60 con 0x7fb150103b80 2024-04-26T01:57:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.153+0000 7fb14f200700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb150112d60 con 0x7fb1501044a0 2024-04-26T01:57:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb150103b80 0x7fb1501017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14d400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1501044a0 0x7fb150101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14de00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb150103b80 0x7fb1501017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35856/0 (socket says 172.21.15.28:35856) 2024-04-26T01:57:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14de00700 1 -- 172.21.15.28:0/3635838282 learned_addr learned my addr 172.21.15.28:0/3635838282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14e800700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15010cfb0 0x7fb150102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14d400700 1 -- 172.21.15.28:0/3635838282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb150103b80 msgr2=0x7fb1501017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14d400700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb150103b80 0x7fb1501017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14d400700 1 -- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15010cfb0 msgr2=0x7fb150102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14d400700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15010cfb0 0x7fb150102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.154+0000 7fb14d400700 1 -- 172.21.15.28:0/3635838282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1501a7680 con 0x7fb1501044a0 2024-04-26T01:57:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.155+0000 7fb14d400700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1501044a0 0x7fb150101d20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00e9c0 tx=0x7fb13c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.155+0000 7fb146a00700 1 -- 172.21.15.28:0/3635838282 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0095c0 con 0x7fb1501044a0 2024-04-26T01:57:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.155+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1501a7970 con 0x7fb1501044a0 2024-04-26T01:57:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.155+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1501a7eb0 con 0x7fb1501044a0 2024-04-26T01:57:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.155+0000 7fb146a00700 1 -- 172.21.15.28:0/3635838282 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb13c004830 con 0x7fb1501044a0 2024-04-26T01:57:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.155+0000 7fb146a00700 1 -- 172.21.15.28:0/3635838282 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c01c070 con 0x7fb1501044a0 2024-04-26T01:57:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.156+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb150069730 con 0x7fb1501044a0 2024-04-26T01:57:10.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.159+0000 7fb146a00700 1 -- 172.21.15.28:0/3635838282 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb13c00cd00 con 0x7fb1501044a0 2024-04-26T01:57:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.160+0000 7fb146a00700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1340a3e30 0x7fb1340a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.161+0000 7fb146a00700 1 -- 172.21.15.28:0/3635838282 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8557+0+0 (secure 0 0 0) 0x7fb13c0d0610 con 0x7fb1501044a0 2024-04-26T01:57:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.161+0000 7fb14de00700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1340a3e30 0x7fb1340a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.161+0000 7fb14de00700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1340a3e30 0x7fb1340a62f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb140005cd0 tx=0x7fb140005c60 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.162+0000 7fb146a00700 1 -- 172.21.15.28:0/3635838282 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb13c0173d0 con 0x7fb1501044a0 2024-04-26T01:57:10.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:09 smithi169 ceph-mon[32851]: pgmap v509: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 19 MiB/s wr, 1.79k op/s 2024-04-26T01:57:10.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.446+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb150102d10 con 0x7fb1501044a0 2024-04-26T01:57:10.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.447+0000 7fb146a00700 1 -- 172.21.15.28:0/3635838282 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11865 (secure 0 0 0) 0x7fb13c09ccc0 con 0x7fb1501044a0 2024-04-26T01:57:10.447 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:10.447 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:57:05.890808+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24523,24509]},"id":1}]} 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1340a3e30 msgr2=0x7fb1340a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1340a3e30 0x7fb1340a62f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb140005cd0 tx=0x7fb140005c60 comp rx=0 tx=0).stop 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1501044a0 msgr2=0x7fb150101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1501044a0 0x7fb150101d20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00e9c0 tx=0x7fb13c00ecd0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 shutdown_connections 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb1340a3e30 0x7fb1340a62f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15010cfb0 0x7fb150102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb150103b80 0x7fb1501017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 --2- 172.21.15.28:0/3635838282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1501044a0 0x7fb150101d20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 >> 172.21.15.28:0/3635838282 conn(0x7fb1500fd3d0 msgr2=0x7fb1500fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:10.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 shutdown_connections 2024-04-26T01:57:10.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:10.451+0000 7fb14f200700 1 -- 172.21.15.28:0/3635838282 wait complete. 2024-04-26T01:57:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 48 2024-04-26T01:57:10.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:10 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3635838282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:10.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:10 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3635838282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:11.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:10 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3635838282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:11.302 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --cancel --all' 2024-04-26T01:57:11.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --cancel --all 2024-04-26T01:57:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:11.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:11 smithi053 ceph-mon[32593]: pgmap v510: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 9.8 MiB/s wr, 1.07k op/s 2024-04-26T01:57:12.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:11 smithi028 ceph-mon[28247]: pgmap v510: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 9.8 MiB/s wr, 1.07k op/s 2024-04-26T01:57:12.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:11 smithi169 ceph-mon[32851]: pgmap v510: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 9.8 MiB/s wr, 1.07k op/s 2024-04-26T01:57:12.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:12 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:13.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:12 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:13.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:12 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:13.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:13 smithi053 ceph-mon[32593]: pgmap v511: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 6.6 MiB/s wr, 730 op/s 2024-04-26T01:57:13.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:13 smithi053 ceph-mon[32593]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:57:14.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:13 smithi028 ceph-mon[28247]: pgmap v511: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 6.6 MiB/s wr, 730 op/s 2024-04-26T01:57:14.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:13 smithi028 ceph-mon[28247]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:57:14.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:13 smithi169 ceph-mon[32851]: pgmap v511: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 6.6 MiB/s wr, 730 op/s 2024-04-26T01:57:14.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:13 smithi169 ceph-mon[32851]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:57:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 -- 172.21.15.28:0/3094720558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 msgr2=0x7f3bd0111a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/3094720558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 0x7f3bd0111a80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009fc0 tx=0x7f3bc800b330 comp rx=0 tx=0).stop 2024-04-26T01:57:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 -- 172.21.15.28:0/3094720558 shutdown_connections 2024-04-26T01:57:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/3094720558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 0x7f3bd0111a80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/3094720558 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd0104dd0 0x7f3bd01051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/3094720558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd0105780 0x7f3bd01097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 -- 172.21.15.28:0/3094720558 >> 172.21.15.28:0/3094720558 conn(0x7f3bd0100660 msgr2=0x7f3bd0102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 -- 172.21.15.28:0/3094720558 shutdown_connections 2024-04-26T01:57:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.636+0000 7f3bd5c00700 1 -- 172.21.15.28:0/3094720558 wait complete. 2024-04-26T01:57:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.637+0000 7f3bd5c00700 1 Processor -- start 2024-04-26T01:57:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.638+0000 7f3bd5c00700 1 -- start start 2024-04-26T01:57:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.638+0000 7f3bd5c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd0104dd0 0x7f3bd019b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.638+0000 7f3bd5c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd0105780 0x7f3bd019b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.638+0000 7f3bd5c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 0x7f3bd019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.638+0000 7f3bd5c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd0114d90 con 0x7f3bd0109eb0 2024-04-26T01:57:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.638+0000 7f3bd5c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd0114c10 con 0x7f3bd0105780 2024-04-26T01:57:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.638+0000 7f3bd5c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd0114f10 con 0x7f3bd0104dd0 2024-04-26T01:57:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd0104dd0 0x7f3bd019b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bd5200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 0x7f3bd019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd0104dd0 0x7f3bd019b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:50750/0 (socket says 172.21.15.28:50750) 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2237561484 learned_addr learned my addr 172.21.15.28:0/2237561484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcf400700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd0105780 0x7f3bd019b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2237561484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd0105780 msgr2=0x7f3bd019b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd0105780 0x7f3bd019b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 msgr2=0x7f3bd019fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 0x7f3bd019fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2237561484 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd01a01d0 con 0x7f3bd0104dd0 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bd5200700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 0x7f3bd019fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.639+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd0104dd0 0x7f3bd019b1c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000e9a0 tx=0x7f3bc000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.640+0000 7f3bccc00700 1 -- 172.21.15.28:0/2237561484 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc000c9c0 con 0x7f3bd0104dd0 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.640+0000 7f3bd5c00700 1 -- 172.21.15.28:0/2237561484 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd01a04c0 con 0x7f3bd0104dd0 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.640+0000 7f3bccc00700 1 -- 172.21.15.28:0/2237561484 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc000cb60 con 0x7f3bd0104dd0 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.640+0000 7f3bccc00700 1 -- 172.21.15.28:0/2237561484 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0018680 con 0x7f3bd0104dd0 2024-04-26T01:57:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.640+0000 7f3bd5c00700 1 -- 172.21.15.28:0/2237561484 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd01a7d40 con 0x7f3bd0104dd0 2024-04-26T01:57:14.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.645+0000 7f3bccc00700 1 -- 172.21.15.28:0/2237561484 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3bc0010430 con 0x7f3bd0104dd0 2024-04-26T01:57:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.646+0000 7f3be1b07b80 1 Processor -- start 2024-04-26T01:57:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.646+0000 7f3be1b07b80 1 -- start start 2024-04-26T01:57:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.646+0000 7f3bccc00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f3bb40a3db0 0x7f3bb40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.646+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 0x55e0dbdff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe019d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0dbd46f00 con 0x55e0dbdf3e10 2024-04-26T01:57:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e0dbced7a0 con 0x55e0dbdff5e0 2024-04-26T01:57:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55e0dbd42ec0 con 0x55e0dbdf4730 2024-04-26T01:57:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3bccc00700 1 -- 172.21.15.28:0/2237561484 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7f3bc00d0b20 con 0x7f3bd0104dd0 2024-04-26T01:57:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3bcfe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3bd5200700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f3bb40a3db0 0x7f3bb40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3bcfe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48208/0 (socket says 172.21.15.28:48208) 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2151082526 learned_addr learned my addr 172.21.15.28:0/2151082526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3bcf400700 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 0x55e0dbdff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3bd5200700 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.647+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2151082526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 msgr2=0x55e0dbdff0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 0x55e0dbdff0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2151082526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdff5e0 msgr2=0x55e0dbe019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2151082526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0dbce36f0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bd5200700 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf41f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0099430 tx=0x7f3bc009c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874b402b1d98e3c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bd5200700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f3bb40a3db0 0x7f3bb40a6270 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800b6c0 tx=0x7f3bc80092e0 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bc5600700 1 -- 172.21.15.28:0/2151082526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00bd0b0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bc5600700 1 -- 172.21.15.28:0/2151082526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc009d160 con 0x55e0dbdf3e10 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.648+0000 7f3bc5600700 1 -- 172.21.15.28:0/2151082526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc009adf0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.649+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2151082526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf3e10 msgr2=0x55e0dbdf41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.649+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf41f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0099430 tx=0x7f3bc009c440 comp rx=0 tx=0).stop 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.649+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2151082526 shutdown_connections 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.649+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.649+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf41f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.649+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2151082526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 0x55e0dbdff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.649+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2151082526 >> 172.21.15.28:0/2151082526 conn(0x55e0dbd22fc0 msgr2=0x55e0dbdf0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.649+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2151082526 shutdown_connections 2024-04-26T01:57:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.650+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2151082526 wait complete. 2024-04-26T01:57:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.650+0000 7f3be1b07b80 1 -- start start 2024-04-26T01:57:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.650+0000 7f3be1b07b80 1 Processor -- start 2024-04-26T01:57:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdfeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf4730 0x55e0dbdfc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdff5e0 0x55e0dbdfc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0dbe01fb0 con 0x55e0dbdf4730 2024-04-26T01:57:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e0dbd008d0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x55e0dbe02130 con 0x55e0dbdff5e0 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3be1b07b80 10 client.0 _ll_register_callbacks cb 0x55e0dbcdf790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3bcf400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf4730 0x55e0dbdfc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3bcf400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf4730 0x55e0dbdfc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48222/0 (socket says 172.21.15.28:48222) 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3bcf400700 1 -- 172.21.15.28:0/3838129109 learned_addr learned my addr 172.21.15.28:0/3838129109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdfeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.651+0000 7f3bd5200700 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdff5e0 0x55e0dbdfc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bcfe00700 1 -- 172.21.15.28:0/3838129109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdff5e0 msgr2=0x55e0dbdfc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdff5e0 0x55e0dbdfc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bcfe00700 1 -- 172.21.15.28:0/3838129109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf4730 msgr2=0x55e0dbdfc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf4730 0x55e0dbdfc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bcfe00700 1 -- 172.21.15.28:0/3838129109 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0dbc94940 con 0x55e0dbdf3e10 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bcf400700 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf4730 0x55e0dbdfc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdfeda0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00bd250 tx=0x7f3bc00ce200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc28721d1619e9d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bba000700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bba000700 1 -- 172.21.15.28:0/3838129109 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00ba310 con 0x55e0dbdf3e10 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3bba000700 1 -- 172.21.15.28:0/3838129109 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7f3bc009ccc0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.652+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3838129109 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bc009d160 con 0x55e0dbdf3e10 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3bba000700 1 -- 172.21.15.28:0/3838129109 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3bc00a0e10 con 0x55e0dbdf3e10 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3be1b07b80 10 client.26537 fetch_fsmap learned FSMap version 49 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3838129109 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e0dbcddb60 con 0x55e0dbdf3e10 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3bba000700 1 -- 172.21.15.28:0/3838129109 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc009d160 con 0x55e0dbdf3e10 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3bba000700 1 -- 172.21.15.28:0/3838129109 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f3bc00a5240 con 0x55e0dbdf3e10 2024-04-26T01:57:14.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3be1b07b80 10 client.26537 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T01:57:14.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3be1b07b80 10 client.26537 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:57:14.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3be1b07b80 20 client.26537 populate_metadata read hostname 'smithi028' 2024-04-26T01:57:14.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e0dbe0eb70 0x55e0dbe10f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3be1b07b80 4 client.26537 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:14.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.653+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3838129109 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0dbd546f0 con 0x55e0dbe0eb70 2024-04-26T01:57:14.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.654+0000 7f3bcf400700 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e0dbe0eb70 0x55e0dbe10f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.654+0000 7f3bcf400700 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e0dbe0eb70 0x55e0dbe10f60 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7de9f6ba1ee48ad0 server_cookie=51c9f6c67daa435f in_seq=0 out_seq=0 2024-04-26T01:57:14.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.654+0000 7f3bba000700 10 client.26537 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:57:14.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.655+0000 7f3bba000700 1 -- 172.21.15.28:0/3838129109 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0dbd546f0 con 0x55e0dbe0eb70 2024-04-26T01:57:14.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.655+0000 7f3bba000700 10 client.26537 handle_command_reply: tid=0 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.655+0000 7f3be1b07b80 1 client.26537 shutdown 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.655+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3838129109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 msgr2=0x55e0dbdfeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.655+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdfeda0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00bd250 tx=0x7f3bc00ce200 comp rx=0 tx=0).stop 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.655+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3838129109 shutdown_connections 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.655+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e0dbe0eb70 0x55e0dbe10f60 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.656+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdff5e0 0x55e0dbdfc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.656+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdfeda0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.656+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3838129109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdf4730 0x55e0dbdfc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.656+0000 7f3bba000700 0 client.26537 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.656+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3838129109 >> 172.21.15.28:0/3838129109 conn(0x55e0dbd22fc0 msgr2=0x55e0dbdeff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.656+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3838129109 shutdown_connections 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.656+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3838129109 wait complete. 2024-04-26T01:57:14.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.656+0000 7f3be1b07b80 20 client.26537 trim_cache size 0 max 16384 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.679+0000 7f3be1b07b80 1 Processor -- start 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.679+0000 7f3be1b07b80 1 -- start start 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.679+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.679+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 0x55e0dbe0ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.679+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe0b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.679+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd01b1cf0 con 0x55e0dbdff5e0 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.679+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd01af520 con 0x55e0dbdf3e10 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.679+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd01ad390 con 0x55e0dbdf4730 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bd5200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe0b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bcfe00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bd5200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe0b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48230/0 (socket says 172.21.15.28:48230) 2024-04-26T01:57:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bd5200700 1 -- 172.21.15.28:0/3790877375 learned_addr learned my addr 172.21.15.28:0/3790877375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:14.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bcf400700 1 --2- 172.21.15.28:0/3790877375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 0x55e0dbe0ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bcfe00700 1 -- 172.21.15.28:0/3790877375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 msgr2=0x55e0dbe0ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:14.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/3790877375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 0x55e0dbe0ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bcfe00700 1 -- 172.21.15.28:0/3790877375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 msgr2=0x55e0dbe0b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:14.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/3790877375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe0b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.680+0000 7f3bcfe00700 1 -- 172.21.15.28:0/3790877375 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0dbdfb750 con 0x55e0dbdf3e10 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.681+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/3790877375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf04e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0010a00 tx=0x7f3bc009af70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8d3532ef0a9d2f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.681+0000 7f3bc4c00700 1 -- 172.21.15.28:0/3790877375 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc009d160 con 0x55e0dbdf3e10 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.681+0000 7f3bc4c00700 1 -- 172.21.15.28:0/3790877375 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc00a56d0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.681+0000 7f3bc4c00700 1 -- 172.21.15.28:0/3790877375 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00987b0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3790877375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 msgr2=0x55e0dbdf04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3790877375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf04e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0010a00 tx=0x7f3bc009af70 comp rx=0 tx=0).stop 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3790877375 shutdown_connections 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3790877375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe0b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3790877375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf3e10 0x55e0dbdf04e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/3790877375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf4730 0x55e0dbe0ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3790877375 >> 172.21.15.28:0/3790877375 conn(0x55e0dbd22fc0 msgr2=0x55e0dbdfe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3790877375 shutdown_connections 2024-04-26T01:57:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 -- 172.21.15.28:0/3790877375 wait complete. 2024-04-26T01:57:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.682+0000 7f3be1b07b80 1 -- start start 2024-04-26T01:57:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3be1b07b80 1 Processor -- start 2024-04-26T01:57:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf3e10 0x55e0dbe14420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf4730 0x55e0dbe127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3be1b07b80 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe12e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd01ad9e0 con 0x55e0dbdff5e0 2024-04-26T01:57:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd01aee50 con 0x55e0dbdf4730 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3be1b07b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd01ae0b0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3be1b07b80 10 client.0 _ll_register_callbacks cb 0x55e0dbcdf790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3bcfe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf3e10 0x55e0dbe14420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.683+0000 7f3bd5200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe12e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcfe00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf3e10 0x55e0dbe14420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:50782/0 (socket says 172.21.15.28:50782) 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2301932471 learned_addr learned my addr 172.21.15.28:0/2301932471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcf400700 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf4730 0x55e0dbe127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2301932471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf4730 msgr2=0x55e0dbe127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf4730 0x55e0dbe127d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2301932471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 msgr2=0x55e0dbe12e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe12e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcfe00700 1 -- 172.21.15.28:0/2301932471 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0dbe135d0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcf400700 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf4730 0x55e0dbe127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bcfe00700 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf3e10 0x55e0dbe14420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00bd220 tx=0x7f3bc00b6430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=861bb27e705aa92c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bba000700 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bba000700 1 -- 172.21.15.28:0/2301932471 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc009ca60 con 0x55e0dbdf3e10 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3bba000700 1 -- 172.21.15.28:0/2301932471 <== mon.1 v2:172.21.15.169:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7f3bc00ba230 con 0x55e0dbdf3e10 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.684+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2301932471 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bc00a56d0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3bba000700 1 -- 172.21.15.28:0/2301932471 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3bc009beb0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3be1b07b80 10 client.26548 fetch_fsmap learned FSMap version 49 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2301932471 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e0dbe135d0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3bba000700 1 -- 172.21.15.28:0/2301932471 <== mon.1 v2:172.21.15.169:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc00a56d0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3bba000700 1 -- 172.21.15.28:0/2301932471 <== mon.1 v2:172.21.15.169:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f3bc00672b0 con 0x55e0dbdf3e10 2024-04-26T01:57:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3be1b07b80 10 client.26548 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3be1b07b80 10 client.26548 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3be1b07b80 20 client.26548 populate_metadata read hostname 'smithi028' 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e0dbe760e0 0x55e0dbe78500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3be1b07b80 4 client.26548 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.685+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2301932471 --> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55e0dbd4e790 con 0x55e0dbe760e0 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.686+0000 7f3bcf400700 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e0dbe760e0 0x55e0dbe78500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.686+0000 7f3bcf400700 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e0dbe760e0 0x55e0dbe78500 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f177b0df8c1f48ed server_cookie=9d213c376d6e3b43 in_seq=0 out_seq=0 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.686+0000 7f3bba000700 10 client.26548 ms_handle_connect on v2:172.21.15.169:6838/2157805186 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3bba000700 1 -- 172.21.15.28:0/2301932471 <== mds.0 v2:172.21.15.169:6838/2157805186 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55e0dbd4e790 con 0x55e0dbe760e0 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3bba000700 10 client.26548 handle_command_reply: tid=0 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 client.26548 shutdown 2024-04-26T01:57:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2301932471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf3e10 msgr2=0x55e0dbe14420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf3e10 0x55e0dbe14420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00bd220 tx=0x7f3bc00b6430 comp rx=0 tx=0).stop 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2301932471 shutdown_connections 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0dbdff5e0 0x55e0dbe12e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.169:6838/2157805186,v1:172.21.15.169:6839/2157805186] conn(0x55e0dbe760e0 0x55e0dbe78500 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0dbdf3e10 0x55e0dbe14420 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 --2- 172.21.15.28:0/2301932471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e0dbdf4730 0x55e0dbe127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3bba000700 0 client.26548 ms_handle_reset on v2:172.21.15.169:6838/2157805186 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2301932471 >> 172.21.15.28:0/2301932471 conn(0x55e0dbd22fc0 msgr2=0x55e0dbdefd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2301932471 shutdown_connections 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 1 -- 172.21.15.28:0/2301932471 wait complete. 2024-04-26T01:57:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.687+0000 7f3be1b07b80 20 client.26548 trim_cache size 0 max 16384 2024-04-26T01:57:14.691 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:57:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 49, 2024-04-26T01:57:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24523, 2024-04-26T01:57:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 15, 2024-04-26T01:57:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": {} 2024-04-26T01:57:14.691 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 -- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f3bb40a3db0 msgr2=0x7f3bb40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f3bb40a3db0 0x7f3bb40a6270 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800b6c0 tx=0x7f3bc80092e0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 -- 172.21.15.28:0/2237561484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd0104dd0 msgr2=0x7f3bd019b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd0104dd0 0x7f3bd019b1c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000e9a0 tx=0x7f3bc000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 -- 172.21.15.28:0/2237561484 shutdown_connections 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f3bb40a3db0 0x7f3bb40a6270 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd0109eb0 0x7f3bd019fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd0104dd0 0x7f3bd019b1c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 --2- 172.21.15.28:0/2237561484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd0105780 0x7f3bd019b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.688+0000 7f3bd5c00700 1 -- 172.21.15.28:0/2237561484 >> 172.21.15.28:0/2237561484 conn(0x7f3bd0100660 msgr2=0x7f3bd0101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.689+0000 7f3bd5c00700 1 -- 172.21.15.28:0/2237561484 shutdown_connections 2024-04-26T01:57:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:14.689+0000 7f3bd5c00700 1 -- 172.21.15.28:0/2237561484 wait complete. 2024-04-26T01:57:15.515 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 419, in task stop_all_quiescers(ctx.ceph[cluster_name].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 378, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error releasing set '6288c6a': 1 (EPERM) 2024-04-26T01:57:15.516 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-26T01:57:15.527 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:57:15.553 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-26T01:57:15.564 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-26T01:57:15.565 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:57:15.565 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:57:15.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-26T01:57:15.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-26T01:57:15.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T01:57:15.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:15 smithi053 ceph-mon[32593]: pgmap v513: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 319 KiB/s wr, 41 op/s 2024-04-26T01:57:15.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2597437163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ae0109490 msgr2=0x7f2ae0111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:15.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2597437163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ae0109490 0x7f2ae0111060 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009900 tx=0x7f2ad802a410 comp rx=0 tx=0).stop 2024-04-26T01:57:15.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2597437163 shutdown_connections 2024-04-26T01:57:15.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2597437163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ae0109490 0x7f2ae0111060 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:15.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2597437163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae0104d70 0x7f2ae0108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:15.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2597437163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ae01043c0 0x7f2ae01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:15.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2597437163 >> 172.21.15.28:0/2597437163 conn(0x7f2ae00ffc80 msgr2=0x7f2ae01020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:15.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2597437163 shutdown_connections 2024-04-26T01:57:15.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.991+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2597437163 wait complete. 2024-04-26T01:57:15.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.992+0000 7f2ae5e00640 1 Processor -- start 2024-04-26T01:57:15.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ae5e00640 1 -- start start 2024-04-26T01:57:15.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ae5e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ae01043c0 0x7f2ae01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ae5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ae0109490 0x7f2ae01a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ae5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae019a6e0 0x7f2ae019ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ae5e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01143e0 con 0x7f2ae0109490 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ae5e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae0114260 con 0x7f2ae019a6e0 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ae5e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae0114560 con 0x7f2ae01043c0 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ad6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ae0109490 0x7f2ae01a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ad6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ae0109490 0x7f2ae01a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48248/0 (socket says 172.21.15.28:48248) 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2adea00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ae01043c0 0x7f2ae01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2ad6000640 1 -- 172.21.15.28:0/2449700717 learned_addr learned my addr 172.21.15.28:0/2449700717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.993+0000 7f2adf400640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae019a6e0 0x7f2ae019ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ad6000640 1 -- 172.21.15.28:0/2449700717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ae01043c0 msgr2=0x7f2ae01a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ad6000640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ae01043c0 0x7f2ae01a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ad6000640 1 -- 172.21.15.28:0/2449700717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae019a6e0 msgr2=0x7f2ae019ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ad6000640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae019a6e0 0x7f2ae019ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:15.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ad6000640 1 -- 172.21.15.28:0/2449700717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae019b2e0 con 0x7f2ae0109490 2024-04-26T01:57:15.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ad6000640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ae0109490 0x7f2ae01a0b60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800d770 tx=0x7f2ac800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:15.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ae4a00640 1 -- 172.21.15.28:0/2449700717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8017070 con 0x7f2ae0109490 2024-04-26T01:57:15.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ae4a00640 1 -- 172.21.15.28:0/2449700717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ac8009c30 con 0x7f2ae0109490 2024-04-26T01:57:15.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ae4a00640 1 -- 172.21.15.28:0/2449700717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac800f620 con 0x7f2ae0109490 2024-04-26T01:57:15.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.994+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae01a73c0 con 0x7f2ae0109490 2024-04-26T01:57:15.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.995+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae01a7770 con 0x7f2ae0109490 2024-04-26T01:57:15.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.996+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae0105890 con 0x7f2ae0109490 2024-04-26T01:57:15.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.998+0000 7f2ae4a00640 1 -- 172.21.15.28:0/2449700717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2ac801a070 con 0x7f2ae0109490 2024-04-26T01:57:15.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.999+0000 7f2ae4a00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2ac40a3e40 0x7f2ac40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:15.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:15.999+0000 7f2ae4a00640 1 -- 172.21.15.28:0/2449700717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7f2ac80d01d0 con 0x7f2ae0109490 2024-04-26T01:57:15.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.000+0000 7f2adea00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2ac40a3e40 0x7f2ac40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:15.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.000+0000 7f2adea00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2ac40a3e40 0x7f2ac40a6300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0005fd0 tx=0x7f2ad0009500 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.004+0000 7f2ae4a00640 1 -- 172.21.15.28:0/2449700717 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ac8009de0 con 0x7f2ae0109490 2024-04-26T01:57:16.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:15 smithi028 ceph-mon[28247]: pgmap v513: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 319 KiB/s wr, 41 op/s 2024-04-26T01:57:16.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:15 smithi169 ceph-mon[32851]: pgmap v513: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 319 KiB/s wr, 41 op/s 2024-04-26T01:57:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.242+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f2ae0104d70 con 0x7f2ac40a3e40 2024-04-26T01:57:16.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.257+0000 7f2ae4a00640 1 -- 172.21.15.28:0/2449700717 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f2ae0104d70 con 0x7f2ac40a3e40 2024-04-26T01:57:16.256 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T01:40:58", "first": "2024-04-26T01:41:00", "last": "2024-04-26T01:57:01", "last_pruned": "2024-04-26T01:57:03", "created_count": 17, "pruned_count": 11, "active": true} 2024-04-26T01:57:16.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2ac40a3e40 msgr2=0x7f2ac40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:16.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2ac40a3e40 0x7f2ac40a6300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0005fd0 tx=0x7f2ad0009500 comp rx=0 tx=0).stop 2024-04-26T01:57:16.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ae0109490 msgr2=0x7f2ae01a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ae0109490 0x7f2ae01a0b60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800d770 tx=0x7f2ac800dc30 comp rx=0 tx=0).stop 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 shutdown_connections 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f2ac40a3e40 0x7f2ac40a6300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae019a6e0 0x7f2ae019ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ae0109490 0x7f2ae01a0b60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 --2- 172.21.15.28:0/2449700717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ae01043c0 0x7f2ae01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.260+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 >> 172.21.15.28:0/2449700717 conn(0x7f2ae00ffc80 msgr2=0x7f2ae0101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.261+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 shutdown_connections 2024-04-26T01:57:16.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.261+0000 7f2ae5e00640 1 -- 172.21.15.28:0/2449700717 wait complete. 2024-04-26T01:57:16.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T01:57:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 -- 172.21.15.28:0/2997940647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 msgr2=0x7fcc5c1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 --2- 172.21.15.28:0/2997940647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 0x7fcc5c1023b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009900 tx=0x7fcc4c02a410 comp rx=0 tx=0).stop 2024-04-26T01:57:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 -- 172.21.15.28:0/2997940647 shutdown_connections 2024-04-26T01:57:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 --2- 172.21.15.28:0/2997940647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5c1070a0 0x7fcc5c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 --2- 172.21.15.28:0/2997940647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc5c102980 0x7fcc5c1069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 --2- 172.21.15.28:0/2997940647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 0x7fcc5c1023b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 -- 172.21.15.28:0/2997940647 >> 172.21.15.28:0/2997940647 conn(0x7fcc5c0fd890 msgr2=0x7fcc5c0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 -- 172.21.15.28:0/2997940647 shutdown_connections 2024-04-26T01:57:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.490+0000 7fcc60a00640 1 -- 172.21.15.28:0/2997940647 wait complete. 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.491+0000 7fcc60a00640 1 Processor -- start 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.491+0000 7fcc60a00640 1 -- start start 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc60a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 0x7fcc5c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc60a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc5c1070a0 0x7fcc5c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc60a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5c19cf50 0x7fcc5c19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc60a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c112030 con 0x7fcc5c101fd0 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc60a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c111eb0 con 0x7fcc5c1070a0 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc60a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c1121b0 con 0x7fcc5c19cf50 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 0x7fcc5c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 0x7fcc5c198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48272/0 (socket says 172.21.15.28:48272) 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5a800640 1 -- 172.21.15.28:0/448360484 learned_addr learned my addr 172.21.15.28:0/448360484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5b200640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5c19cf50 0x7fcc5c19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc59e00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc5c1070a0 0x7fcc5c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5a800640 1 -- 172.21.15.28:0/448360484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5c19cf50 msgr2=0x7fcc5c19d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5a800640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5c19cf50 0x7fcc5c19d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5a800640 1 -- 172.21.15.28:0/448360484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc5c1070a0 msgr2=0x7fcc5c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5a800640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc5c1070a0 0x7fcc5c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.492+0000 7fcc5a800640 1 -- 172.21.15.28:0/448360484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5c19da40 con 0x7fcc5c101fd0 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.493+0000 7fcc5a800640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 0x7fcc5c198690 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c005ab0 tx=0x7fcc4c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.493+0000 7fcc53400640 1 -- 172.21.15.28:0/448360484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c03f070 con 0x7fcc5c101fd0 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.493+0000 7fcc53400640 1 -- 172.21.15.28:0/448360484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc4c031b60 con 0x7fcc5c101fd0 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.493+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5c19dcd0 con 0x7fcc5c101fd0 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.493+0000 7fcc53400640 1 -- 172.21.15.28:0/448360484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c035030 con 0x7fcc5c101fd0 2024-04-26T01:57:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.493+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5c1a54f0 con 0x7fcc5c101fd0 2024-04-26T01:57:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.495+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc5c050530 con 0x7fcc5c101fd0 2024-04-26T01:57:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.497+0000 7fcc53400640 1 -- 172.21.15.28:0/448360484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcc4c042070 con 0x7fcc5c101fd0 2024-04-26T01:57:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.498+0000 7fcc53400640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc400a3d90 0x7fcc400a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.498+0000 7fcc53400640 1 -- 172.21.15.28:0/448360484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7fcc4c0f1620 con 0x7fcc5c101fd0 2024-04-26T01:57:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.498+0000 7fcc59e00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc400a3d90 0x7fcc400a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.499+0000 7fcc59e00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc400a3d90 0x7fcc400a6250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc480095f0 tx=0x7fcc48005f90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.501+0000 7fcc53400640 1 -- 172.21.15.28:0/448360484 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc4c0bd610 con 0x7fcc5c101fd0 2024-04-26T01:57:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.746+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 --> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fcc5c102980 con 0x7fcc400a3d90 2024-04-26T01:57:16.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.759+0000 7fcc53400640 1 -- 172.21.15.28:0/448360484 <== mgr.25034 v2:172.21.15.28:6800/4096350009 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fcc5c102980 con 0x7fcc400a3d90 2024-04-26T01:57:16.758 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc400a3d90 msgr2=0x7fcc400a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc400a3d90 0x7fcc400a6250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc480095f0 tx=0x7fcc48005f90 comp rx=0 tx=0).stop 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 msgr2=0x7fcc5c198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 0x7fcc5c198690 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c005ab0 tx=0x7fcc4c003c10 comp rx=0 tx=0).stop 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 shutdown_connections 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcc400a3d90 0x7fcc400a6250 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5c19cf50 0x7fcc5c19d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc5c1070a0 0x7fcc5c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 --2- 172.21.15.28:0/448360484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc5c101fd0 0x7fcc5c198690 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 >> 172.21.15.28:0/448360484 conn(0x7fcc5c0fd890 msgr2=0x7fcc5c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.762+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 shutdown_connections 2024-04-26T01:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:16.763+0000 7fcc60a00640 1 -- 172.21.15.28:0/448360484 wait complete. 2024-04-26T01:57:16.781 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-26T01:57:16.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:57:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 -- 172.21.15.28:0/1438465558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 msgr2=0x7fdc90115200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1438465558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 0x7fdc90115200 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009ca0 tx=0x7fdc8802a950 comp rx=0 tx=0).stop 2024-04-26T01:57:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 -- 172.21.15.28:0/1438465558 shutdown_connections 2024-04-26T01:57:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1438465558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 0x7fdc90115200 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1438465558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc901083f0 0x7fdc90108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1438465558 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc9010e460 0x7fdc9010e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 -- 172.21.15.28:0/1438465558 >> 172.21.15.28:0/1438465558 conn(0x7fdc90077ad0 msgr2=0x7fdc90079ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 -- 172.21.15.28:0/1438465558 shutdown_connections 2024-04-26T01:57:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.008+0000 7fdc95e00640 1 -- 172.21.15.28:0/1438465558 wait complete. 2024-04-26T01:57:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.009+0000 7fdc95e00640 1 Processor -- start 2024-04-26T01:57:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.009+0000 7fdc95e00640 1 -- start start 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.009+0000 7fdc95e00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc901083f0 0x7fdc901a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc95e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 0x7fdc901a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc95e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc9010e460 0x7fdc9019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc95e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc90118640 con 0x7fdc90108d90 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc95e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdc901184c0 con 0x7fdc9010e460 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc95e00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdc901187c0 con 0x7fdc901083f0 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 0x7fdc901a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 0x7fdc901a4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48294/0 (socket says 172.21.15.28:48294) 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc8fe00640 1 -- 172.21.15.28:0/1949566428 learned_addr learned my addr 172.21.15.28:0/1949566428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc94a00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc901083f0 0x7fdc901a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc95400640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc9010e460 0x7fdc9019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc8fe00640 1 -- 172.21.15.28:0/1949566428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc901083f0 msgr2=0x7fdc901a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc901083f0 0x7fdc901a46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc8fe00640 1 -- 172.21.15.28:0/1949566428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc9010e460 msgr2=0x7fdc9019e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc9010e460 0x7fdc9019e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.010+0000 7fdc8fe00640 1 -- 172.21.15.28:0/1949566428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9019f020 con 0x7fdc90108d90 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.011+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 0x7fdc901a4c30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00ea60 tx=0x7fdc7c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.011+0000 7fdc8d600640 1 -- 172.21.15.28:0/1949566428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c00cde0 con 0x7fdc90108d90 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.011+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9019f310 con 0x7fdc90108d90 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.011+0000 7fdc8d600640 1 -- 172.21.15.28:0/1949566428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc7c0035f0 con 0x7fdc90108d90 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.011+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc901ab5b0 con 0x7fdc90108d90 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.011+0000 7fdc8d600640 1 -- 172.21.15.28:0/1949566428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c005220 con 0x7fdc90108d90 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.012+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc900690e0 con 0x7fdc90108d90 2024-04-26T01:57:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.015+0000 7fdc8d600640 1 -- 172.21.15.28:0/1949566428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdc7c0037a0 con 0x7fdc90108d90 2024-04-26T01:57:17.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.016+0000 7fdc8d600640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdc780a3d20 0x7fdc780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.016+0000 7fdc8d600640 1 -- 172.21.15.28:0/1949566428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7fdc7c0d0a80 con 0x7fdc90108d90 2024-04-26T01:57:17.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.017+0000 7fdc94a00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdc780a3d20 0x7fdc780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.017+0000 7fdc94a00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdc780a3d20 0x7fdc780a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdc9019f9e0 tx=0x7fdc84005f90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:17.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.020+0000 7fdc8d600640 1 -- 172.21.15.28:0/1949566428 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc7c0d2e20 con 0x7fdc90108d90 2024-04-26T01:57:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.232+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc9002d5c0 con 0x7fdc90108d90 2024-04-26T01:57:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.236+0000 7fdc8d600640 1 -- 172.21.15.28:0/1949566428 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+23101 (secure 0 0 0) 0x7fdc7c09d0c0 con 0x7fdc90108d90 2024-04-26T01:57:17.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":113,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:57:13.436912+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":7,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3431203710":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/1295766352":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3441292878":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6800/3431203710":"2024-04-27T01:40:57.926618+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/218486566":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3010978828":"2024-04-27T01:40:57.926618+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:57:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.238+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdc780a3d20 msgr2=0x7fdc780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdc780a3d20 0x7fdc780a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdc9019f9e0 tx=0x7fdc84005f90 comp rx=0 tx=0).stop 2024-04-26T01:57:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 msgr2=0x7fdc901a4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 0x7fdc901a4c30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00ea60 tx=0x7fdc7c00ef20 comp rx=0 tx=0).stop 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 shutdown_connections 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fdc780a3d20 0x7fdc780a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc9010e460 0x7fdc9019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90108d90 0x7fdc901a4c30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 --2- 172.21.15.28:0/1949566428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc901083f0 0x7fdc901a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 >> 172.21.15.28:0/1949566428 conn(0x7fdc90077ad0 msgr2=0x7fdc90079130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 shutdown_connections 2024-04-26T01:57:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.239+0000 7fdc95e00640 1 -- 172.21.15.28:0/1949566428 wait complete. 2024-04-26T01:57:17.259 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T01:31:11.765874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T01:36:49.128472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T01:36:53.237400+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T01:57:17.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.476+0000 7f8e9f400640 1 -- 172.21.15.28:0/606616316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e98076480 msgr2=0x7f8e98076860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.476+0000 7f8e9f400640 1 --2- 172.21.15.28:0/606616316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e98076480 0x7f8e98076860 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009930 tx=0x7f8e8c02a440 comp rx=0 tx=0).stop 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.477+0000 7f8e9f400640 1 -- 172.21.15.28:0/606616316 shutdown_connections 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.477+0000 7f8e9f400640 1 --2- 172.21.15.28:0/606616316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e98112e30 0x7f8e98115220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.477+0000 7f8e9f400640 1 --2- 172.21.15.28:0/606616316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98076da0 0x7f8e981128f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.477+0000 7f8e9f400640 1 --2- 172.21.15.28:0/606616316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e98076480 0x7f8e98076860 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.477+0000 7f8e9f400640 1 -- 172.21.15.28:0/606616316 >> 172.21.15.28:0/606616316 conn(0x7f8e980ffce0 msgr2=0x7f8e98102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.477+0000 7f8e9f400640 1 -- 172.21.15.28:0/606616316 shutdown_connections 2024-04-26T01:57:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.477+0000 7f8e9f400640 1 -- 172.21.15.28:0/606616316 wait complete. 2024-04-26T01:57:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.478+0000 7f8e9f400640 1 Processor -- start 2024-04-26T01:57:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.478+0000 7f8e9f400640 1 -- start start 2024-04-26T01:57:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.478+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e98076480 0x7f8e9819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.478+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98076da0 0x7f8e9819f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.478+0000 7f8e9f400640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e98112e30 0x7f8e981a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.478+0000 7f8e9f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e98118840 con 0x7f8e98076da0 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.478+0000 7f8e9f400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981186c0 con 0x7f8e98076480 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.478+0000 7f8e9f400640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981189c0 con 0x7f8e98112e30 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e98076480 0x7f8e9819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e98076480 0x7f8e9819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:35996/0 (socket says 172.21.15.28:35996) 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 -- 172.21.15.28:0/690350669 learned_addr learned my addr 172.21.15.28:0/690350669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e97e00640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98076da0 0x7f8e9819f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9d400640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e98112e30 0x7f8e981a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 -- 172.21.15.28:0/690350669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e98112e30 msgr2=0x7f8e981a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e98112e30 0x7f8e981a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 -- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98076da0 msgr2=0x7f8e9819f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98076da0 0x7f8e9819f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 -- 172.21.15.28:0/690350669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e981a3e20 con 0x7f8e98076480 2024-04-26T01:57:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.479+0000 7f8e9ca00640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e98076480 0x7f8e9819ee80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c005940 tx=0x7f8e8c0040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:17.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.480+0000 7f8e95600640 1 -- 172.21.15.28:0/690350669 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c03f030 con 0x7f8e98076480 2024-04-26T01:57:17.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.480+0000 7f8e95600640 1 -- 172.21.15.28:0/690350669 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e8c0044c0 con 0x7f8e98076480 2024-04-26T01:57:17.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.480+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e981a40b0 con 0x7f8e98076480 2024-04-26T01:57:17.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.480+0000 7f8e95600640 1 -- 172.21.15.28:0/690350669 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c035070 con 0x7f8e98076480 2024-04-26T01:57:17.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.480+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e981ab9a0 con 0x7f8e98076480 2024-04-26T01:57:17.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.481+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e980691c0 con 0x7f8e98076480 2024-04-26T01:57:17.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.483+0000 7f8e95600640 1 -- 172.21.15.28:0/690350669 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8e8c042030 con 0x7f8e98076480 2024-04-26T01:57:17.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.484+0000 7f8e95600640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8e800a3d20 0x7f8e800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.484+0000 7f8e95600640 1 -- 172.21.15.28:0/690350669 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7f8e8c0f0ed0 con 0x7f8e98076480 2024-04-26T01:57:17.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.485+0000 7f8e97e00640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8e800a3d20 0x7f8e800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.485+0000 7f8e97e00640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8e800a3d20 0x7f8e800a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e981021b0 tx=0x7f8e84005f90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.487+0000 7f8e95600640 1 -- 172.21.15.28:0/690350669 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e8c0bd510 con 0x7f8e98076480 2024-04-26T01:57:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.697+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8e9810dfc0 con 0x7f8e98076480 2024-04-26T01:57:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.698+0000 7f8e95600640 1 -- 172.21.15.28:0/690350669 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7f8e8c0bce10 con 0x7f8e98076480 2024-04-26T01:57:17.697 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-26T01:57:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.701+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8e800a3d20 msgr2=0x7f8e800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.701+0000 7f8e9f400640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8e800a3d20 0x7f8e800a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e981021b0 tx=0x7f8e84005f90 comp rx=0 tx=0).stop 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.701+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e98076480 msgr2=0x7f8e9819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.701+0000 7f8e9f400640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e98076480 0x7f8e9819ee80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c005940 tx=0x7f8e8c0040c0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.702+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 shutdown_connections 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.702+0000 7f8e9f400640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f8e800a3d20 0x7f8e800a61e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.702+0000 7f8e9f400640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e98112e30 0x7f8e981a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.702+0000 7f8e9f400640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98076da0 0x7f8e9819f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.702+0000 7f8e9f400640 1 --2- 172.21.15.28:0/690350669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e98076480 0x7f8e9819ee80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.702+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 >> 172.21.15.28:0/690350669 conn(0x7f8e980ffce0 msgr2=0x7f8e98114550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.702+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 shutdown_connections 2024-04-26T01:57:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.702+0000 7f8e9f400640 1 -- 172.21.15.28:0/690350669 wait complete. 2024-04-26T01:57:17.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T01:57:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 -- 172.21.15.28:0/3490468918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48104260 msgr2=0x7f6d48104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/3490468918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48104260 0x7f6d48104640 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009860 tx=0x7f6d3802a340 comp rx=0 tx=0).stop 2024-04-26T01:57:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 -- 172.21.15.28:0/3490468918 shutdown_connections 2024-04-26T01:57:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/3490468918 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d48109330 0x7f6d48110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/3490468918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d48104c10 0x7f6d48108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/3490468918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48104260 0x7f6d48104640 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 -- 172.21.15.28:0/3490468918 >> 172.21.15.28:0/3490468918 conn(0x7f6d480ffac0 msgr2=0x7f6d48101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 -- 172.21.15.28:0/3490468918 shutdown_connections 2024-04-26T01:57:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.937+0000 7f6d4cc00640 1 -- 172.21.15.28:0/3490468918 wait complete. 2024-04-26T01:57:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.938+0000 7f6d4cc00640 1 Processor -- start 2024-04-26T01:57:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.938+0000 7f6d4cc00640 1 -- start start 2024-04-26T01:57:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d4cc00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d48104260 0x7f6d481a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d4cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48109330 0x7f6d481a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d4cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d4819a670 0x7f6d4819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d4cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48114270 con 0x7f6d48109330 2024-04-26T01:57:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d4cc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481140f0 con 0x7f6d4819a670 2024-04-26T01:57:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d4cc00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481143f0 con 0x7f6d48104260 2024-04-26T01:57:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48109330 0x7f6d481a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d4819a670 0x7f6d4819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d4819a670 0x7f6d4819ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36016/0 (socket says 172.21.15.28:36016) 2024-04-26T01:57:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47e00640 1 -- 172.21.15.28:0/1758068639 learned_addr learned my addr 172.21.15.28:0/1758068639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47400640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d48104260 0x7f6d481a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47e00640 1 -- 172.21.15.28:0/1758068639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d48104260 msgr2=0x7f6d481a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47e00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d48104260 0x7f6d481a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47e00640 1 -- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48109330 msgr2=0x7f6d481a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47e00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48109330 0x7f6d481a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.939+0000 7f6d47e00640 1 -- 172.21.15.28:0/1758068639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4819b190 con 0x7f6d4819a670 2024-04-26T01:57:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.940+0000 7f6d47e00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d4819a670 0x7f6d4819ab20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009fd0 tx=0x7f6d4000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.940+0000 7f6d44c00640 1 -- 172.21.15.28:0/1758068639 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40004b20 con 0x7f6d4819a670 2024-04-26T01:57:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.940+0000 7f6d44c00640 1 -- 172.21.15.28:0/1758068639 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d40003e20 con 0x7f6d4819a670 2024-04-26T01:57:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.940+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4819b480 con 0x7f6d4819a670 2024-04-26T01:57:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.940+0000 7f6d44c00640 1 -- 172.21.15.28:0/1758068639 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40004820 con 0x7f6d4819a670 2024-04-26T01:57:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.940+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d481a7630 con 0x7f6d4819a670 2024-04-26T01:57:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.941+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d480690e0 con 0x7f6d4819a670 2024-04-26T01:57:17.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.943+0000 7f6d44c00640 1 -- 172.21.15.28:0/1758068639 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6d4001d020 con 0x7f6d4819a670 2024-04-26T01:57:17.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.945+0000 7f6d44c00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f6d2c0a3d20 0x7f6d2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:17.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.945+0000 7f6d44c00640 1 -- 172.21.15.28:0/1758068639 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7f6d400d07a0 con 0x7f6d4819a670 2024-04-26T01:57:17.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.945+0000 7f6d47400640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f6d2c0a3d20 0x7f6d2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:17.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.946+0000 7f6d47400640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f6d2c0a3d20 0x7f6d2c0a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009830 tx=0x7f6d38030340 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:17.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:17.949+0000 7f6d44c00640 1 -- 172.21.15.28:0/1758068639 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d40004de0 con 0x7f6d4819a670 2024-04-26T01:57:18.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:17 smithi028 ceph-mon[28247]: from='client.16251 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:57:18.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:17 smithi028 ceph-mon[28247]: pgmap v514: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 332 KiB/s wr, 43 op/s 2024-04-26T01:57:18.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:17 smithi028 ceph-mon[28247]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:57:18.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:17 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1949566428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:57:18.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:17 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/690350669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:57:18.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.159+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6d4810c590 con 0x7f6d4819a670 2024-04-26T01:57:18.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.160+0000 7f6d44c00640 1 -- 172.21.15.28:0/1758068639 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7f6d4009cde0 con 0x7f6d4819a670 2024-04-26T01:57:18.159 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-26T01:57:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f6d2c0a3d20 msgr2=0x7f6d2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f6d2c0a3d20 0x7f6d2c0a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009830 tx=0x7f6d38030340 comp rx=0 tx=0).stop 2024-04-26T01:57:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d4819a670 msgr2=0x7f6d4819ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d4819a670 0x7f6d4819ab20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009fd0 tx=0x7f6d4000c800 comp rx=0 tx=0).stop 2024-04-26T01:57:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 shutdown_connections 2024-04-26T01:57:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f6d2c0a3d20 0x7f6d2c0a61e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d4819a670 0x7f6d4819ab20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48109330 0x7f6d481a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 --2- 172.21.15.28:0/1758068639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d48104260 0x7f6d481a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 >> 172.21.15.28:0/1758068639 conn(0x7f6d480ffac0 msgr2=0x7f6d48100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 shutdown_connections 2024-04-26T01:57:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.163+0000 7f6d4cc00640 1 -- 172.21.15.28:0/1758068639 wait complete. 2024-04-26T01:57:18.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:17 smithi169 ceph-mon[32851]: from='client.16251 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:57:18.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:17 smithi169 ceph-mon[32851]: pgmap v514: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 332 KiB/s wr, 43 op/s 2024-04-26T01:57:18.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:17 smithi169 ceph-mon[32851]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:57:18.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:17 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1949566428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:57:18.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:17 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/690350669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:57:18.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T01:57:18.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:17 smithi053 ceph-mon[32593]: from='client.16251 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:57:18.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:17 smithi053 ceph-mon[32593]: pgmap v514: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 332 KiB/s wr, 43 op/s 2024-04-26T01:57:18.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:17 smithi053 ceph-mon[32593]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:57:18.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:17 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1949566428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:57:18.221 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:17 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/690350669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 -- 172.21.15.28:0/4112907538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd3810c430 msgr2=0x7fcd3810e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 --2- 172.21.15.28:0/4112907538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd3810c430 0x7fcd3810e8d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009be0 tx=0x7fcd3402a750 comp rx=0 tx=0).stop 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 -- 172.21.15.28:0/4112907538 shutdown_connections 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 --2- 172.21.15.28:0/4112907538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd3810c430 0x7fcd3810e8d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 --2- 172.21.15.28:0/4112907538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd38103930 0x7fcd3810be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 --2- 172.21.15.28:0/4112907538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd38103010 0x7fcd381033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 -- 172.21.15.28:0/4112907538 >> 172.21.15.28:0/4112907538 conn(0x7fcd380fc870 msgr2=0x7fcd380fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 -- 172.21.15.28:0/4112907538 shutdown_connections 2024-04-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.409+0000 7fcd3f200640 1 -- 172.21.15.28:0/4112907538 wait complete. 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.410+0000 7fcd3f200640 1 Processor -- start 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.410+0000 7fcd3f200640 1 -- start start 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3f200640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd38103010 0x7fcd38198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd38103930 0x7fcd38198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3810c430 0x7fcd3819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd381120c0 con 0x7fcd3810c430 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcd38111f40 con 0x7fcd38103930 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3f200640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcd38112240 con 0x7fcd38103010 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd38103930 0x7fcd38198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3810c430 0x7fcd3819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd38103930 0x7fcd38198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:36026/0 (socket says 172.21.15.28:36026) 2024-04-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3d400640 1 -- 172.21.15.28:0/2929228447 learned_addr learned my addr 172.21.15.28:0/2929228447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.411+0000 7fcd3de00640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd38103010 0x7fcd38198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd3d400640 1 -- 172.21.15.28:0/2929228447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd38103010 msgr2=0x7fcd38198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd3d400640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd38103010 0x7fcd38198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd3d400640 1 -- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3810c430 msgr2=0x7fcd3819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd3d400640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3810c430 0x7fcd3819cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd3d400640 1 -- 172.21.15.28:0/2929228447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3819d650 con 0x7fcd38103930 2024-04-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd3e800640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3810c430 0x7fcd3819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd3d400640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd38103930 0x7fcd38198ba0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800e770 tx=0x7fcd2800ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd32a00640 1 -- 172.21.15.28:0/2929228447 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800ccc0 con 0x7fcd38103930 2024-04-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd32a00640 1 -- 172.21.15.28:0/2929228447 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd2800ee80 con 0x7fcd38103930 2024-04-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd32a00640 1 -- 172.21.15.28:0/2929228447 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28010430 con 0x7fcd38103930 2024-04-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.412+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3819d940 con 0x7fcd38103930 2024-04-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.413+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd381a51d0 con 0x7fcd38103930 2024-04-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.414+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd3806ae50 con 0x7fcd38103930 2024-04-26T01:57:18.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.416+0000 7fcd32a00640 1 -- 172.21.15.28:0/2929228447 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcd28004f90 con 0x7fcd38103930 2024-04-26T01:57:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.417+0000 7fcd32a00640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcd200a3d20 0x7fcd200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.417+0000 7fcd32a00640 1 -- 172.21.15.28:0/2929228447 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7fcd280d09d0 con 0x7fcd38103930 2024-04-26T01:57:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.418+0000 7fcd3de00640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcd200a3d20 0x7fcd200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:18.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.418+0000 7fcd3de00640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcd200a3d20 0x7fcd200a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009bd0 tx=0x7fcd2c005f90 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:18.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.422+0000 7fcd32a00640 1 -- 172.21.15.28:0/2929228447 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd2809d010 con 0x7fcd38103930 2024-04-26T01:57:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.632+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcd381074d0 con 0x7fcd38103930 2024-04-26T01:57:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.633+0000 7fcd32a00640 1 -- 172.21.15.28:0/2929228447 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7fcd2809c910 con 0x7fcd38103930 2024-04-26T01:57:18.632 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-26T01:57:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcd200a3d20 msgr2=0x7fcd200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcd200a3d20 0x7fcd200a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009bd0 tx=0x7fcd2c005f90 comp rx=0 tx=0).stop 2024-04-26T01:57:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd38103930 msgr2=0x7fcd38198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd38103930 0x7fcd38198ba0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800e770 tx=0x7fcd2800ec30 comp rx=0 tx=0).stop 2024-04-26T01:57:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 shutdown_connections 2024-04-26T01:57:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcd200a3d20 0x7fcd200a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3810c430 0x7fcd3819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd38103930 0x7fcd38198ba0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 --2- 172.21.15.28:0/2929228447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd38103010 0x7fcd38198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.635+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 >> 172.21.15.28:0/2929228447 conn(0x7fcd380fc870 msgr2=0x7fcd380fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.636+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 shutdown_connections 2024-04-26T01:57:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.636+0000 7fcd3f200640 1 -- 172.21.15.28:0/2929228447 wait complete. 2024-04-26T01:57:18.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-26T01:57:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.868+0000 7f2419a00640 1 -- 172.21.15.28:0/937457044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2414104b70 msgr2=0x7f2414108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.868+0000 7f2419a00640 1 --2- 172.21.15.28:0/937457044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2414104b70 0x7f2414108bb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009860 tx=0x7f23fc02a340 comp rx=0 tx=0).stop 2024-04-26T01:57:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.868+0000 7f2419a00640 1 -- 172.21.15.28:0/937457044 shutdown_connections 2024-04-26T01:57:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.868+0000 7f2419a00640 1 --2- 172.21.15.28:0/937457044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2414109290 0x7f2414110e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.868+0000 7f2419a00640 1 --2- 172.21.15.28:0/937457044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2414104b70 0x7f2414108bb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.868+0000 7f2419a00640 1 --2- 172.21.15.28:0/937457044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24141041c0 0x7f24141045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.868+0000 7f2419a00640 1 -- 172.21.15.28:0/937457044 >> 172.21.15.28:0/937457044 conn(0x7f24140ffa20 msgr2=0x7f2414101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.868+0000 7f2419a00640 1 -- 172.21.15.28:0/937457044 shutdown_connections 2024-04-26T01:57:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.869+0000 7f2419a00640 1 -- 172.21.15.28:0/937457044 wait complete. 2024-04-26T01:57:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.870+0000 7f2419a00640 1 Processor -- start 2024-04-26T01:57:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.870+0000 7f2419a00640 1 -- start start 2024-04-26T01:57:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.870+0000 7f2419a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24141041c0 0x7f24141a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.870+0000 7f2419a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2414109290 0x7f24141a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.870+0000 7f2419a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241419a690 0x7f241419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.870+0000 7f2419a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24141141b0 con 0x7f241419a690 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.870+0000 7f2419a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2414114030 con 0x7f24141041c0 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.870+0000 7f2419a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2414114330 con 0x7f2414109290 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2413e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24141041c0 0x7f24141a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2413e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24141041c0 0x7f24141a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:57860/0 (socket says 172.21.15.28:57860) 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2413e00640 1 -- 172.21.15.28:0/2094173530 learned_addr learned my addr 172.21.15.28:0/2094173530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2419000640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241419a690 0x7f241419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f240b400640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2414109290 0x7f24141a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2419000640 1 -- 172.21.15.28:0/2094173530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2414109290 msgr2=0x7f24141a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2419000640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2414109290 0x7f24141a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2419000640 1 -- 172.21.15.28:0/2094173530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24141041c0 msgr2=0x7f24141a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2419000640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24141041c0 0x7f24141a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.871+0000 7f2419000640 1 -- 172.21.15.28:0/2094173530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241419b290 con 0x7f241419a690 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.872+0000 7f2419000640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241419a690 0x7f241419ab10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f240c00e970 tx=0x7f240c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.872+0000 7f2411600640 1 -- 172.21.15.28:0/2094173530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c00ccc0 con 0x7f241419a690 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.872+0000 7f2411600640 1 -- 172.21.15.28:0/2094173530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f240c005020 con 0x7f241419a690 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.872+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24141a7320 con 0x7f241419a690 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.872+0000 7f2411600640 1 -- 172.21.15.28:0/2094173530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c004a00 con 0x7f241419a690 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.872+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24141a76d0 con 0x7f241419a690 2024-04-26T01:57:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.873+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2414105690 con 0x7f241419a690 2024-04-26T01:57:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.875+0000 7f2411600640 1 -- 172.21.15.28:0/2094173530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f240c004ba0 con 0x7f241419a690 2024-04-26T01:57:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.877+0000 7f2411600640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f23f80a3d20 0x7f23f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.877+0000 7f2411600640 1 -- 172.21.15.28:0/2094173530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7f240c0d0740 con 0x7f241419a690 2024-04-26T01:57:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.877+0000 7f2413e00640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f23f80a3d20 0x7f23f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.878+0000 7f2413e00640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f23f80a3d20 0x7f23f80a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2404009a60 tx=0x7f2404010040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:18.882+0000 7f2411600640 1 -- 172.21.15.28:0/2094173530 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f240c09cd80 con 0x7f241419a690 2024-04-26T01:57:19.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:18 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:19.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:18 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1758068639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T01:57:19.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:18 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2929228447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T01:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.142+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f241410c4c0 con 0x7f241419a690 2024-04-26T01:57:19.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:18 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:19.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:18 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1758068639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T01:57:19.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:18 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2929228447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T01:57:19.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.172+0000 7f2411600640 1 -- 172.21.15.28:0/2094173530 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746063 (secure 0 0 0) 0x7f240c09c680 con 0x7f241419a690 2024-04-26T01:57:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:57:19.185 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:57:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 26, 2024-04-26T01:57:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 25034, 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4096350009 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-04-26T01:57:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4096350009 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/4096350009", 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-04-26T01:40:57.927242+0000", 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25031, 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-26T01:57:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-26T01:57:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-26T01:57:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-26T01:57:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-26T01:57:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-26T01:57:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T01:57:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-26T01:57:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-26T01:57:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-26T01:57:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-26T01:57:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-26T01:57:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-26T01:57:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T01:57:19.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-26T01:57:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-26T01:57:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-26T01:57:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-26T01:57:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-26T01:57:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T01:57:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-26T01:57:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-26T01:57:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T01:57:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-26T01:57:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-26T01:57:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-26T01:57:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-26T01:57:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T01:57:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-26T01:57:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-26T01:57:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-26T01:57:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-26T01:57:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T01:57:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T01:57:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-26T01:57:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-26T01:57:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-26T01:57:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T01:57:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-26T01:57:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-26T01:57:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-26T01:57:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-26T01:57:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-26T01:57:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-26T01:57:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-26T01:57:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-26T01:57:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-26T01:57:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-26T01:57:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T01:57:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T01:57:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-26T01:57:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-26T01:57:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T01:57:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T01:57:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T01:57:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T01:57:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T01:57:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-26T01:57:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-26T01:57:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.374 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-26T01:57:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-26T01:57:19.387 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T01:57:19.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-26T01:57:19.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-26T01:57:19.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-26T01:57:19.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-26T01:57:19.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.409 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-26T01:57:19.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-26T01:57:19.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-26T01:57:19.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:19.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.418 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.420 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.421 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.429 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.435 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.436 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.445 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.459 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.459 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.459 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.460 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-26T01:57:19.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-26T01:57:19.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-26T01:57:19.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T01:57:19.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.466 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-26T01:57:19.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-26T01:57:19.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-26T01:57:19.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-26T01:57:19.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.468 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-26T01:57:19.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-26T01:57:19.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-26T01:57:19.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T01:57:19.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.471 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-26T01:57:19.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-26T01:57:19.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-26T01:57:19.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.473 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-26T01:57:19.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-26T01:57:19.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-26T01:57:19.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T01:57:19.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T01:57:19.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.475 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.475 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-26T01:57:19.476 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.476 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.476 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.481 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.490 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-26T01:57:19.495 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.497 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.498 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.499 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.500 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-26T01:57:19.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.508 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.510 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.521 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.522 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-26T01:57:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-26T01:57:19.548 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-26T01:57:19.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-26T01:57:19.550 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.551 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.554 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-26T01:57:19.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.558 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.563 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.570 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.571 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.574 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-26T01:57:19.576 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.577 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.580 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.582 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-26T01:57:19.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.593 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-26T01:57:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-26T01:57:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T01:57:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-26T01:57:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T01:57:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-26T01:57:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-26T01:57:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-26T01:57:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-26T01:57:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-26T01:57:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25069, 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-26T01:57:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-26T01:57:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-26T01:57:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-26T01:57:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-26T01:57:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-26T01:57:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-26T01:57:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-26T01:57:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-26T01:57:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-26T01:57:19.715 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T01:57:19.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T01:57:19.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-26T01:57:19.722 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-26T01:57:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-26T01:57:19.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-26T01:57:19.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-26T01:57:19.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.740 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T01:57:19.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T01:57:19.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T01:57:19.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-26T01:57:19.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-26T01:57:19.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.753 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-26T01:57:19.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T01:57:19.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T01:57:19.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T01:57:19.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.775 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-26T01:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.784 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-26T01:57:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.836 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.838 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.839 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-26T01:57:19.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-26T01:57:19.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-26T01:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.850 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.851 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.852 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.859 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.862 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.867 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-26T01:57:19.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-26T01:57:19.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-26T01:57:19.874 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:19.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.879 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-26T01:57:19.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-26T01:57:19.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-26T01:57:19.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.893 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.903 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-26T01:57:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-26T01:57:19.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-26T01:57:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-26T01:57:19.916 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.917 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.923 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.930 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.931 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-26T01:57:19.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.935 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.942 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.947 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.955 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-26T01:57:19.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.959 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.961 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.962 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.964 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-26T01:57:19.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:19.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.972 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.973 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.974 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-26T01:57:19.982 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-26T01:57:19.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.987 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-26T01:57:19.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T01:57:19.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-26T01:57:19.992 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.995 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.005 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.008 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.017 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-26T01:57:20.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-26T01:57:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-26T01:57:20.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-26T01:57:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-26T01:57:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-26T01:57:20.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-26T01:57:20.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-26T01:57:20.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.028 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-26T01:57:20.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-26T01:57:20.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.036 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.037 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.039 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.040 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.046 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-26T01:57:20.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.054 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-26T01:57:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T01:57:20.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-26T01:57:20.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-26T01:57:20.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.074 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T01:57:20.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.079 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.080 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-04-26T01:57:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-26T01:57:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-26T01:57:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-26T01:57:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-26T01:57:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-26T01:57:20.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-26T01:57:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:20.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T01:57:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-26T01:57:20.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-26T01:57:20.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-26T01:57:20.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-26T01:57:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-26T01:57:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-26T01:57:20.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-26T01:57:20.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.176 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.177 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.180 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.186 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.189 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.191 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.192 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-26T01:57:20.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-26T01:57:20.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.204 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T01:57:20.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.228 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.229 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.231 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T01:57:20.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T01:57:20.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.257 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.259 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.261 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-26T01:57:20.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.273 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-26T01:57:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-26T01:57:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T01:57:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-26T01:57:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-26T01:57:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-26T01:57:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-26T01:57:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-26T01:57:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-26T01:57:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-26T01:57:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-26T01:57:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-26T01:57:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-26T01:57:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-26T01:57:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-26T01:57:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-26T01:57:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.451 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-26T01:57:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.472 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.475 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.476 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.479 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.483 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.489 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.506 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.510 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.517 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-26T01:57:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-26T01:57:20.536 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-26T01:57:20.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-26T01:57:20.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.540 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-26T01:57:20.541 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-26T01:57:20.542 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-26T01:57:20.543 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-26T01:57:20.598 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-26T01:57:20.598 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-26T01:57:20.598 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-26T01:57:20.598 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-26T01:57:20.598 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-26T01:57:20.599 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-26T01:57:20.600 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 91, 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-26T01:57:20.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4288675791 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-26T01:57:20.602 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1258015040 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1952261910 2024-04-26T01:57:20.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1942125167 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-26T01:57:20.604 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1269780670 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-26T01:57:20.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-26T01:57:20.606 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-26T01:57:20.606 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2471280761 2024-04-26T01:57:20.606 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.606 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.606 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:20.606 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-26T01:57:20.606 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:57:20.607 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:19 smithi169 ceph-mon[32851]: pgmap v515: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 289 KiB/s wr, 37 op/s 2024-04-26T01:57:20.607 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:19 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2094173530' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T01:57:20.608 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:18 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:20.608 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:18 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1758068639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T01:57:20.608 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:18 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2929228447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T01:57:20.608 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:19 smithi053 ceph-mon[32593]: pgmap v515: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 289 KiB/s wr, 37 op/s 2024-04-26T01:57:20.608 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:19 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2094173530' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.188+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f23f80a3d20 msgr2=0x7f23f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.188+0000 7f2419a00640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f23f80a3d20 0x7f23f80a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2404009a60 tx=0x7f2404010040 comp rx=0 tx=0).stop 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241419a690 msgr2=0x7f241419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241419a690 0x7f241419ab10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f240c00e970 tx=0x7f240c00ee30 comp rx=0 tx=0).stop 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 shutdown_connections 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f23f80a3d20 0x7f23f80a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f241419a690 0x7f241419ab10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2414109290 0x7f24141a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 --2- 172.21.15.28:0/2094173530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24141041c0 0x7f24141a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 >> 172.21.15.28:0/2094173530 conn(0x7f24140ffa20 msgr2=0x7f2414100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 shutdown_connections 2024-04-26T01:57:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:57:19.189+0000 7f2419a00640 1 -- 172.21.15.28:0/2094173530 wait complete. 2024-04-26T01:57:20.615 DEBUG:tasks.check_counter:Getting stats from a 2024-04-26T01:57:20.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-26T01:57:20.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:19 smithi028 ceph-mon[28247]: pgmap v515: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 289 KiB/s wr, 37 op/s 2024-04-26T01:57:20.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:19 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2094173530' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T01:57:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:21 smithi028 ceph-mon[28247]: pgmap v516: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 132 KiB/s wr, 21 op/s 2024-04-26T01:57:22.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:21 smithi169 ceph-mon[32851]: pgmap v516: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 132 KiB/s wr, 21 op/s 2024-04-26T01:57:22.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:21 smithi053 ceph-mon[32593]: pgmap v516: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 132 KiB/s wr, 21 op/s 2024-04-26T01:57:23.391 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1604, 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1564, 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2177040, 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 592640, 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.368281580, 2024-04-26T01:57:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.111633666, 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.163150885, 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.093288783, 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1564, 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.262982012, 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000168147 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1773840, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 211040 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 5, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 257, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1036, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 695498, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-04-26T01:57:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.048025836, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.035189099, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.006683259, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000052961, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 254, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.053398035, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000210228 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 694800 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2547, 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2548, 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 7285222, 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 809635, 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-26T01:57:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.364424329, 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.088049429, 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.121715234, 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.129818466, 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2546, 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.320893273, 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000126038 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-04-26T01:57:23.399 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-26T01:57:23.399 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-26T01:57:23.399 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6129, 2024-04-26T01:57:23.399 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.294354586, 2024-04-26T01:57:23.399 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000048026 2024-04-26T01:57:23.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.399 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-04-26T01:57:23.400 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-26T01:57:23.400 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-26T01:57:23.400 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.400 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.400 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.401 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-04-26T01:57:23.401 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-04-26T01:57:23.401 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-04-26T01:57:23.401 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-04-26T01:57:23.401 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.401 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.401 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.402 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-04-26T01:57:23.402 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-04-26T01:57:23.402 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-04-26T01:57:23.402 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-04-26T01:57:23.402 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 529, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 72, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 457, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 426, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 7, 2024-04-26T01:57:23.403 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-04-26T01:57:23.404 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-04-26T01:57:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-04-26T01:57:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-04-26T01:57:23.407 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-04-26T01:57:23.408 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 3022, 2024-04-26T01:57:23.409 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 5, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 10832723, 2024-04-26T01:57:23.410 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 10824013, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 3022 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 171, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 1124, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 953, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 18, 2024-04-26T01:57:23.411 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 43, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 25, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 529, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 1385, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 856, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 53348, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215416 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.412 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-04-26T01:57:23.413 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.414 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.416 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.417 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-04-26T01:57:23.418 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.418 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.418 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.418 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-04-26T01:57:23.419 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.419 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.419 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.419 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-04-26T01:57:23.420 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.420 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.420 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.420 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-04-26T01:57:23.420 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.420 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.421 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.421 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-04-26T01:57:23.421 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.421 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.421 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-04-26T01:57:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-04-26T01:57:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-04-26T01:57:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-04-26T01:57:23.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 4, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 3, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 3, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 5223, 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-04-26T01:57:23.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:23.430 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:23.431 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:23.432 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 190665, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 280, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 8032, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 167, 2024-04-26T01:57:23.433 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 907940, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 29114, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-04-26T01:57:23.434 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 3748, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 3748, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3748, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.115828530, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001631757 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-04-26T01:57:23.435 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3748, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 3749 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 3748, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 3748, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 2303, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 1443, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-04-26T01:57:23.436 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-04-26T01:57:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-04-26T01:57:23.438 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-04-26T01:57:23.439 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 113, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 25, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 4, 2024-04-26T01:57:23.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 5, 2024-04-26T01:57:23.441 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 1, 2024-04-26T01:57:23.441 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-04-26T01:57:23.441 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-04-26T01:57:23.441 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-04-26T01:57:23.441 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-04-26T01:57:23.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.441 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-04-26T01:57:23.442 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-04-26T01:57:23.442 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-04-26T01:57:23.442 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-04-26T01:57:23.442 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-04-26T01:57:23.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.442 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-04-26T01:57:23.442 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-04-26T01:57:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-04-26T01:57:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-04-26T01:57:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T01:57:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-04-26T01:57:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-04-26T01:57:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-04-26T01:57:23.443 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 4152, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 9066261, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 4152, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-26T01:57:23.444 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4152, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 9066261, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:23.445 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 3748, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 6638424, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 3748, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 6638424, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.446 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 3748, 2024-04-26T01:57:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 3748, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 3748, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 3748, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55e69b4b6780": { 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-26T01:57:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-26T01:57:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-26T01:57:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:23.451 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:57:24.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:23 smithi028 ceph-mon[28247]: pgmap v517: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 90 KiB/s wr, 14 op/s 2024-04-26T01:57:24.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:23 smithi169 ceph-mon[32851]: pgmap v517: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 90 KiB/s wr, 14 op/s 2024-04-26T01:57:24.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:23 smithi053 ceph-mon[32593]: pgmap v517: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 90 KiB/s wr, 14 op/s 2024-04-26T01:57:24.388 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-26T01:57:24.388 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-26T01:57:24.388 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-04-26T01:57:24.388 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-26T01:57:24.389 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T01:57:24.389 DEBUG:tasks.check_counter:Getting stats from d 2024-04-26T01:57:24.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-26T01:57:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:26.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:25 smithi028 ceph-mon[28247]: pgmap v518: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 89 KiB/s wr, 14 op/s 2024-04-26T01:57:26.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:25 smithi169 ceph-mon[32851]: pgmap v518: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 89 KiB/s wr, 14 op/s 2024-04-26T01:57:26.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:25 smithi053 ceph-mon[32593]: pgmap v518: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 89 KiB/s wr, 14 op/s 2024-04-26T01:57:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:57:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-26T01:57:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-26T01:57:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.004105976, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001136102, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 3, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 261, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 480, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 280176, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.047252999, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.035308923, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.004668208, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 258, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.049048494, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000190110 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T01:57:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 280176 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 355, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 342, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1629616, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 210624, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.143432112, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.054782227, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.103330593, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.015573063, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 342, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.055705477, 2024-04-26T01:57:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000162881 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1629616, 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 210624 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-26T01:57:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149745, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-26T01:57:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 354, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1577140, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 354, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 354, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1577140, 2024-04-26T01:57:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:27.053 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:57:27.881 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-26T01:57:27.881 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-26T01:57:27.881 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-26T01:57:27.881 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-26T01:57:27.881 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-26T01:57:27.881 DEBUG:tasks.check_counter:Getting stats from g 2024-04-26T01:57:27.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-26T01:57:28.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:27 smithi028 ceph-mon[28247]: pgmap v519: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 77 KiB/s wr, 16 op/s 2024-04-26T01:57:28.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:27 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:28.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:27 smithi169 ceph-mon[32851]: pgmap v519: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 77 KiB/s wr, 16 op/s 2024-04-26T01:57:28.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:27 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:28.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:27 smithi053 ceph-mon[32593]: pgmap v519: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 77 KiB/s wr, 16 op/s 2024-04-26T01:57:28.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:27 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:28.354 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:30.023 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:29 smithi053 ceph-mon[32593]: pgmap v520: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 49 KiB/s wr, 12 op/s 2024-04-26T01:57:30.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:29 smithi028 ceph-mon[28247]: pgmap v520: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 49 KiB/s wr, 12 op/s 2024-04-26T01:57:30.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:29 smithi169 ceph-mon[32851]: pgmap v520: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 49 KiB/s wr, 12 op/s 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1008, 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1010, 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 4464612, 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 320608, 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.202762681, 2024-04-26T01:57:30.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.059568085, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.066491067, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.052903482, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1008, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.104773937, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000103942 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T01:57:30.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1489, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1448, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 6206142, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 556741, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.349931016, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.126174573, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.140788241, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.080788684, 2024-04-26T01:57:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1448, 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.209371868, 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000144593 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1779920, 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 213296 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:30.663 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1544, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1798, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 488399, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 1100886, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.371887143, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.174686867, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.072368416, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.092094973, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1795, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.320140711, 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000178351 2024-04-26T01:57:30.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 610848 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-04-26T01:57:30.665 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6179, 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.295672781, 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000047851 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.666 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-04-26T01:57:30.667 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 1172, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 128, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 1044, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 988, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-04-26T01:57:30.668 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 7, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-04-26T01:57:30.669 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T01:57:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-04-26T01:57:30.671 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-04-26T01:57:30.672 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-04-26T01:57:30.673 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 1837, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 4, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-04-26T01:57:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 12543432, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 12535606, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 1837 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1135, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 1513, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 378, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 43, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 54, 2024-04-26T01:57:30.675 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 11, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 1172, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 1620, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 448, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 55448, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215416 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-04-26T01:57:30.676 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.677 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.678 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.679 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-04-26T01:57:30.680 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.681 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.681 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.681 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-04-26T01:57:30.681 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-04-26T01:57:30.683 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-04-26T01:57:30.686 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.687 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.688 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 2, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 0, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 2401, 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-04-26T01:57:30.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:30.691 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 243288, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 1228, 2024-04-26T01:57:30.692 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 8032, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 167, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-26T01:57:30.693 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 3519117, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 32606, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 3629, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-04-26T01:57:30.694 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 3629, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3629, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 11.579675518, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.003190872 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3629, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 3630 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 3629, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 3629, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-04-26T01:57:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 2185, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 1442, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-04-26T01:57:30.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-04-26T01:57:30.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-04-26T01:57:30.698 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 113, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 25, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 6, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 7, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 1, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.699 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-04-26T01:57:30.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 4037, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 10772261, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 4037, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-26T01:57:30.701 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4037, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 10772261, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-26T01:57:30.702 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 3629, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 8356978, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 3629, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 8356978, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 3629, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 3629, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 3629, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 3629, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.704 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-26T01:57:30.705 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x564b0db90780": { 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:30.706 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:30.707 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:57:31.663 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-26T01:57:31.663 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-26T01:57:31.663 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-04-26T01:57:31.663 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-26T01:57:31.663 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T01:57:31.663 DEBUG:tasks.check_counter:Getting stats from j 2024-04-26T01:57:31.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-26T01:57:32.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:31 smithi028 ceph-mon[28247]: pgmap v521: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 69 KiB/s wr, 16 op/s 2024-04-26T01:57:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:32.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:31 smithi169 ceph-mon[32851]: pgmap v521: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 69 KiB/s wr, 16 op/s 2024-04-26T01:57:32.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:31 smithi053 ceph-mon[32593]: pgmap v521: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 69 KiB/s wr, 16 op/s 2024-04-26T01:57:33.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:32 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:33.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:32 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:33.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:32 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:34.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:33 smithi028 ceph-mon[28247]: pgmap v522: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 54 KiB/s wr, 12 op/s 2024-04-26T01:57:34.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:33 smithi028 ceph-mon[28247]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:34.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:33 smithi169 ceph-mon[32851]: pgmap v522: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 54 KiB/s wr, 12 op/s 2024-04-26T01:57:34.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:33 smithi169 ceph-mon[32851]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:34.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:33 smithi053 ceph-mon[32593]: pgmap v522: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 54 KiB/s wr, 12 op/s 2024-04-26T01:57:34.220 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:33 smithi053 ceph-mon[32593]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:34.657 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-26T01:57:34.657 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:34.657 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-04-26T01:57:34.657 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-04-26T01:57:34.657 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2944, 2024-04-26T01:57:34.657 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.004558048, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000062876, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001264337, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001654895, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000551631 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.658 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 263, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 202272, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:34.659 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.049500208, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.036545963, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.006267206, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 260, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.039918536, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000153532 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T01:57:34.660 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 202272 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 355, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 344, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1627760, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 212736, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.136078674, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.050670186, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.092535965, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.015046974, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 343, 2024-04-26T01:57:34.661 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.072136394, 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000210310 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1627760, 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 212736 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-26T01:57:34.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:34.664 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149745, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-26T01:57:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-26T01:57:34.666 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 357, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1577633, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 357, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-26T01:57:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 357, 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1577633, 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-26T01:57:34.668 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-26T01:57:35.619 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-26T01:57:35.619 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-26T01:57:35.619 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-26T01:57:35.619 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-26T01:57:35.619 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-26T01:57:35.619 DEBUG:tasks.check_counter:Getting stats from b 2024-04-26T01:57:35.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-26T01:57:35.878 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:57:35.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:35 smithi053 ceph-mon[32593]: pgmap v523: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T01:57:36.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:35 smithi169 ceph-mon[32851]: pgmap v523: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T01:57:36.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:35 smithi028 ceph-mon[28247]: pgmap v523: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T01:57:37.710 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-26T01:57:37.710 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:37.710 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 6361, 2024-04-26T01:57:37.710 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 6549, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 2249040, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 9313234, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 16, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 4, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.359500577, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.137066441, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.200839496, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.037658381, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 6545, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.234401769, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000035813 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.711 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 10073, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.001522868, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000151 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 4400, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 5584 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 11105, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 14097, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 6024732, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 6894854, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 10, 2024-04-26T01:57:37.712 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 6, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.770981057, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.327008462, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.557292113, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.017656798, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 14098, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.612674444, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000043458 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 20096, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.003797056, 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000188 2024-04-26T01:57:37.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 3724128, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 4596480 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 441, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 697, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 154695, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 956471, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 12, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 6, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.057244907, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.022091865, 2024-04-26T01:57:37.714 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.016873924, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.012667534, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 695, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.092071175, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000132476 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 19, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000003012, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000158 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 6064, 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 704608 2024-04-26T01:57:37.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "cct": { 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "total_workers": 1, 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "unhealthy_workers": 0 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "finisher-MDSRank": { 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "queue_len": 0, 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "complete_latency": { 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2234, 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 1.512580767, 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000677072 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "finisher-PurgeQueue": { 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "queue_len": 0, 2024-04-26T01:57:37.716 INFO:teuthology.orchestra.run.smithi053.stdout: "complete_latency": { 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 690, 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.030097213, 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000043619 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "request": 2481, 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "reply": 2404, 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "reply_latency": { 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2404, 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 109.249717172, 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.045444973 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.717 INFO:teuthology.orchestra.run.smithi053.stdout: "slow_reply": 0, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "forward": 77, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_fetch_complete": 8, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_fetch_keys": 0, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_commit": 11, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_split": 0, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_merge": 0, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes": 584, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_top": 14, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_bottom": 464, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_pin_tail": 106, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_pinned": 564, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_expired": 962, 2024-04-26T01:57:37.718 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_with_caps": 211, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": 211, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "subtrees": 17, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse": 6016, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_hit": 5154, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_forward": 76, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_discover": 27, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_dir_fetch": 4, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_remote_ino": 1, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_lock": 177, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "load_cent": 904, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "q": 0, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "exported": 20, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "exported_inodes": 866, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "imported": 28, 2024-04-26T01:57:37.719 INFO:teuthology.orchestra.run.smithi053.stdout: "imported_inodes": 1045, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_dir_fetch": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_backtrace_fetch": 1, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_peer_discover": 21, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rfiles": 1026, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rbytes": 1646698034, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rsnaps": 6, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_set_tag": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_inotable_repaired": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dir_inodes": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_file_inodes": 0, 2024-04-26T01:57:37.720 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_inode_file_caps": 12, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_revoke": 1570, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_grant": 2266, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_caps": 2431, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_caps_dirty": 508, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_cap_release": 164, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "process_request_cap_release": 1074 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_cache": { 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_update": 0, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_update_receipt": 0, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_try_discover": 0, 2024-04-26T01:57:37.721 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_send_discover": 46, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_handle_discover": 49, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays": 15, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays_delayed": 0, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays_enqueuing": 0, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_created": 272, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_enqueued": 199, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_reintegrated": 67, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_migrated": 2, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_processing": 0, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_enqueued": 0, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_prioritized": 0, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "recovery_started": 0, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "recovery_completed": 0, 2024-04-26T01:57:37.722 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_quiesce_path": 4, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_quiesce_inode": 1316, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_exportdir": 20, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_flush": 0, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_fragmentdir": 0, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_fragstats": 0, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_inodestats": 0 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_log": { 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "evlrg": 0, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "evadd": 3064, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "evex": 0, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "evtrm": 0, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "ev": 3064, 2024-04-26T01:57:37.723 INFO:teuthology.orchestra.run.smithi053.stdout: "evexg": 0, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "evexd": 0, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "segadd": 5, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "segex": 0, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "segtrm": 0, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "seg": 5, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "segmjr": 2, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "segexg": 0, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "segexd": 0, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "expos": 4194304, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "wrpos": 11141849, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "rdpos": 0, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "jlat": { 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1156, 2024-04-26T01:57:37.724 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 5.346134058, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.004624683 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "replayed": 0 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_mem": { 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "ino": 449, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "ino+": 1485, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "ino-": 1036, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "dir": 53, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "dir+": 71, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "dir-": 18, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "dn": 593, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "dn+": 1804, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "dn-": 1211, 2024-04-26T01:57:37.725 INFO:teuthology.orchestra.run.smithi053.stdout: "cap": 211, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "cap+": 931, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "cap-": 720, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "rss": 63564, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "heap": 223608 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_server": { 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "dispatch_client_request": 6356, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "dispatch_server_request": 301, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_request": 2481, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_session": 170, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_peer_request": 2083, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "req_create_latency": { 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 200, 2024-04-26T01:57:37.726 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 23.783980378, 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.118919901 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getattr_latency": { 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 953, 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 51.935793084, 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.054497159 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getfilelock_latency": { 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "req_link_latency": { 2024-04-26T01:57:37.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 69, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.293930245, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.004259858 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookup_latency": { 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 572, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 16.730685035, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.029249449 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookuphash_latency": { 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupino_latency": { 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.728 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupname_latency": { 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupparent_latency": { 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.729 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupsnap_latency": { 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lssnap_latency": { 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mkdir_latency": { 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mknod_latency": { 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mksnap_latency": { 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "req_open_latency": { 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 231, 2024-04-26T01:57:37.731 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.136032312, 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000588884 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "req_readdir_latency": { 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 29, 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 4.934172034, 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.170143863 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rename_latency": { 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 264, 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 11.196867811, 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.042412378 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "req_renamesnap_latency": { 2024-04-26T01:57:37.732 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "req_snapdiff_latency": { 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmdir_latency": { 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmsnap_latency": { 2024-04-26T01:57:37.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmxattr_latency": { 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setattr_latency": { 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.734 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setdirlayout_latency": { 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setfilelock_latency": { 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setlayout_latency": { 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.735 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setxattr_latency": { 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "req_symlink_latency": { 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "req_unlink_latency": { 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 86, 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.238256273, 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.002770421 2024-04-26T01:57:37.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "cap_revoke_eviction": 0, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "cap_acquisition_throttle": 0, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getvxattr_latency": { 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_sessions": { 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "session_count": 1, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "session_add": 4, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "session_remove": 3, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "sessions_open": 1, 2024-04-26T01:57:37.737 INFO:teuthology.orchestra.run.smithi053.stdout: "sessions_stale": 0, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "total_load": 343, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "average_load": 343, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "avg_session_uptime": 997, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "mdthresh_evicted": 0 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "mempool": { 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:37.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:37.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_items": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_bytes": 773082, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_items": 672, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_bytes": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_items": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_bytes": 8032, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_items": 167, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_items": 0, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_bytes": 1806396, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_items": 40848, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_items": 0, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_items": 0 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "objecter": { 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "op_active": 0, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "op_laggy": 0, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "op_send": 1881, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "op_send_bytes": 7068215, 2024-04-26T01:57:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: "op_resend": 33, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "op_reply": 1881, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "op_latency": { 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1881, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 10.855772425, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.005771277 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "op_inflight": 0, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "oplen_avg": { 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1881, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 2116 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "op": 1881, 2024-04-26T01:57:37.743 INFO:teuthology.orchestra.run.smithi053.stdout: "op_r": 93, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "op_w": 1788, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "op_rmw": 0, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "op_pg": 0, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_stat": 0, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_create": 10, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_read": 84, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_write": 1240, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_writefull": 135, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_writesame": 0, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_append": 0, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_zero": 0, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_truncate": 0, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_delete": 212, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_mapext": 0, 2024-04-26T01:57:37.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_sparse_read": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_clonerange": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_getxattr": 3, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_setxattr": 20, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_cmpxattr": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_rmxattr": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_resetxattrs": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_call": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_watch": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_notify": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_pgls": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_pgls_filter": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_other": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_active": 0, 2024-04-26T01:57:37.745 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_send": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_resend": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_ping": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_active": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_send": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_resend": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_active": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_send": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_resend": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_active": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_send": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_resend": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "command_active": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "command_send": 0, 2024-04-26T01:57:37.746 INFO:teuthology.orchestra.run.smithi053.stdout: "command_resend": 0, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "map_epoch": 113, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "map_full": 0, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "map_inc": 25, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_sessions": 12, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_session_open": 12, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_session_close": 0, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_laggy": 0, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_wr": 300, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_rd": 16, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_del": 96 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "oft": { 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_objs": 1, 2024-04-26T01:57:37.747 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_kv_pairs": 0, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_updates": 0, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_removes": 0 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "osd-slow-ops": { 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "slow_ops_count": 1460 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "purge_queue": { 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_ops": 0, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_ops_high_water": 10, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing": 0, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_high_water": 10, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executed_ops": 199, 2024-04-26T01:57:37.748 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executed": 199, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_item_in_journal": 0 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 17903, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 6563777, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 17903, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 17903, 2024-04-26T01:57:37.749 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 6563777, 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-objecter_bytes": { 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-04-26T01:57:37.750 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 1881, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 7088882, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 1881, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 7088882, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.751 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-objecter_ops": { 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1024, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 1881, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 1881, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 1881, 2024-04-26T01:57:37.752 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 1881, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-write_buf_throttle": { 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 3758096384, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 199, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 20795, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 199, 2024-04-26T01:57:37.753 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 84, 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 20795, 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-write_buf_throttle-0x564f715e2500": { 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 2296, 2024-04-26T01:57:37.754 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 3758096384, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 3064, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 6947545, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 3064, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 1156, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 6945249, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.755 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:37.756 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-26T01:57:37.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:37 smithi053 ceph-mon[32593]: pgmap v524: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 62 KiB/s wr, 15 op/s 2024-04-26T01:57:38.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:37 smithi169 ceph-mon[32851]: pgmap v524: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 62 KiB/s wr, 15 op/s 2024-04-26T01:57:38.276 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:37 smithi028 ceph-mon[28247]: pgmap v524: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 62 KiB/s wr, 15 op/s 2024-04-26T01:57:38.538 INFO:tasks.check_counter:Daemon mds.b mds.exported=20 2024-04-26T01:57:38.538 INFO:tasks.check_counter:Daemon mds.b mds.imported=28 2024-04-26T01:57:38.538 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-26T01:57:38.538 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-04-26T01:57:38.538 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T01:57:38.538 DEBUG:tasks.check_counter:Getting stats from e 2024-04-26T01:57:38.538 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-26T01:57:38.789 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:57:40.127 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:39 smithi053 ceph-mon[32593]: pgmap v525: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 10 op/s 2024-04-26T01:57:40.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:39 smithi169 ceph-mon[32851]: pgmap v525: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 10 op/s 2024-04-26T01:57:40.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:39 smithi028 ceph-mon[28247]: pgmap v525: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 10 op/s 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 354, 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 343, 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 1627488, 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 212096, 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 4, 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 0, 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.062094479, 2024-04-26T01:57:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.018607497, 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.041427022, 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.008412071, 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 342, 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.017596424, 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000051451 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 1627488, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 212096 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 6, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 265, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 3264, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 280976, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 6, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.016618976, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.011060143, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.002333474, 2024-04-26T01:57:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 263, 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.023852847, 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000090695 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 280976 2024-04-26T01:57:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 0, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 2, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 0, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 0, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 2, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 1, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.000444511, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.000376272, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "cct": { 2024-04-26T01:57:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "total_workers": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "unhealthy_workers": 0 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "mempool": { 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_items": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_bytes": 148714, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_items": 30, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_bytes": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_items": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_bytes": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_items": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_items": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_bytes": 928, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_items": 2, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_items": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_items": 0 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 356, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 1577503, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 356, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 356, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 1577503, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:40.733 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:40.733 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:40.733 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-26T01:57:41.410 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-04-26T01:57:41.410 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-04-26T01:57:41.410 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-26T01:57:41.410 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-26T01:57:41.410 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-26T01:57:41.411 DEBUG:tasks.check_counter:Getting stats from h 2024-04-26T01:57:41.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-26T01:57:41.665 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:57:42.038 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:41 smithi053 ceph-mon[32593]: pgmap v526: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 13 op/s 2024-04-26T01:57:42.038 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:41 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:42.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:41 smithi169 ceph-mon[32851]: pgmap v526: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 13 op/s 2024-04-26T01:57:42.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:41 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:42.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:41 smithi028 ceph-mon[28247]: pgmap v526: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 13 op/s 2024-04-26T01:57:42.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:41 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:43.086 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:42 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:43.086 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:42 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:43.086 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:42 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:57:43.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:42 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:43.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:42 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:43.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:42 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:57:43.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:42 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:43.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:42 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:43.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:42 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 0, 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 2, 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 0, 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 0, 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 4, 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 0, 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.001272047, 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T01:57:43.355 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.000220493, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T01:57:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 353, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 342, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 1627216, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 211456, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 2, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 1, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.059094997, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.018995414, 2024-04-26T01:57:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.038690357, 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.008527664, 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 341, 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.015720775, 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000046101 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 1627216, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 211456 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 8, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 265, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 3584, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 359232, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 7, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.015809342, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.010560185, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.002333587, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:43.359 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:43.360 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 263, 2024-04-26T01:57:43.360 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.043921232, 2024-04-26T01:57:43.360 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000167000 2024-04-26T01:57:43.360 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.361 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:43.361 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:43.361 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:43.361 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:43.361 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.362 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 3584, 2024-04-26T01:57:43.362 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 359232 2024-04-26T01:57:43.362 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.362 INFO:teuthology.orchestra.run.smithi053.stdout: "cct": { 2024-04-26T01:57:43.362 INFO:teuthology.orchestra.run.smithi053.stdout: "total_workers": 0, 2024-04-26T01:57:43.362 INFO:teuthology.orchestra.run.smithi053.stdout: "unhealthy_workers": 0 2024-04-26T01:57:43.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.363 INFO:teuthology.orchestra.run.smithi053.stdout: "mempool": { 2024-04-26T01:57:43.363 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:43.363 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:43.363 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:43.363 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:43.363 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:43.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:43.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:43.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:43.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:43.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:43.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:43.365 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:43.365 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:43.365 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:43.365 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:43.365 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:43.365 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:43.365 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:43.366 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:43.366 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:43.366 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:43.366 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:43.366 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:43.366 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:43.367 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:43.367 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:43.367 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:43.367 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:43.367 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:43.367 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:43.368 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:43.368 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_items": 0, 2024-04-26T01:57:43.368 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:43.368 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:43.368 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:43.368 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:43.368 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_bytes": 148692, 2024-04-26T01:57:43.369 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_items": 30, 2024-04-26T01:57:43.369 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:43.369 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:43.369 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_bytes": 0, 2024-04-26T01:57:43.369 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_items": 0, 2024-04-26T01:57:43.369 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:43.369 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:43.370 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:43.370 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:43.370 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_bytes": 0, 2024-04-26T01:57:43.370 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_items": 0, 2024-04-26T01:57:43.370 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:43.370 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:43.371 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:43.371 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_items": 0, 2024-04-26T01:57:43.371 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_bytes": 928, 2024-04-26T01:57:43.371 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_items": 2, 2024-04-26T01:57:43.371 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:43.371 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_items": 0, 2024-04-26T01:57:43.372 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:43.372 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_items": 0 2024-04-26T01:57:43.372 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:43.372 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:43.372 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-04-26T01:57:43.372 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-04-26T01:57:43.372 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-04-26T01:57:43.373 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 357, 2024-04-26T01:57:43.373 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 1577427, 2024-04-26T01:57:43.373 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:43.373 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 357, 2024-04-26T01:57:43.373 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-04-26T01:57:43.373 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-04-26T01:57:43.374 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 357, 2024-04-26T01:57:43.374 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 1577427, 2024-04-26T01:57:43.374 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-04-26T01:57:43.374 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:43.374 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:43.374 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:43.375 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:43.375 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:43.375 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-26T01:57:44.088 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:43 smithi053 ceph-mon[32593]: pgmap v527: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-26T01:57:44.116 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.h 2024-04-26T01:57:44.117 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.h 2024-04-26T01:57:44.117 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-04-26T01:57:44.117 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-04-26T01:57:44.117 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-04-26T01:57:44.117 DEBUG:tasks.check_counter:Getting stats from k 2024-04-26T01:57:44.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-26T01:57:44.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:43 smithi169 ceph-mon[32851]: pgmap v527: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-26T01:57:44.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:43 smithi028 ceph-mon[28247]: pgmap v527: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-26T01:57:44.364 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:57:45.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:44 smithi169 ceph-mon[32851]: pgmap v528: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 9 op/s 2024-04-26T01:57:45.196 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:44 smithi053 ceph-mon[32593]: pgmap v528: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 9 op/s 2024-04-26T01:57:45.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:44 smithi028 ceph-mon[28247]: pgmap v528: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 9 op/s 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 0, 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 2, 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 0, 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 0, 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 4, 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 0, 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.001026119, 2024-04-26T01:57:46.205 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.000100408, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T01:57:46.206 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 363, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 350, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 1631792, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 215744, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 2, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 1, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.060199059, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.019474711, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.040205717, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.008256265, 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:46.207 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 350, 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.017358036, 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000049594 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 1631792, 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 215744 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.208 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 2, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 266, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 320, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 202800, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 5, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.014121596, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.010045298, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.001225659, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 263, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.024814435, 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000094351 2024-04-26T01:57:46.209 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 202800 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "cct": { 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "total_workers": 0, 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: "unhealthy_workers": 0 2024-04-26T01:57:46.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "mempool": { 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:46.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:46.212 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_items": 0, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_bytes": 148692, 2024-04-26T01:57:46.213 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_items": 30, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_bytes": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_items": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_bytes": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_items": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:46.214 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_items": 0, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_bytes": 928, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_items": 2, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_items": 0, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_items": 0 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 361, 2024-04-26T01:57:46.215 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 1578153, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 361, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 361, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 1578153, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-26T01:57:46.216 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-26T01:57:46.836 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-26T01:57:46.837 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-26T01:57:46.837 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-26T01:57:46.837 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-26T01:57:46.837 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-26T01:57:46.837 DEBUG:tasks.check_counter:Getting stats from c 2024-04-26T01:57:46.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-26T01:57:47.089 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:57:47.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:47 smithi169 ceph-mon[32851]: pgmap v529: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T01:57:47.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:47 smithi053 ceph-mon[32593]: pgmap v529: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T01:57:47.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:47 smithi028 ceph-mon[28247]: pgmap v529: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T01:57:48.889 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-04-26T01:57:48.889 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:48.889 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 10213, 2024-04-26T01:57:48.889 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 9771, 2024-04-26T01:57:48.889 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 4179284, 2024-04-26T01:57:48.889 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 8166182, 2024-04-26T01:57:48.889 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 10, 2024-04-26T01:57:48.889 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 4, 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.573186089, 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.253699306, 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.360556513, 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.028284813, 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 9773, 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.289884017, 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000029661 2024-04-26T01:57:48.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 17535, 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.002788127, 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000159 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 842, 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 802, 2024-04-26T01:57:48.891 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 1935760, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 4559814, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 10, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 6, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.105402747, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.037787056, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.053458371, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.021245404, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 802, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.050354592, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000062786 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:48.892 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 2, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000176, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000088 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 1803440, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 224256 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 3068, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 5620, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 1193002, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 3776546, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 15, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 7, 2024-04-26T01:57:48.893 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.257410645, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.126698474, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.155948721, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.001044796, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 5613, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.183391628, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000032672 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 5581, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000975504, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000174 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.894 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 1164368, 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 3587056 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 1, 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-MDSRank": { 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1582, 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 1.165681510, 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000736840 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-PurgeQueue": { 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 71, 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.003217582, 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000045318 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "mds": { 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "request": 697, 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "reply": 638, 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "reply_latency": { 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 638, 2024-04-26T01:57:48.896 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 9.448210358, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.014809107 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_reply": 0, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "forward": 59, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_complete": 3, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_keys": 0, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_commit": 11, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_split": 0, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_merge": 0, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes": 812, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_top": 25, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_bottom": 710, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pin_tail": 77, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pinned": 777, 2024-04-26T01:57:48.897 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_expired": 890, 2024-04-26T01:57:48.898 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_with_caps": 674, 2024-04-26T01:57:48.898 INFO:teuthology.orchestra.run.smithi169.stdout: "caps": 674, 2024-04-26T01:57:48.898 INFO:teuthology.orchestra.run.smithi169.stdout: "subtrees": 17, 2024-04-26T01:57:48.898 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse": 1435, 2024-04-26T01:57:48.898 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_hit": 1187, 2024-04-26T01:57:48.898 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_forward": 54, 2024-04-26T01:57:48.898 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_discover": 40, 2024-04-26T01:57:48.898 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_dir_fetch": 1, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_remote_ino": 0, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_lock": 11, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "load_cent": 1044, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "q": 0, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "exported": 16, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "exported_inodes": 775, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "imported": 38, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "imported_inodes": 1299, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_dir_fetch": 0, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_backtrace_fetch": 0, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_peer_discover": 0, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rfiles": 1026, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rbytes": 1646698034, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rsnaps": 6, 2024-04-26T01:57:48.899 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_set_tag": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_inotable_repaired": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_inodes": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_file_inodes": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_inode_file_caps": 6, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_revoke": 1613, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_grant": 1385, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps": 2106, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps_dirty": 386, 2024-04-26T01:57:48.900 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_cap_release": 26, 2024-04-26T01:57:48.901 INFO:teuthology.orchestra.run.smithi169.stdout: "process_request_cap_release": 126 2024-04-26T01:57:48.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.901 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache": { 2024-04-26T01:57:48.901 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update": 0, 2024-04-26T01:57:48.901 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update_receipt": 0, 2024-04-26T01:57:48.901 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_try_discover": 0, 2024-04-26T01:57:48.901 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_send_discover": 57, 2024-04-26T01:57:48.901 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_handle_discover": 24, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays": 6, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_delayed": 0, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_enqueuing": 0, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_created": 32, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_enqueued": 18, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_reintegrated": 3, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_migrated": 6, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_processing": 0, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_enqueued": 0, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_prioritized": 0, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_started": 0, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_completed": 0, 2024-04-26T01:57:48.902 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_quiesce_path": 4, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_quiesce_inode": 2504, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_exportdir": 16, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_flush": 0, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragmentdir": 0, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragstats": 0, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_inodestats": 0 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log": { 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "evlrg": 0, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "evadd": 1856, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "evex": 0, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "evtrm": 0, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "ev": 1856, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "evexg": 0, 2024-04-26T01:57:48.903 INFO:teuthology.orchestra.run.smithi169.stdout: "evexd": 0, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "segadd": 5, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "segex": 0, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "segtrm": 0, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "seg": 5, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "segmjr": 2, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "segexg": 0, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "segexd": 0, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "expos": 4194304, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "wrpos": 12796083, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "rdpos": 0, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "jlat": { 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1122, 2024-04-26T01:57:48.904 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 12.399899825, 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.011051604 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "replayed": 0 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_mem": { 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "ino": 807, 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "ino+": 1578, 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "ino-": 771, 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "dir": 67, 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "dir+": 72, 2024-04-26T01:57:48.905 INFO:teuthology.orchestra.run.smithi169.stdout: "dir-": 5, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "dn": 813, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "dn+": 1727, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "dn-": 914, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "cap": 674, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "cap+": 1561, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "cap-": 887, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "rss": 66812, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "heap": 223608 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_server": { 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_client_request": 981, 2024-04-26T01:57:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_server_request": 1088, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_request": 697, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_session": 68, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_peer_request": 2078, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "req_create_latency": { 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 21, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.050647573, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.002411789 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getattr_latency": { 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 37, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 2.114681626, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.057153557 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.907 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getfilelock_latency": { 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: "req_link_latency": { 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 13, 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.119930622, 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.009225432 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.908 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookup_latency": { 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 434, 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 3.020737412, 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.006960224 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookuphash_latency": { 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupino_latency": { 2024-04-26T01:57:48.909 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000337885, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000337885 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupname_latency": { 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupparent_latency": { 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.910 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupsnap_latency": { 2024-04-26T01:57:48.911 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.911 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.911 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.911 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lssnap_latency": { 2024-04-26T01:57:48.911 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.912 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.912 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.912 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mkdir_latency": { 2024-04-26T01:57:48.912 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.912 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.912 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mknod_latency": { 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mksnap_latency": { 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "req_open_latency": { 2024-04-26T01:57:48.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 60, 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.045374415, 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000756240 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "req_readdir_latency": { 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 36, 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.032891629, 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000913656 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rename_latency": { 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 6, 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 3.966127249, 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.661021208 2024-04-26T01:57:48.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "req_renamesnap_latency": { 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "req_snapdiff_latency": { 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmdir_latency": { 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.915 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmsnap_latency": { 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmxattr_latency": { 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.916 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setattr_latency": { 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setdirlayout_latency": { 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setfilelock_latency": { 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setlayout_latency": { 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setxattr_latency": { 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.918 INFO:teuthology.orchestra.run.smithi169.stdout: "req_symlink_latency": { 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "req_unlink_latency": { 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 30, 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.097481947, 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.003249398 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_revoke_eviction": 0, 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_acquisition_throttle": 0, 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getvxattr_latency": { 2024-04-26T01:57:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_sessions": { 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "session_count": 2, 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "session_add": 2, 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "session_remove": 0, 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_open": 2, 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_stale": 0, 2024-04-26T01:57:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: "total_load": 162, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "average_load": 81, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "avg_session_uptime": 1007, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "mdthresh_evicted": 0 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:48.921 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:48.922 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:48.923 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 723653, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 968, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:48.924 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 8032, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 167, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 2905901, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 32992, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-04-26T01:57:48.925 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter": { 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "op_active": 0, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "op_laggy": 0, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send": 1390, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send_bytes": 8684864, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "op_resend": 33, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "op_reply": 1390, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "op_latency": { 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1390, 2024-04-26T01:57:48.926 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 16.783641066, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.012074561 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "op_inflight": 0, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "oplen_avg": { 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1390, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 1554 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "op": 1390, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "op_r": 11, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "op_w": 1379, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "op_rmw": 0, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "op_pg": 0, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_stat": 0, 2024-04-26T01:57:48.927 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_create": 10, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_read": 8, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_write": 1130, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writefull": 72, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writesame": 0, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_append": 0, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_zero": 0, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_truncate": 0, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_delete": 32, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_mapext": 0, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_sparse_read": 0, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_clonerange": 0, 2024-04-26T01:57:48.928 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_getxattr": 1, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_setxattr": 20, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_cmpxattr": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_rmxattr": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_resetxattrs": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_call": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_watch": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_notify": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls_filter": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_other": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_active": 0, 2024-04-26T01:57:48.929 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_send": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_resend": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_ping": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_active": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_send": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_resend": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_active": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_send": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_resend": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_active": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_send": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_resend": 0, 2024-04-26T01:57:48.930 INFO:teuthology.orchestra.run.smithi169.stdout: "command_active": 0, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "command_send": 0, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "command_resend": 0, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "map_epoch": 113, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "map_full": 0, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "map_inc": 25, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_sessions": 12, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_open": 12, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_close": 0, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_laggy": 0, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_wr": 214, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_rd": 6, 2024-04-26T01:57:48.931 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_del": 61 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "oft": { 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_objs": 1, 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_kv_pairs": 0, 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_updates": 0, 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_removes": 0 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "osd-slow-ops": { 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_ops_count": 2592 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "purge_queue": { 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops": 0, 2024-04-26T01:57:48.932 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops_high_water": 7, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing": 0, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_high_water": 7, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed_ops": 18, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed": 18, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_item_in_journal": 0 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 14119, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 5880177, 2024-04-26T01:57:48.933 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 14119, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 14119, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 5880177, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_bytes": { 2024-04-26T01:57:48.934 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 1390, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 8687066, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 1390, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 8687066, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.935 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_ops": { 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 1024, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 1390, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 1390, 2024-04-26T01:57:48.936 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 1390, 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 1390, 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle": { 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:48.937 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 18, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 2250, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 18, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 8, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 2250, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.938 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle-0x5635cb846500": { 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 1856, 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 8601779, 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 1856, 2024-04-26T01:57:48.939 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 1122, 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 8601779, 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:48.940 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-04-26T01:57:49.623 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:49 smithi169 ceph-mon[32851]: pgmap v530: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T01:57:49.652 INFO:tasks.check_counter:Daemon mds.c mds.exported=16 2024-04-26T01:57:49.652 INFO:tasks.check_counter:Daemon mds.c mds.imported=38 2024-04-26T01:57:49.652 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-26T01:57:49.652 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-26T01:57:49.652 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T01:57:49.652 DEBUG:tasks.check_counter:Getting stats from f 2024-04-26T01:57:49.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-26T01:57:49.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:49 smithi053 ceph-mon[32593]: pgmap v530: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T01:57:49.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:49 smithi028 ceph-mon[28247]: pgmap v530: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T01:57:49.903 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:57:51.548 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-04-26T01:57:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 1883, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 1885, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 8970333, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 598804, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 6, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 2, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.180816275, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.052029154, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.052634876, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.059850235, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1883, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.090207771, 2024-04-26T01:57:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000047906 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 1113, 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 1374, 2024-04-26T01:57:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 10225506, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 970116, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 7, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 5, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.104353614, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.036366418, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.028267186, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.030473002, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1371, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.062941723, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000045909 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 616752 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 1523, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 1480, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 10458699, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 569653, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 4, 2024-04-26T01:57:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.155506923, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.048148243, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.072164007, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.037100500, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1480, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.065047553, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000043951 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 1791264, 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 219216 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 1, 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-MDSRank": { 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 6901, 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.200614450, 2024-04-26T01:57:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000029070 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-PurgeQueue": { 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "mds": { 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "request": 0, 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "reply": 0, 2024-04-26T01:57:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "reply_latency": { 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_reply": 0, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "forward": 0, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_complete": 0, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_keys": 0, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_commit": 0, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_split": 0, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_merge": 0, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes": 732, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_top": 321, 2024-04-26T01:57:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_bottom": 411, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pin_tail": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pinned": 283, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_expired": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_with_caps": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "caps": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "subtrees": 11, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_hit": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_forward": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_discover": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_dir_fetch": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_remote_ino": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_lock": 0, 2024-04-26T01:57:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "load_cent": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "q": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "exported": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "exported_inodes": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "imported": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "imported_inodes": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_dir_fetch": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_backtrace_fetch": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_peer_discover": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rfiles": 996, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rbytes": 1185494761, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rsnaps": 6, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T01:57:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_set_tag": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_inotable_repaired": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_inodes": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_file_inodes": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_inode_file_caps": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_grant": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps": 0, 2024-04-26T01:57:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps_dirty": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_cap_release": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "process_request_cap_release": 0 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache": { 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update_receipt": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_try_discover": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_send_discover": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_handle_discover": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_delayed": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_enqueuing": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_created": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_enqueued": 0, 2024-04-26T01:57:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_reintegrated": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_migrated": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_processing": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_enqueued": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_prioritized": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_started": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_completed": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_quiesce_path": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_quiesce_inode": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_exportdir": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_flush": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragmentdir": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragstats": 0, 2024-04-26T01:57:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_inodestats": 0 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log": { 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "evlrg": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "evadd": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "evex": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "evtrm": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "ev": 4675, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "evexg": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "evexd": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "segadd": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "segex": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "segtrm": 0, 2024-04-26T01:57:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "seg": 3, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "segmjr": 69, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "segexg": 0, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "segexd": 0, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "expos": 29728585, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "wrpos": 30917560, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "rdpos": 30883586, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "jlat": { 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "replayed": 4675 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_mem": { 2024-04-26T01:57:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "ino": 477, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "ino+": 1978, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "ino-": 1501, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "dir": 51, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "dir+": 94, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "dir-": 43, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "dn": 732, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "dn+": 2327, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "dn-": 1595, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "cap": 0, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "cap+": 0, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "cap-": 0, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "rss": 55568, 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "heap": 215416 2024-04-26T01:57:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_server": { 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_client_request": 0, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_server_request": 0, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_request": 0, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_session": 0, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_peer_request": 0, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "req_create_latency": { 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getattr_latency": { 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getfilelock_latency": { 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "req_link_latency": { 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookup_latency": { 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookuphash_latency": { 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupino_latency": { 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupname_latency": { 2024-04-26T01:57:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupparent_latency": { 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupsnap_latency": { 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lssnap_latency": { 2024-04-26T01:57:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.569 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:51 smithi169 ceph-mon[32851]: pgmap v531: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 10 op/s 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mkdir_latency": { 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mknod_latency": { 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mksnap_latency": { 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "req_open_latency": { 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "req_readdir_latency": { 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rename_latency": { 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "req_renamesnap_latency": { 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "req_snapdiff_latency": { 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmdir_latency": { 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmsnap_latency": { 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmxattr_latency": { 2024-04-26T01:57:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setattr_latency": { 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setdirlayout_latency": { 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setfilelock_latency": { 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setlayout_latency": { 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setxattr_latency": { 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "req_symlink_latency": { 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "req_unlink_latency": { 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_revoke_eviction": 0, 2024-04-26T01:57:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_acquisition_throttle": 0, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getvxattr_latency": { 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_sessions": { 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "session_count": 2, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "session_add": 4, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "session_remove": 2, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_open": 3, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_stale": 0, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "total_load": 0, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "average_load": 0, 2024-04-26T01:57:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "avg_session_uptime": 3457, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "mdthresh_evicted": 0 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 220355, 2024-04-26T01:57:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 635, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 8032, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 167, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 1839898, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 33648, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter": { 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "op_active": 0, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "op_laggy": 0, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send": 4096, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send_bytes": 26, 2024-04-26T01:57:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "op_resend": 4, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "op_reply": 4096, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "op_latency": { 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 4096, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 6.831854042, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.001667933 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "op_inflight": 0, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "oplen_avg": { 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 4096, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 4097 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "op": 4096, 2024-04-26T01:57:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "op_r": 4096, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "op_w": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "op_rmw": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "op_pg": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_stat": 2445, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_create": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_read": 1648, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_write": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writefull": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writesame": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_append": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_zero": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_truncate": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_delete": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_mapext": 0, 2024-04-26T01:57:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_sparse_read": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_clonerange": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_getxattr": 2, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_setxattr": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_cmpxattr": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_rmxattr": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_resetxattrs": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_call": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_watch": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_notify": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls_filter": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_other": 0, 2024-04-26T01:57:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_active": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_send": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_resend": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_ping": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_active": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_send": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_resend": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_active": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_send": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_resend": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_active": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_send": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_resend": 0, 2024-04-26T01:57:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "command_active": 0, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "command_send": 0, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "command_resend": 0, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "map_epoch": 113, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "map_full": 0, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "map_inc": 25, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_sessions": 6, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_open": 6, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_close": 0, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_laggy": 0, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_wr": 0, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_rd": 2, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_del": 0 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "oft": { 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_objs": 0, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_kv_pairs": 0, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_updates": 0, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_removes": 0 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "osd-slow-ops": { 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_ops_count": 0 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "purge_queue": { 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops": 0, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing": 0, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_high_water": 0, 2024-04-26T01:57:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed_ops": 0, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed": 0, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_item_in_journal": 0 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 4515, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 29224010, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 4515, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 4515, 2024-04-26T01:57:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 29224010, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_bytes": { 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-04-26T01:57:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 4096, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 26723266, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 4096, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 26723266, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_ops": { 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 1024, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 4096, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 4096, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 4096, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 4096, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle": { 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 0, 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle-0x55e8bbfbe6e0": { 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 0, 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 0, 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:51.595 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-04-26T01:57:51.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:51 smithi053 ceph-mon[32593]: pgmap v531: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 10 op/s 2024-04-26T01:57:51.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:51 smithi028 ceph-mon[28247]: pgmap v531: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 10 op/s 2024-04-26T01:57:52.390 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-26T01:57:52.391 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-26T01:57:52.391 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-26T01:57:52.391 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-26T01:57:52.391 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T01:57:52.391 DEBUG:tasks.check_counter:Getting stats from i 2024-04-26T01:57:52.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-26T01:57:52.634 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:57:53.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:53 smithi169 ceph-mon[32851]: pgmap v532: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2024-04-26T01:57:53.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:53 smithi053 ceph-mon[32593]: pgmap v532: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2024-04-26T01:57:53.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:53 smithi028 ceph-mon[28247]: pgmap v532: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2024-04-26T01:57:54.288 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-04-26T01:57:54.288 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:54.288 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 6604, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 7032, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 2611368, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 17066244, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 11, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 4, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.524694639, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.227105899, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.203767309, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.090054214, 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:54.289 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 7030, 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 1.287625790, 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000183161 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 7646, 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.001428074, 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000186 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 456560 2024-04-26T01:57:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 9531, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 11404, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 3639068, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 19604131, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 154, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 6, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.853261582, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.294820520, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.431547184, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.170473112, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 11401, 2024-04-26T01:57:54.291 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 1.187161329, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000104127 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 14266, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.002277378, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000159 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 2977360, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 5338992 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 7173, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 7006, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 4429078, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 7683245, 2024-04-26T01:57:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 24, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 6, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.479892359, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.178586492, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.282573181, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.066167592, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 7006, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 1.120010812, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000159864 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 10403, 2024-04-26T01:57:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.001696646, 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000163 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 1815472, 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 229456 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 1, 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-MDSRank": { 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-04-26T01:57:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 5777, 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 2.838724218, 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000491383 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-PurgeQueue": { 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 160, 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.005370613, 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000033566 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "mds": { 2024-04-26T01:57:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "request": 2871, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "reply": 2696, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "reply_latency": { 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 2696, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 9.024300782, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.003347292 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_reply": 0, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "forward": 175, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_complete": 7, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_keys": 0, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_commit": 848, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_split": 5, 2024-04-26T01:57:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_merge": 0, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes": 587, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_top": 151, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_bottom": 422, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pin_tail": 14, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pinned": 369, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_expired": 1657, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_with_caps": 131, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "caps": 132, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "subtrees": 17, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse": 4432, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_hit": 3269, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_forward": 171, 2024-04-26T01:57:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_discover": 30, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_dir_fetch": 1, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_remote_ino": 0, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_lock": 184, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "load_cent": 1701, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "q": 0, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "exported": 53, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "exported_inodes": 1562, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "imported": 23, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "imported_inodes": 859, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_dir_fetch": 1, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_backtrace_fetch": 0, 2024-04-26T01:57:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_peer_discover": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rfiles": 996, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rbytes": 1185494761, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rsnaps": 6, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_set_tag": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_inotable_repaired": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_inodes": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_file_inodes": 0, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_inode_file_caps": 29, 2024-04-26T01:57:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_revoke": 583, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_grant": 1876, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps": 2107, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps_dirty": 1426, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_cap_release": 58, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "process_request_cap_release": 496 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache": { 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update": 0, 2024-04-26T01:57:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update_receipt": 0, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_try_discover": 0, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_send_discover": 30, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_handle_discover": 60, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays": 9, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_delayed": 0, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_enqueuing": 0, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_created": 143, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_enqueued": 36, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_reintegrated": 93, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_migrated": 7, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_processing": 0, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_enqueued": 0, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_prioritized": 0, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_started": 0, 2024-04-26T01:57:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_completed": 0, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_quiesce_path": 4, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_quiesce_inode": 632, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_exportdir": 53, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_flush": 0, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragmentdir": 5, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragstats": 0, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_inodestats": 0 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log": { 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "evlrg": 0, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "evadd": 4683, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "evex": 4549, 2024-04-26T01:57:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "evtrm": 4549, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "ev": 134, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "evexg": 0, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "evexd": 117, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "segadd": 75, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "segex": 74, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "segtrm": 74, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "seg": 1, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "segmjr": 1, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "segexg": 0, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "segexd": 2, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "expos": 29728585, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "wrpos": 30991106, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "rdpos": 0, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "jlat": { 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 2601, 2024-04-26T01:57:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 16.881587915, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.006490422 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "replayed": 0 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_mem": { 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "ino": 492, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "ino+": 2022, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "ino-": 1530, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "dir": 59, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "dir+": 98, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "dir-": 39, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "dn": 600, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "dn+": 2687, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "dn-": 2087, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "cap": 132, 2024-04-26T01:57:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "cap+": 1409, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "cap-": 1277, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "rss": 69624, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "heap": 231800 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_server": { 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_client_request": 3732, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_server_request": 1047, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_request": 2871, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_session": 372, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_peer_request": 2098, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "req_create_latency": { 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1062, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 5.764579225, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.005428040 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getattr_latency": { 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 247, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.162886469, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000659459 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getfilelock_latency": { 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "req_link_latency": { 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 111, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.387283015, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.003489036 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookup_latency": { 2024-04-26T01:57:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 428, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.476136738, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.001112469 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookuphash_latency": { 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupino_latency": { 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000139893, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000139893 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupname_latency": { 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupparent_latency": { 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupsnap_latency": { 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lssnap_latency": { 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 17, 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.003566765, 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000209809 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mkdir_latency": { 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 37, 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.448875687, 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.012131775 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mknod_latency": { 2024-04-26T01:57:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mksnap_latency": { 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 17, 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.200323598, 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.011783741 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "req_open_latency": { 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 440, 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.196538475, 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000446678 2024-04-26T01:57:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "req_readdir_latency": { 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 65, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.135389330, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.002082912 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rename_latency": { 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 133, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.826211578, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.006212117 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "req_renamesnap_latency": { 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "req_snapdiff_latency": { 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmdir_latency": { 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 2, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.011866942, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.005933471 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmsnap_latency": { 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 11, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.089574654, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.008143150 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.312 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmxattr_latency": { 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setattr_latency": { 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.001210371, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.001210371 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setdirlayout_latency": { 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setfilelock_latency": { 2024-04-26T01:57:54.313 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setlayout_latency": { 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setxattr_latency": { 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 7, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.022045718, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.003149388 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "req_symlink_latency": { 2024-04-26T01:57:54.314 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "req_unlink_latency": { 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 117, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.297672324, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.002544207 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_revoke_eviction": 0, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_acquisition_throttle": 0, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getvxattr_latency": { 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_sessions": { 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "session_count": 2, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "session_add": 4, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "session_remove": 2, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_open": 2, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_stale": 0, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "total_load": 140, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "average_load": 70, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "avg_session_uptime": 1013, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "mdthresh_evicted": 0 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-04-26T01:57:54.316 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 360, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 360, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:54.317 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:54.318 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 683664, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 712, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:54.319 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 8032, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 167, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 1929350, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 42281, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter": { 2024-04-26T01:57:54.320 INFO:teuthology.orchestra.run.smithi169.stdout: "op_active": 0, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "op_laggy": 0, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send": 5749, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send_bytes": 31335757, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "op_resend": 37, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "op_reply": 5749, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "op_latency": { 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 5749, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 110.173644878, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.019163966 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "op_inflight": 0, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "oplen_avg": { 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 5749, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 9697 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.321 INFO:teuthology.orchestra.run.smithi169.stdout: "op": 5749, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "op_r": 24, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "op_w": 5725, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "op_rmw": 0, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "op_pg": 0, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_stat": 760, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_create": 1075, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_read": 17, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_write": 2618, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writefull": 298, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writesame": 0, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_append": 0, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_zero": 0, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_truncate": 4, 2024-04-26T01:57:54.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_delete": 575, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_mapext": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_sparse_read": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_clonerange": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_getxattr": 7, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_setxattr": 2150, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_cmpxattr": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_rmxattr": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_resetxattrs": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_call": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_watch": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_notify": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls_filter": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_other": 0, 2024-04-26T01:57:54.323 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_active": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_send": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_resend": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_ping": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_active": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_send": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_resend": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_active": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_send": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_resend": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_active": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_send": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_resend": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "command_active": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "command_send": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "command_resend": 0, 2024-04-26T01:57:54.324 INFO:teuthology.orchestra.run.smithi169.stdout: "map_epoch": 113, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "map_full": 0, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "map_inc": 25, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_sessions": 12, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_open": 12, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_close": 0, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_laggy": 0, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_wr": 2014, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_rd": 14, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_del": 165 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "oft": { 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_objs": 1, 2024-04-26T01:57:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_kv_pairs": 0, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_updates": 0, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_removes": 0 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "osd-slow-ops": { 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_ops_count": 734 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "purge_queue": { 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops": 0, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops_high_water": 17, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing": 0, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_high_water": 9, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed_ops": 50, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed": 36, 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_item_in_journal": 0 2024-04-26T01:57:54.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 23304, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 8199026, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 23304, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 23304, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 8199026, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.327 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_bytes": { 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 5749, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 31339353, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 5749, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 31339353, 2024-04-26T01:57:54.328 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_ops": { 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 1024, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:54.329 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 5749, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 5749, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 5749, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 5749, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle": { 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 36, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 3708, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 36, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 17, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 3708, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle-0x561a41da6500": { 2024-04-26T01:57:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 4683, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 26796802, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 4683, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 2601, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 26796802, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.332 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:54.333 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-04-26T01:57:55.092 INFO:tasks.check_counter:Daemon mds.i mds.exported=53 2024-04-26T01:57:55.092 INFO:tasks.check_counter:Daemon mds.i mds.imported=23 2024-04-26T01:57:55.092 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-26T01:57:55.092 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=17 2024-04-26T01:57:55.092 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=11 2024-04-26T01:57:55.092 DEBUG:tasks.check_counter:Getting stats from l 2024-04-26T01:57:55.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-26T01:57:55.331 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:57:55.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:55 smithi053 ceph-mon[32593]: pgmap v533: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 60 KiB/s wr, 8 op/s 2024-04-26T01:57:55.743 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:55 smithi169 ceph-mon[32851]: pgmap v533: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 60 KiB/s wr, 8 op/s 2024-04-26T01:57:55.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:55 smithi028 ceph-mon[28247]: pgmap v533: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 60 KiB/s wr, 8 op/s 2024-04-26T01:57:57.121 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 362, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 351, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 1629664, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 217216, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 4, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 0, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.058903207, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.019371768, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.038309125, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.008017178, 2024-04-26T01:57:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 350, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.016989967, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000048542 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 1629664, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 217216 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 2, 2024-04-26T01:57:57.123 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 268, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 320, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 203152, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 5, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 3, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.015280808, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.011041216, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.001247033, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 265, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.030274246, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000114242 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:57.124 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 203152 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 4, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 4, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 2944, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 448, 2024-04-26T01:57:57.125 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 2, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 1, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.000677122, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.000035590, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.000617118, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 3, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000767188, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000255729 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-04-26T01:57:57.126 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 0, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T01:57:57.127 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-04-26T01:57:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-04-26T01:57:57.129 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 148692, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 30, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-04-26T01:57:57.130 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 928, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 2, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-04-26T01:57:57.131 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 364, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 1578543, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 364, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 364, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 1578543, 2024-04-26T01:57:57.132 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-04-26T01:57:57.133 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-04-26T01:57:57.133 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-04-26T01:57:57.133 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-04-26T01:57:57.133 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:57.133 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-04-26T01:57:57.133 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-04-26T01:57:57.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:57 smithi053 ceph-mon[32593]: pgmap v534: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 10 op/s 2024-04-26T01:57:57.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:57 smithi028 ceph-mon[28247]: pgmap v534: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 10 op/s 2024-04-26T01:57:57.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:57 smithi169 ceph-mon[32851]: pgmap v534: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 10 op/s 2024-04-26T01:57:57.819 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-26T01:57:57.819 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-26T01:57:57.819 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-26T01:57:57.819 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-26T01:57:57.819 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-26T01:57:57.820 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T01:57:57.832 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T01:57:57.832 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:57:57.833 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T01:57:57.892 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:57:57.892 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T01:57:57.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd blocklist ls 2024-04-26T01:57:58.446 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:59.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:57:59 smithi053 ceph-mon[32593]: pgmap v535: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2024-04-26T01:57:59.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:57:59 smithi028 ceph-mon[28247]: pgmap v535: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2024-04-26T01:57:59.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:57:59 smithi169 ceph-mon[32851]: pgmap v535: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2024-04-26T01:58:00.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 -- 172.21.15.28:0/2652573051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8105870 msgr2=0x7f0cf81098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 --2- 172.21.15.28:0/2652573051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8105870 0x7f0cf81098c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00098f0 tx=0x7f0ce0009c00 comp rx=0 tx=0).stop 2024-04-26T01:58:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 -- 172.21.15.28:0/2652573051 shutdown_connections 2024-04-26T01:58:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 --2- 172.21.15.28:0/2652573051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf8109fa0 0x7f0cf8111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 --2- 172.21.15.28:0/2652573051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf8104ec0 0x7f0cf81052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 --2- 172.21.15.28:0/2652573051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8105870 0x7f0cf81098c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 -- 172.21.15.28:0/2652573051 >> 172.21.15.28:0/2652573051 conn(0x7f0cf8100750 msgr2=0x7f0cf8102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 -- 172.21.15.28:0/2652573051 shutdown_connections 2024-04-26T01:58:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.830+0000 7f0cff800700 1 -- 172.21.15.28:0/2652573051 wait complete. 2024-04-26T01:58:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.832+0000 7f0cff800700 1 Processor -- start 2024-04-26T01:58:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.833+0000 7f0cff800700 1 -- start start 2024-04-26T01:58:00.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.833+0000 7f0cff800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8104ec0 0x7f0cf819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cff800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf8105870 0x7f0cf819b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cff800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf8109fa0 0x7f0cf819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cff800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf8114e10 con 0x7f0cf8105870 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cff800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf8114c90 con 0x7f0cf8104ec0 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cff800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf8114f90 con 0x7f0cf8109fa0 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cfce00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8104ec0 0x7f0cf819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cefe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf8105870 0x7f0cf819b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cfce00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8104ec0 0x7f0cf819b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:58944/0 (socket says 172.21.15.28:58944) 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cfce00700 1 -- 172.21.15.28:0/3296513589 learned_addr learned my addr 172.21.15.28:0/3296513589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:58:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.834+0000 7f0cfd800700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf8109fa0 0x7f0cf819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:00.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cefe00700 1 -- 172.21.15.28:0/3296513589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf8109fa0 msgr2=0x7f0cf819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cefe00700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf8109fa0 0x7f0cf819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cefe00700 1 -- 172.21.15.28:0/3296513589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8104ec0 msgr2=0x7f0cf819b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cefe00700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8104ec0 0x7f0cf819b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cefe00700 1 -- 172.21.15.28:0/3296513589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf81a0390 con 0x7f0cf8105870 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cfce00700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8104ec0 0x7f0cf819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cefe00700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf8105870 0x7f0cf819b8c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009230 tx=0x7f0ce000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cf6000700 1 -- 172.21.15.28:0/3296513589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0005140 con 0x7f0cf8105870 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf81a0620 con 0x7f0cf8105870 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cf6000700 1 -- 172.21.15.28:0/3296513589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ce000b560 con 0x7f0cf8105870 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.835+0000 7f0cf6000700 1 -- 172.21.15.28:0/3296513589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0003a40 con 0x7f0cf8105870 2024-04-26T01:58:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.836+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf81a7f00 con 0x7f0cf8105870 2024-04-26T01:58:00.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.837+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf8069730 con 0x7f0cf8105870 2024-04-26T01:58:00.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.839+0000 7f0cf6000700 1 -- 172.21.15.28:0/3296513589 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0ce000ba10 con 0x7f0cf8105870 2024-04-26T01:58:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.840+0000 7f0cf6000700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0cdc0a3e30 0x7f0cdc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.840+0000 7f0cf6000700 1 -- 172.21.15.28:0/3296513589 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7f0ce00d15c0 con 0x7f0cf8105870 2024-04-26T01:58:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.841+0000 7f0cfce00700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0cdc0a3e30 0x7f0cdc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.841+0000 7f0cfce00700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0cdc0a3e30 0x7f0cdc0a62f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80104e0 tx=0x7f0ce8007480 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:00.845+0000 7f0cf6000700 1 -- 172.21.15.28:0/3296513589 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ce0013030 con 0x7f0cf8105870 2024-04-26T01:58:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.132+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0cf802df80 con 0x7f0cf8105870 2024-04-26T01:58:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.133+0000 7f0cf6000700 1 -- 172.21.15.28:0/3296513589 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v113) ==== 81+0+943 (secure 0 0 0) 0x7f0ce009dc90 con 0x7f0cf8105870 2024-04-26T01:58:01.133 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3431203710 2024-04-27T01:40:57.926618+0000 2024-04-26T01:58:01.133 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1295766352 2024-04-27T01:40:57.926618+0000 2024-04-26T01:58:01.133 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3441292878 2024-04-27T01:40:57.926618+0000 2024-04-26T01:58:01.133 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3962708174 2024-04-27T01:28:23.601445+0000 2024-04-26T01:58:01.133 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/259754378 2024-04-27T01:28:23.601445+0000 2024-04-26T01:58:01.134 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3431203710 2024-04-27T01:40:57.926618+0000 2024-04-26T01:58:01.134 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2991743583 2024-04-27T01:28:23.601445+0000 2024-04-26T01:58:01.134 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/218486566 2024-04-27T01:40:57.926618+0000 2024-04-26T01:58:01.134 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3106064028 2024-04-27T01:28:23.601445+0000 2024-04-26T01:58:01.134 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1803582359 2024-04-27T01:27:49.755979+0000 2024-04-26T01:58:01.134 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/615407590 2024-04-27T01:27:49.755979+0000 2024-04-26T01:58:01.135 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/1017286588 2024-04-27T01:27:49.755979+0000 2024-04-26T01:58:01.135 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2119832118 2024-04-27T01:27:49.755979+0000 2024-04-26T01:58:01.135 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2991743583 2024-04-27T01:28:23.601445+0000 2024-04-26T01:58:01.135 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3010978828 2024-04-27T01:40:57.926618+0000 2024-04-26T01:58:01.135 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/1017286588 2024-04-27T01:27:49.755979+0000 2024-04-26T01:58:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.136+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0cdc0a3e30 msgr2=0x7f0cdc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.136+0000 7f0cff800700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0cdc0a3e30 0x7f0cdc0a62f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80104e0 tx=0x7f0ce8007480 comp rx=0 tx=0).stop 2024-04-26T01:58:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.136+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf8105870 msgr2=0x7f0cf819b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.136+0000 7f0cff800700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf8105870 0x7f0cf819b8c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009230 tx=0x7f0ce000fa70 comp rx=0 tx=0).stop 2024-04-26T01:58:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.137+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 shutdown_connections 2024-04-26T01:58:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.137+0000 7f0cff800700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f0cdc0a3e30 0x7f0cdc0a62f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.137+0000 7f0cff800700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf8109fa0 0x7f0cf819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.137+0000 7f0cff800700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0cf8104ec0 0x7f0cf819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.137+0000 7f0cff800700 1 --2- 172.21.15.28:0/3296513589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf8105870 0x7f0cf819b8c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.137+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 >> 172.21.15.28:0/3296513589 conn(0x7f0cf8100750 msgr2=0x7f0cf8102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.137+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 shutdown_connections 2024-04-26T01:58:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:01.137+0000 7f0cff800700 1 -- 172.21.15.28:0/3296513589 wait complete. 2024-04-26T01:58:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:listed 16 entries 2024-04-26T01:58:01.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:01 smithi028 ceph-mon[28247]: pgmap v536: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 10 op/s 2024-04-26T01:58:01.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:01 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3296513589' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T01:58:01.809 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:01 smithi053 ceph-mon[32593]: pgmap v536: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 10 op/s 2024-04-26T01:58:01.809 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:01 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3296513589' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T01:58:01.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:01 smithi169 ceph-mon[32851]: pgmap v536: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 10 op/s 2024-04-26T01:58:01.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:01 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3296513589' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T01:58:02.063 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T01:58:02.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:58:02.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /tmp/cephfs 2024-04-26T01:58:02.370 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T01:58:02.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:58:02.371 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-04-26T01:58:02.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:58:02.463 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-26T01:58:02.544 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-04-26T01:58:02.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:58:02.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-04-26T01:58:02.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:58:02.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-26T01:58:02.834 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T01:58:02.834 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-26T01:58:02.834 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T01:58:02.903 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/a8ba7c1d4152ab12c6f62201ed96aedc715af46ea14597c1a6f02356baa5b852/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/a8ba7c1d4152ab12c6f62201ed96aedc715af46ea14597c1a6f02356baa5b852/diff,workdir=/var/lib/containers/storage/overlay/a8ba7c1d4152ab12c6f62201ed96aedc715af46ea14597c1a6f02356baa5b852/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5bb29d42e26ecae63cc8b879c4cb2920d62989651c79db12cb57f8c4591e717f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/5bb29d42e26ecae63cc8b879c4cb2920d62989651c79db12cb57f8c4591e717f/diff,workdir=/var/lib/containers/storage/overlay/5bb29d42e26ecae63cc8b879c4cb2920d62989651c79db12cb57f8c4591e717f/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/1ce514935404edf02121314466577a4e81cecf5c84111e7963cc8c0995f38aec/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/1ce514935404edf02121314466577a4e81cecf5c84111e7963cc8c0995f38aec/diff,workdir=/var/lib/containers/storage/overlay/1ce514935404edf02121314466577a4e81cecf5c84111e7963cc8c0995f38aec/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/58f5649a196dfb2c316d8c95864eca4d0ffe63c2ef7afd1e3125cd104f92a5de/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/58f5649a196dfb2c316d8c95864eca4d0ffe63c2ef7afd1e3125cd104f92a5de/diff,workdir=/var/lib/containers/storage/overlay/58f5649a196dfb2c316d8c95864eca4d0ffe63c2ef7afd1e3125cd104f92a5de/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/70cc93de5c5090a983dddf89f20176bc2ce5b292c3b49a27c5e56ff56ea9656c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/70cc93de5c5090a983dddf89f20176bc2ce5b292c3b49a27c5e56ff56ea9656c/diff,workdir=/var/lib/containers/storage/overlay/70cc93de5c5090a983dddf89f20176bc2ce5b292c3b49a27c5e56ff56ea9656c/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/537b52167de8f8d19e51216a281015a29ca6ba0a423b318514b35655b3703c80/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/537b52167de8f8d19e51216a281015a29ca6ba0a423b318514b35655b3703c80/diff,workdir=/var/lib/containers/storage/overlay/537b52167de8f8d19e51216a281015a29ca6ba0a423b318514b35655b3703c80/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4a1cfc6acb5210bd72085e0ce222a09e763684fadd820a3d189e5562c1e35338/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/4a1cfc6acb5210bd72085e0ce222a09e763684fadd820a3d189e5562c1e35338/diff,workdir=/var/lib/containers/storage/overlay/4a1cfc6acb5210bd72085e0ce222a09e763684fadd820a3d189e5562c1e35338/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e269bdb4d10c318ce3c9a03eb229bbe731346213d1fed5b18cb9059752f10dfa/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/e269bdb4d10c318ce3c9a03eb229bbe731346213d1fed5b18cb9059752f10dfa/diff,workdir=/var/lib/containers/storage/overlay/e269bdb4d10c318ce3c9a03eb229bbe731346213d1fed5b18cb9059752f10dfa/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/8a34dc1a0e020428db1b5242c49f9abb972b155afae79006f5ca5ed5e16601ba/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/8a34dc1a0e020428db1b5242c49f9abb972b155afae79006f5ca5ed5e16601ba/diff,workdir=/var/lib/containers/storage/overlay/8a34dc1a0e020428db1b5242c49f9abb972b155afae79006f5ca5ed5e16601ba/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d2b98773e77b7deb95155057854adabf6daab62e85a99a95f61853ba41f67c2d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDROD4Q7APQY6VOEWUUDQXKK6Q:/var/lib/containers/storage/overlay/l/CXQXEG33HE4UVGIB7ESIMCAEGB,upperdir=/var/lib/containers/storage/overlay/d2b98773e77b7deb95155057854adabf6daab62e85a99a95f61853ba41f67c2d/diff,workdir=/var/lib/containers/storage/overlay/d2b98773e77b7deb95155057854adabf6daab62e85a99a95f61853ba41f67c2d/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 2024-04-26T01:58:02.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:58:02.904 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-26T01:58:02.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:58:02.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-26T01:58:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-04-26T01:58:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:58:03.073 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T01:58:03.084 INFO:tasks.cephadm:Teardown begin 2024-04-26T01:58:03.084 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T01:58:03.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:58:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:03.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:03 smithi028 ceph-mon[28247]: pgmap v537: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-04-26T01:58:03.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:03 smithi169 ceph-mon[32851]: pgmap v537: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-04-26T01:58:03.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:03 smithi053 ceph-mon[32593]: pgmap v537: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-04-26T01:58:05.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:05 smithi028 ceph-mon[28247]: pgmap v538: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-26T01:58:05.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:05 smithi169 ceph-mon[32851]: pgmap v538: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.954+0000 7fce0d600700 1 -- 172.21.15.28:0/2797574517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 msgr2=0x7fce08111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.954+0000 7fce0d600700 1 --2- 172.21.15.28:0/2797574517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 0x7fce08111830 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b0a0 tx=0x7fce0000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.954+0000 7fce0d600700 1 -- 172.21.15.28:0/2797574517 shutdown_connections 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.954+0000 7fce0d600700 1 --2- 172.21.15.28:0/2797574517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 0x7fce08111830 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.954+0000 7fce0d600700 1 --2- 172.21.15.28:0/2797574517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08074c70 0x7fce08075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.954+0000 7fce0d600700 1 --2- 172.21.15.28:0/2797574517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce08075590 0x7fce0810ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.954+0000 7fce0d600700 1 -- 172.21.15.28:0/2797574517 >> 172.21.15.28:0/2797574517 conn(0x7fce08079ba0 msgr2=0x7fce08079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.955+0000 7fce0d600700 1 -- 172.21.15.28:0/2797574517 shutdown_connections 2024-04-26T01:58:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.955+0000 7fce0d600700 1 -- 172.21.15.28:0/2797574517 wait complete. 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.956+0000 7fce0d600700 1 Processor -- start 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.956+0000 7fce0d600700 1 -- start start 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce0d600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce08074c70 0x7fce0819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce0d600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08075590 0x7fce0819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce0d600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 0x7fce0819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce0d600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce08114e50 con 0x7fce08075590 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce0d600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fce08114cd0 con 0x7fce0810f440 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce0d600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fce08114fd0 con 0x7fce08074c70 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce06a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce08074c70 0x7fce0819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce06a00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce08074c70 0x7fce0819b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:37242/0 (socket says 172.21.15.28:37242) 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce06a00700 1 -- 172.21.15.28:0/3659150968 learned_addr learned my addr 172.21.15.28:0/3659150968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fcdfe000700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08075590 0x7fce0819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce07400700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 0x7fce0819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce07400700 1 -- 172.21.15.28:0/3659150968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce08074c70 msgr2=0x7fce0819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.957+0000 7fce07400700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce08074c70 0x7fce0819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fce07400700 1 -- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08075590 msgr2=0x7fce0819b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fce07400700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08075590 0x7fce0819b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fce07400700 1 -- 172.21.15.28:0/3659150968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce081a02f0 con 0x7fce0810f440 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fce07400700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 0x7fce0819fbb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b070 tx=0x7fce00004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fcdffe00700 1 -- 172.21.15.28:0/3659150968 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00021070 con 0x7fce0810f440 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fcdffe00700 1 -- 172.21.15.28:0/3659150968 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce00004e30 con 0x7fce0810f440 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce081a0580 con 0x7fce0810f440 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fcdffe00700 1 -- 172.21.15.28:0/3659150968 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00004620 con 0x7fce0810f440 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.958+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce081a7e60 con 0x7fce0810f440 2024-04-26T01:58:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.960+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce08069730 con 0x7fce0810f440 2024-04-26T01:58:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.962+0000 7fcdffe00700 1 -- 172.21.15.28:0/3659150968 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fce00007980 con 0x7fce0810f440 2024-04-26T01:58:05.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.963+0000 7fcdffe00700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcdec0a3e30 0x7fcdec0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.963+0000 7fcdffe00700 1 -- 172.21.15.28:0/3659150968 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8513+0+0 (secure 0 0 0) 0x7fce0000f020 con 0x7fce0810f440 2024-04-26T01:58:05.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.964+0000 7fce06a00700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcdec0a3e30 0x7fcdec0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.964+0000 7fce06a00700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcdec0a3e30 0x7fcdec0a62f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8007cf0 tx=0x7fcdf8007480 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:05.965+0000 7fcdffe00700 1 -- 172.21.15.28:0/3659150968 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce0002a020 con 0x7fce0810f440 2024-04-26T01:58:05.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:05 smithi053 ceph-mon[32593]: pgmap v538: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-26T01:58:06.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.247+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce0802df80 con 0x7fce0810f440 2024-04-26T01:58:06.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.248+0000 7fcdffe00700 1 -- 172.21.15.28:0/3659150968 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11867 (secure 0 0 0) 0x7fce0009e5e0 con 0x7fce0810f440 2024-04-26T01:58:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:58:06.247 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:57:32.113086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24509,24523]},"id":1}]} 2024-04-26T01:58:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.251+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcdec0a3e30 msgr2=0x7fcdec0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.251+0000 7fce0d600700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcdec0a3e30 0x7fcdec0a62f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8007cf0 tx=0x7fcdf8007480 comp rx=0 tx=0).stop 2024-04-26T01:58:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.251+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 msgr2=0x7fce0819fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.251+0000 7fce0d600700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 0x7fce0819fbb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b070 tx=0x7fce00004c80 comp rx=0 tx=0).stop 2024-04-26T01:58:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.252+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 shutdown_connections 2024-04-26T01:58:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.252+0000 7fce0d600700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcdec0a3e30 0x7fcdec0a62f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.252+0000 7fce0d600700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0810f440 0x7fce0819fbb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.252+0000 7fce0d600700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce08074c70 0x7fce0819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.252+0000 7fce0d600700 1 --2- 172.21.15.28:0/3659150968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08075590 0x7fce0819b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.252+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 >> 172.21.15.28:0/3659150968 conn(0x7fce08079ba0 msgr2=0x7fce080ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.252+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 shutdown_connections 2024-04-26T01:58:06.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:06.252+0000 7fce0d600700 1 -- 172.21.15.28:0/3659150968 wait complete. 2024-04-26T01:58:06.251 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 53 2024-04-26T01:58:06.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:06 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3659150968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:06.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:06 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3659150968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:06.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:06 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3659150968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:07.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:58:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:07.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:07 smithi028 ceph-mon[28247]: pgmap v539: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-04-26T01:58:07.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:07 smithi169 ceph-mon[32851]: pgmap v539: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-04-26T01:58:07.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:07 smithi053 ceph-mon[32593]: pgmap v539: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-04-26T01:58:08.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:08 smithi028 ceph-mon[28247]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:58:08.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:08 smithi169 ceph-mon[32851]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:58:08.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:08 smithi053 ceph-mon[32593]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:58:09.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:09 smithi028 ceph-mon[28247]: pgmap v541: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 63 KiB/s wr, 29 op/s 2024-04-26T01:58:09.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:09 smithi169 ceph-mon[32851]: pgmap v541: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 63 KiB/s wr, 29 op/s 2024-04-26T01:58:09.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:09 smithi053 ceph-mon[32593]: pgmap v541: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 63 KiB/s wr, 29 op/s 2024-04-26T01:58:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 -- 172.21.15.28:0/18502678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bac075590 msgr2=0x7f5bac10ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 --2- 172.21.15.28:0/18502678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bac075590 0x7f5bac10ef30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009990 tx=0x7f5b94009ca0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 -- 172.21.15.28:0/18502678 shutdown_connections 2024-04-26T01:58:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 --2- 172.21.15.28:0/18502678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bac10f470 0x7f5bac111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 --2- 172.21.15.28:0/18502678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bac074c70 0x7f5bac075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 --2- 172.21.15.28:0/18502678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bac075590 0x7f5bac10ef30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 -- 172.21.15.28:0/18502678 >> 172.21.15.28:0/18502678 conn(0x7f5bac079bf0 msgr2=0x7f5bac07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 -- 172.21.15.28:0/18502678 shutdown_connections 2024-04-26T01:58:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.574+0000 7f5bb3800700 1 -- 172.21.15.28:0/18502678 wait complete. 2024-04-26T01:58:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.575+0000 7f5bb3800700 1 Processor -- start 2024-04-26T01:58:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.576+0000 7f5bb3800700 1 -- start start 2024-04-26T01:58:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.576+0000 7f5bb3800700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bac074c70 0x7f5bac19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.576+0000 7f5bb3800700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bac075590 0x7f5bac19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.576+0000 7f5bb3800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bac10f470 0x7f5bac19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.576+0000 7f5bb3800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac114ea0 con 0x7f5bac10f470 2024-04-26T01:58:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.576+0000 7f5bb3800700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac114d20 con 0x7f5bac075590 2024-04-26T01:58:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.576+0000 7f5bb3800700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac115020 con 0x7f5bac074c70 2024-04-26T01:58:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bac074c70 0x7f5bac19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bac074c70 0x7f5bac19b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46808/0 (socket says 172.21.15.28:46808) 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 -- 172.21.15.28:0/2515160889 learned_addr learned my addr 172.21.15.28:0/2515160889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5ba3e00700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bac075590 0x7f5bac19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb1800700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bac10f470 0x7f5bac19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 -- 172.21.15.28:0/2515160889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bac075590 msgr2=0x7f5bac19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bac075590 0x7f5bac19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 -- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bac10f470 msgr2=0x7f5bac19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bac10f470 0x7f5bac19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 -- 172.21.15.28:0/2515160889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac1a0350 con 0x7f5bac074c70 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.577+0000 7f5bb0e00700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bac074c70 0x7f5bac19b390 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00e9a0 tx=0x7f5b9c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.578+0000 7f5baa000700 1 -- 172.21.15.28:0/2515160889 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00cbd0 con 0x7f5bac074c70 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.578+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bac1a0640 con 0x7f5bac074c70 2024-04-26T01:58:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.578+0000 7f5baa000700 1 -- 172.21.15.28:0/2515160889 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9c00cd70 con 0x7f5bac074c70 2024-04-26T01:58:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.578+0000 7f5baa000700 1 -- 172.21.15.28:0/2515160889 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c018680 con 0x7f5bac074c70 2024-04-26T01:58:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.578+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bac1a7ec0 con 0x7f5bac074c70 2024-04-26T01:58:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.579+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bac069730 con 0x7f5bac074c70 2024-04-26T01:58:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.581+0000 7f5baa000700 1 -- 172.21.15.28:0/2515160889 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b9c010430 con 0x7f5bac074c70 2024-04-26T01:58:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.582+0000 7f5baa000700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5b900a3e30 0x7f5b900a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.583+0000 7f5baa000700 1 -- 172.21.15.28:0/2515160889 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8557+0+0 (secure 0 0 0) 0x7f5b9c0d0a50 con 0x7f5bac074c70 2024-04-26T01:58:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.583+0000 7f5ba3e00700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5b900a3e30 0x7f5b900a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.584+0000 7f5ba3e00700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5b900a3e30 0x7f5b900a62f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5b940098e0 tx=0x7f5b94005960 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.587+0000 7f5baa000700 1 -- 172.21.15.28:0/2515160889 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b9c09d070 con 0x7f5bac074c70 2024-04-26T01:58:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.868+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bac02df80 con 0x7f5bac074c70 2024-04-26T01:58:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.870+0000 7f5baa000700 1 -- 172.21.15.28:0/2515160889 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11867 (secure 0 0 0) 0x7f5b9c09c970 con 0x7f5bac074c70 2024-04-26T01:58:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:58:10.868 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14358,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2607969943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2607969943},{"type":"v1","addr":"172.21.15.28:6835","nonce":2607969943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14370,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/447949088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":447949088},{"type":"v1","addr":"172.21.15.28:6837","nonce":447949088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/332324514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":332324514},{"type":"v1","addr":"172.21.15.53:6833","nonce":332324514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2175861856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2175861856},{"type":"v1","addr":"172.21.15.53:6835","nonce":2175861856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/451902326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":451902326},{"type":"v1","addr":"172.21.15.169:6833","nonce":451902326}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2195700399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2195700399},{"type":"v1","addr":"172.21.15.53:6837","nonce":2195700399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:36:57.152667+0000","modified":"2024-04-26T01:57:32.113086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24509,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2114874192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2114874192},{"type":"v1","addr":"172.21.15.28:6839","nonce":2114874192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2299937376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2299937376},{"type":"v1","addr":"172.21.15.28:6841","nonce":2299937376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/1802608598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":1802608598},{"type":"v1","addr":"172.21.15.169:6835","nonce":1802608598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.169:6837/3777955238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":3777955238},{"type":"v1","addr":"172.21.15.169:6837","nonce":3777955238}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":1,"incarnation":18,"state":"up:active","state_seq":11,"addr":"172.21.15.53:6839/955096275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":955096275},{"type":"v1","addr":"172.21.15.53:6839","nonce":955096275}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.169:6839/2157805186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2157805186},{"type":"v1","addr":"172.21.15.169:6839","nonce":2157805186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24505,24509,24523]},"id":1}]} 2024-04-26T01:58:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5b900a3e30 msgr2=0x7f5b900a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5b900a3e30 0x7f5b900a62f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5b940098e0 tx=0x7f5b94005960 comp rx=0 tx=0).stop 2024-04-26T01:58:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bac074c70 msgr2=0x7f5bac19b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bac074c70 0x7f5bac19b390 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00e9a0 tx=0x7f5b9c00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 shutdown_connections 2024-04-26T01:58:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f5b900a3e30 0x7f5b900a62f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bac10f470 0x7f5bac19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bac074c70 0x7f5bac19b390 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 --2- 172.21.15.28:0/2515160889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bac075590 0x7f5bac19b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.873+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 >> 172.21.15.28:0/2515160889 conn(0x7f5bac079bf0 msgr2=0x7f5bac0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.874+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 shutdown_connections 2024-04-26T01:58:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:10.874+0000 7f5bb3800700 1 -- 172.21.15.28:0/2515160889 wait complete. 2024-04-26T01:58:10.873 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 53 2024-04-26T01:58:11.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:58:11.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:11 smithi028 ceph-mon[28247]: pgmap v542: 97 pgs: 97 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-26T01:58:11.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:11 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2515160889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:11.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:11 smithi169 ceph-mon[32851]: pgmap v542: 97 pgs: 97 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-26T01:58:11.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:11 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2515160889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:11.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:11 smithi053 ceph-mon[32593]: pgmap v542: 97 pgs: 97 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-26T01:58:11.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:11 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2515160889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:13.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:13 smithi169 ceph-mon[32851]: pgmap v543: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-26T01:58:13.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:13 smithi169 ceph-mon[32851]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:58:13.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:13 smithi053 ceph-mon[32593]: pgmap v543: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-26T01:58:13.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:13 smithi053 ceph-mon[32593]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:58:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:13 smithi028 ceph-mon[28247]: pgmap v543: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-04-26T01:58:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:13 smithi028 ceph-mon[28247]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:58:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 -- 172.21.15.28:0/2900181985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 msgr2=0x7f99341098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 --2- 172.21.15.28:0/2900181985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 0x7f99341098b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9920009920 tx=0x7f9920009c30 comp rx=0 tx=0).stop 2024-04-26T01:58:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 -- 172.21.15.28:0/2900181985 shutdown_connections 2024-04-26T01:58:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 --2- 172.21.15.28:0/2900181985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9934109f90 0x7f9934111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 --2- 172.21.15.28:0/2900181985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9934104eb0 0x7f9934105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 --2- 172.21.15.28:0/2900181985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 0x7f99341098b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 -- 172.21.15.28:0/2900181985 >> 172.21.15.28:0/2900181985 conn(0x7f9934100740 msgr2=0x7f9934102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 -- 172.21.15.28:0/2900181985 shutdown_connections 2024-04-26T01:58:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.528+0000 7f993b000700 1 -- 172.21.15.28:0/2900181985 wait complete. 2024-04-26T01:58:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.529+0000 7f993b000700 1 Processor -- start 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.529+0000 7f993b000700 1 -- start start 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f993b000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9934104eb0 0x7f993419b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f993b000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 0x7f993419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f993b000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9934109f90 0x7f993419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f993b000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9934114e00 con 0x7f9934104eb0 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f993b000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9934114c80 con 0x7f9934109f90 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f993b000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9934114f80 con 0x7f9934105860 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f9933e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9934104eb0 0x7f993419b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f9933400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 0x7f993419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f9933400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 0x7f993419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:46826/0 (socket says 172.21.15.28:46826) 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.530+0000 7f9933400700 1 -- 172.21.15.28:0/1605854653 learned_addr learned my addr 172.21.15.28:0/1605854653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9933400700 1 -- 172.21.15.28:0/1605854653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9934109f90 msgr2=0x7f993419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9939000700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9934109f90 0x7f993419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:14.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9933400700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9934109f90 0x7f993419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9933400700 1 -- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9934104eb0 msgr2=0x7f993419b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:14.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9933400700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9934104eb0 0x7f993419b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9933400700 1 -- 172.21.15.28:0/1605854653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99341a0420 con 0x7f9934105860 2024-04-26T01:58:14.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9933e00700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9934104eb0 0x7f993419b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:14.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9939000700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9934109f90 0x7f993419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:58:14.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9933400700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 0x7f993419b9a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9920000c00 tx=0x7f992000f950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:14.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9930c00700 1 -- 172.21.15.28:0/1605854653 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99200050c0 con 0x7f9934105860 2024-04-26T01:58:14.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99341a06b0 con 0x7f9934105860 2024-04-26T01:58:14.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9930c00700 1 -- 172.21.15.28:0/1605854653 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f992000b560 con 0x7f9934105860 2024-04-26T01:58:14.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.531+0000 7f9930c00700 1 -- 172.21.15.28:0/1605854653 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920003a80 con 0x7f9934105860 2024-04-26T01:58:14.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.532+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99341a7f90 con 0x7f9934105860 2024-04-26T01:58:14.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.533+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9934106390 con 0x7f9934105860 2024-04-26T01:58:14.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.535+0000 7f9930c00700 1 -- 172.21.15.28:0/1605854653 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f992000ba10 con 0x7f9934105860 2024-04-26T01:58:14.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.536+0000 7f9930c00700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f99180a3db0 0x7f99180a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:14.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.537+0000 7f9930c00700 1 -- 172.21.15.28:0/1605854653 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8513+0+0 (secure 0 0 0) 0x7f99200d1920 con 0x7f9934105860 2024-04-26T01:58:14.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.537+0000 7f9933e00700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f99180a3db0 0x7f99180a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:14.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.537+0000 7f9933e00700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f99180a3db0 0x7f99180a6270 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9924011490 tx=0x7f9924009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:14.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.538+0000 7f9930c00700 1 -- 172.21.15.28:0/1605854653 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f992005eff0 con 0x7f9934105860 2024-04-26T01:58:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.777+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f993402df80 con 0x7f9934105860 2024-04-26T01:58:14.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.780+0000 7f9930c00700 1 -- 172.21.15.28:0/1605854653 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+23101 (secure 0 0 0) 0x7f9920013030 con 0x7f9934105860 2024-04-26T01:58:14.779 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-26T01:58:14.779 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":115,"fsid":"ead407d4-036b-11ef-bc93-c7b262605968","created":"2024-04-26T01:27:15.883926+0000","modified":"2024-04-26T01:58:13.460568+0000","last_up_change":"2024-04-26T01:35:36.998069+0000","last_in_change":"2024-04-26T01:35:16.870901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:31:11.765874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:36:49.128472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:36:53.237400+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0017e56-53b2-4568-ab76-85ef9640ce85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6803","nonce":3881623886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6805","nonce":3881623886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6809","nonce":3881623886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3881623886},{"type":"v1","addr":"172.21.15.28:6807","nonce":3881623886}]},"public_addr":"172.21.15.28:6803/3881623886","cluster_addr":"172.21.15.28:6805/3881623886","heartbeat_back_addr":"172.21.15.28:6809/3881623886","heartbeat_front_addr":"172.21.15.28:6807/3881623886","state":["exists","up"]},{"osd":1,"uuid":"4bfc13cc-6c4d-45ec-8b43-6a346569692f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6801","nonce":1246761036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6803","nonce":1246761036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6807","nonce":1246761036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1246761036},{"type":"v1","addr":"172.21.15.53:6805","nonce":1246761036}]},"public_addr":"172.21.15.53:6801/1246761036","cluster_addr":"172.21.15.53:6803/1246761036","heartbeat_back_addr":"172.21.15.53:6807/1246761036","heartbeat_front_addr":"172.21.15.53:6805/1246761036","state":["exists","up"]},{"osd":2,"uuid":"147b31a0-5128-4e08-a252-ada0832abff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6801","nonce":3360829572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6803","nonce":3360829572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6807","nonce":3360829572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3360829572},{"type":"v1","addr":"172.21.15.169:6805","nonce":3360829572}]},"public_addr":"172.21.15.169:6801/3360829572","cluster_addr":"172.21.15.169:6803/3360829572","heartbeat_back_addr":"172.21.15.169:6807/3360829572","heartbeat_front_addr":"172.21.15.169:6805/3360829572","state":["exists","up"]},{"osd":3,"uuid":"0929291f-f56e-4c6b-813d-886624278d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6811","nonce":1623836559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6813","nonce":1623836559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6817","nonce":1623836559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1623836559},{"type":"v1","addr":"172.21.15.28:6815","nonce":1623836559}]},"public_addr":"172.21.15.28:6811/1623836559","cluster_addr":"172.21.15.28:6813/1623836559","heartbeat_back_addr":"172.21.15.28:6817/1623836559","heartbeat_front_addr":"172.21.15.28:6815/1623836559","state":["exists","up"]},{"osd":4,"uuid":"3adbfa0d-bced-4cd1-bf74-06e2fd42a015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6809","nonce":3129809581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6811","nonce":3129809581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6815","nonce":3129809581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3129809581},{"type":"v1","addr":"172.21.15.53:6813","nonce":3129809581}]},"public_addr":"172.21.15.53:6809/3129809581","cluster_addr":"172.21.15.53:6811/3129809581","heartbeat_back_addr":"172.21.15.53:6815/3129809581","heartbeat_front_addr":"172.21.15.53:6813/3129809581","state":["exists","up"]},{"osd":5,"uuid":"f9211a41-97a0-4477-8dfb-7354f6020bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6809","nonce":1028981135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6811","nonce":1028981135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6815","nonce":1028981135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1028981135},{"type":"v1","addr":"172.21.15.169:6813","nonce":1028981135}]},"public_addr":"172.21.15.169:6809/1028981135","cluster_addr":"172.21.15.169:6811/1028981135","heartbeat_back_addr":"172.21.15.169:6815/1028981135","heartbeat_front_addr":"172.21.15.169:6813/1028981135","state":["exists","up"]},{"osd":6,"uuid":"05c575f6-3cd7-46be-9c3e-66fcc47b4bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6819","nonce":4283307315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6821","nonce":4283307315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6825","nonce":4283307315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":4283307315},{"type":"v1","addr":"172.21.15.28:6823","nonce":4283307315}]},"public_addr":"172.21.15.28:6819/4283307315","cluster_addr":"172.21.15.28:6821/4283307315","heartbeat_back_addr":"172.21.15.28:6825/4283307315","heartbeat_front_addr":"172.21.15.28:6823/4283307315","state":["exists","up"]},{"osd":7,"uuid":"630a21f2-701f-4b41-8d73-88c86c371714","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6817","nonce":1268967758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6819","nonce":1268967758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6823","nonce":1268967758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1268967758},{"type":"v1","addr":"172.21.15.53:6821","nonce":1268967758}]},"public_addr":"172.21.15.53:6817/1268967758","cluster_addr":"172.21.15.53:6819/1268967758","heartbeat_back_addr":"172.21.15.53:6823/1268967758","heartbeat_front_addr":"172.21.15.53:6821/1268967758","state":["exists","up"]},{"osd":8,"uuid":"1e807449-7760-4d62-801e-bf02535de567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6817","nonce":3120946551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6819","nonce":3120946551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6823","nonce":3120946551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3120946551},{"type":"v1","addr":"172.21.15.169:6821","nonce":3120946551}]},"public_addr":"172.21.15.169:6817/3120946551","cluster_addr":"172.21.15.169:6819/3120946551","heartbeat_back_addr":"172.21.15.169:6823/3120946551","heartbeat_front_addr":"172.21.15.169:6821/3120946551","state":["exists","up"]},{"osd":9,"uuid":"41a7832e-3641-4f51-b26d-c928e268c72f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6827","nonce":2760989310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6829","nonce":2760989310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6833","nonce":2760989310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2760989310},{"type":"v1","addr":"172.21.15.28:6831","nonce":2760989310}]},"public_addr":"172.21.15.28:6827/2760989310","cluster_addr":"172.21.15.28:6829/2760989310","heartbeat_back_addr":"172.21.15.28:6833/2760989310","heartbeat_front_addr":"172.21.15.28:6831/2760989310","state":["exists","up"]},{"osd":10,"uuid":"ef2c5cbd-b772-4910-b007-c50899067c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6825","nonce":786160308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6827","nonce":786160308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6831","nonce":786160308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":786160308},{"type":"v1","addr":"172.21.15.53:6829","nonce":786160308}]},"public_addr":"172.21.15.53:6825/786160308","cluster_addr":"172.21.15.53:6827/786160308","heartbeat_back_addr":"172.21.15.53:6831/786160308","heartbeat_front_addr":"172.21.15.53:6829/786160308","state":["exists","up"]},{"osd":11,"uuid":"673532a1-0742-4a52-82bf-8dd61d64a955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6825","nonce":2005675057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6827","nonce":2005675057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6831","nonce":2005675057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2005675057},{"type":"v1","addr":"172.21.15.169:6829","nonce":2005675057}]},"public_addr":"172.21.15.169:6825/2005675057","cluster_addr":"172.21.15.169:6827/2005675057","heartbeat_back_addr":"172.21.15.169:6831/2005675057","heartbeat_front_addr":"172.21.15.169:6829/2005675057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:20.979675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:30:45.958801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:09.407693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:31:43.785947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:10.654137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:32:35.181087+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:11.061687+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:33:39.173173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:04.978105+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:40.628411+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:08.876948+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:35.185061+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":7,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3431203710":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/1295766352":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3441292878":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3962708174":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/259754378":"2024-04-27T01:28:23.601445+0000","172.21.15.28:6800/3431203710":"2024-04-27T01:40:57.926618+0000","172.21.15.28:6801/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/218486566":"2024-04-27T01:40:57.926618+0000","172.21.15.28:0/3106064028":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/1803582359":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/615407590":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/1017286588":"2024-04-27T01:27:49.755979+0000","172.21.15.28:0/2119832118":"2024-04-27T01:27:49.755979+0000","172.21.15.28:6800/2991743583":"2024-04-27T01:28:23.601445+0000","172.21.15.28:0/3010978828":"2024-04-27T01:40:57.926618+0000","172.21.15.28:6801/1017286588":"2024-04-27T01:27:49.755979+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:58:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f99180a3db0 msgr2=0x7f99180a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f99180a3db0 0x7f99180a6270 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9924011490 tx=0x7f9924009260 comp rx=0 tx=0).stop 2024-04-26T01:58:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 msgr2=0x7f993419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 0x7f993419b9a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9920000c00 tx=0x7f992000f950 comp rx=0 tx=0).stop 2024-04-26T01:58:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 shutdown_connections 2024-04-26T01:58:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f99180a3db0 0x7f99180a6270 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9934109f90 0x7f993419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9934104eb0 0x7f993419b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 --2- 172.21.15.28:0/1605854653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9934105860 0x7f993419b9a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 >> 172.21.15.28:0/1605854653 conn(0x7f9934100740 msgr2=0x7f9934101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 shutdown_connections 2024-04-26T01:58:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:14.783+0000 7f993b000700 1 -- 172.21.15.28:0/1605854653 wait complete. 2024-04-26T01:58:15.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs fail cephfs 2024-04-26T01:58:15.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:15 smithi028 ceph-mon[28247]: pgmap v545: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.5 KiB/s wr, 4 op/s 2024-04-26T01:58:15.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:15 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/1605854653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:58:15.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:15 smithi169 ceph-mon[32851]: pgmap v545: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.5 KiB/s wr, 4 op/s 2024-04-26T01:58:15.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:15 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/1605854653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:58:15.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:15 smithi053 ceph-mon[32593]: pgmap v545: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.5 KiB/s wr, 4 op/s 2024-04-26T01:58:15.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:15 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/1605854653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:58:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:17.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:17 smithi028 ceph-mon[28247]: pgmap v546: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.9 KiB/s wr, 7 op/s 2024-04-26T01:58:17.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:17 smithi169 ceph-mon[32851]: pgmap v546: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.9 KiB/s wr, 7 op/s 2024-04-26T01:58:17.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:17 smithi053 ceph-mon[32593]: pgmap v546: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.9 KiB/s wr, 7 op/s 2024-04-26T01:58:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 -- 172.21.15.28:0/133081859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcdfc06b0f0 msgr2=0x7fcdfc10f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 --2- 172.21.15.28:0/133081859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcdfc06b0f0 0x7fcdfc10f900 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009990 tx=0x7fcdf4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 -- 172.21.15.28:0/133081859 shutdown_connections 2024-04-26T01:58:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 --2- 172.21.15.28:0/133081859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcdfc06b0f0 0x7fcdfc10f900 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 --2- 172.21.15.28:0/133081859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc108a30 0x7fcdfc108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 --2- 172.21.15.28:0/133081859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdfc06a750 0x7fcdfc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 -- 172.21.15.28:0/133081859 >> 172.21.15.28:0/133081859 conn(0x7fcdfc0788f0 msgr2=0x7fcdfc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:18.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 -- 172.21.15.28:0/133081859 shutdown_connections 2024-04-26T01:58:18.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.312+0000 7fce02c00700 1 -- 172.21.15.28:0/133081859 wait complete. 2024-04-26T01:58:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.313+0000 7fce02c00700 1 Processor -- start 2024-04-26T01:58:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.313+0000 7fce02c00700 1 -- start start 2024-04-26T01:58:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fce02c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06a750 0x7fcdfc101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fce02c00700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdfc06b0f0 0x7fcdfc101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fce02c00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcdfc108a30 0x7fcdfc102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fce02c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc112bd0 con 0x7fcdfc06a750 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fce02c00700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc112a50 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fce02c00700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc112d50 con 0x7fcdfc108a30 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fcdfbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06a750 0x7fcdfc101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fcdfbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06a750 0x7fcdfc101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46050/0 (socket says 172.21.15.28:46050) 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fcdfbe00700 1 -- 172.21.15.28:0/413183728 learned_addr learned my addr 172.21.15.28:0/413183728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fcdf3400700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdfc06b0f0 0x7fcdfc101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.314+0000 7fce00c00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcdfc108a30 0x7fcdfc102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf3400700 1 -- 172.21.15.28:0/413183728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcdfc108a30 msgr2=0x7fcdfc102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf3400700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcdfc108a30 0x7fcdfc102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf3400700 1 -- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06a750 msgr2=0x7fcdfc101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf3400700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06a750 0x7fcdfc101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf3400700 1 -- 172.21.15.28:0/413183728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdfc1a75a0 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdfbe00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06a750 0x7fcdfc101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf3400700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdfc06b0f0 0x7fcdfc101f50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcde400e9f0 tx=0x7fcde400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf9600700 1 -- 172.21.15.28:0/413183728 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde400c9c0 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf9600700 1 -- 172.21.15.28:0/413183728 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcde400cb60 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdfc1a7890 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fcdf9600700 1 -- 172.21.15.28:0/413183728 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4018680 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.315+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdfc1a7dd0 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.316+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdfc069730 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.319+0000 7fcdf9600700 1 -- 172.21.15.28:0/413183728 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcde4010430 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.320+0000 7fcdf9600700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcde00a3db0 0x7fcde00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:18.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.320+0000 7fcdf9600700 1 -- 172.21.15.28:0/413183728 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8513+0+0 (secure 0 0 0) 0x7fcde40d0ce0 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.321+0000 7fcdfbe00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcde00a3db0 0x7fcde00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.321+0000 7fcdfbe00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcde00a3db0 0x7fcde00a6270 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009480 tx=0x7fcdec00b040 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.323+0000 7fcdf9600700 1 -- 172.21.15.28:0/413183728 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcde409d320 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.608+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcdfc02df80 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.618+0000 7fcdf9600700 1 -- 172.21.15.28:0/413183728 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) ==== 161+0+0 (secure 0 0 0) 0x7fcde409cc20 con 0x7fcdfc06b0f0 2024-04-26T01:58:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcde00a3db0 msgr2=0x7fcde00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcde00a3db0 0x7fcde00a6270 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009480 tx=0x7fcdec00b040 comp rx=0 tx=0).stop 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdfc06b0f0 msgr2=0x7fcdfc101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdfc06b0f0 0x7fcdfc101f50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcde400e9f0 tx=0x7fcde400ed00 comp rx=0 tx=0).stop 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 shutdown_connections 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fcde00a3db0 0x7fcde00a6270 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcdfc108a30 0x7fcdfc102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06a750 0x7fcdfc101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 --2- 172.21.15.28:0/413183728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdfc06b0f0 0x7fcdfc101f50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.622+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 >> 172.21.15.28:0/413183728 conn(0x7fcdfc0788f0 msgr2=0x7fcdfc0ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.623+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 shutdown_connections 2024-04-26T01:58:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:18.623+0000 7fce02c00700 1 -- 172.21.15.28:0/413183728 wait complete. 2024-04-26T01:58:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T01:58:18.912 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c[63770]: ignoring --setuser ceph since I am not root 2024-04-26T01:58:18.912 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c[63770]: ignoring --setgroup ceph since I am not root 2024-04-26T01:58:18.912 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-mds[63774]: -- 172.21.15.169:0/4201688184 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e38085a40 con 0x557e38058c00 2024-04-26T01:58:18.912 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c[63770]: starting mds.c at 2024-04-26T01:58:18.913 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: ignoring --setuser ceph since I am not root 2024-04-26T01:58:18.913 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: ignoring --setgroup ceph since I am not root 2024-04-26T01:58:18.913 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-mds[64084]: -- 172.21.15.169:0/2633957250 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c8a598ba40 con 0x55c8a595ec00 2024-04-26T01:58:18.913 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: starting mds.i at 2024-04-26T01:58:18.913 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f[63461]: ignoring --setuser ceph since I am not root 2024-04-26T01:58:18.913 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f[63461]: ignoring --setgroup ceph since I am not root 2024-04-26T01:58:18.913 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-mds[63465]: -- 172.21.15.169:0/2541611491 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5613b7207a40 con 0x5613b71da800 2024-04-26T01:58:18.913 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 01:58:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f[63461]: starting mds.f at 2024-04-26T01:58:18.970 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:58:18 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b[62669]: ignoring --setuser ceph since I am not root 2024-04-26T01:58:18.970 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:58:18 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b[62669]: ignoring --setgroup ceph since I am not root 2024-04-26T01:58:18.971 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:58:18 smithi053 ceph-mds[62673]: -- 172.21.15.53:0/1989430549 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5606d3d81a40 con 0x5606d3d54c00 2024-04-26T01:58:18.971 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 01:58:18 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b[62669]: starting mds.b at 2024-04-26T01:58:19.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a[28243]: 2024-04-26T01:58:18.610+0000 7fa86c200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T01:58:19.033 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a[68175]: ignoring --setuser ceph since I am not root 2024-04-26T01:58:19.033 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a[68175]: ignoring --setgroup ceph since I am not root 2024-04-26T01:58:19.033 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-mds[68180]: -- 172.21.15.28:0/1741403174 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55809f1f3a40 con 0x55809f1c7000 2024-04-26T01:58:19.033 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a[68175]: starting mds.a at 2024-04-26T01:58:19.033 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g[67762]: ignoring --setuser ceph since I am not root 2024-04-26T01:58:19.033 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g[67762]: ignoring --setgroup ceph since I am not root 2024-04-26T01:58:19.033 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-mds[67768]: -- 172.21.15.28:0/1306385763 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5558511fba40 con 0x5558511cf000 2024-04-26T01:58:19.033 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 01:58:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g[67762]: starting mds.g at 2024-04-26T01:58:19.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T01:58:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:19 smithi028 ceph-mon[28247]: pgmap v547: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-26T01:58:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:19 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/413183728' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T01:58:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:19 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T01:58:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:19 smithi028 ceph-mon[28247]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T01:58:19.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:19 smithi028 ceph-mon[28247]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T01:58:19.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:19 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T01:58:19.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:19 smithi028 ceph-mon[28247]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T01:58:19.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:19 smithi028 ceph-mon[28247]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T01:58:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:19 smithi169 ceph-mon[32851]: pgmap v547: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-26T01:58:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:19 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/413183728' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T01:58:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:19 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T01:58:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:19 smithi169 ceph-mon[32851]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T01:58:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:19 smithi169 ceph-mon[32851]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T01:58:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:19 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T01:58:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:19 smithi169 ceph-mon[32851]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T01:58:19.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:19 smithi169 ceph-mon[32851]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T01:58:19.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:19 smithi053 ceph-mon[32593]: pgmap v547: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-26T01:58:19.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:19 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/413183728' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T01:58:19.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:19 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T01:58:19.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:19 smithi053 ceph-mon[32593]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T01:58:19.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:19 smithi053 ceph-mon[32593]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T01:58:19.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:19 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T01:58:19.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:19 smithi053 ceph-mon[32593]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T01:58:19.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:19 smithi053 ceph-mon[32593]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T01:58:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:20.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.53:6838/1396139895,v1:172.21.15.53:6839/1396139895] up:boot 2024-04-26T01:58:20.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6834/1088368811,v1:172.21.15.169:6835/1088368811] up:boot 2024-04-26T01:58:20.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6837/2625374233,v1:172.21.15.169:6838/2625374233] up:boot 2024-04-26T01:58:20.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.169:6836/851680291,v1:172.21.15.169:6839/851680291] up:boot 2024-04-26T01:58:20.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.28:6838/2353219779,v1:172.21.15.28:6839/2353219779] up:boot 2024-04-26T01:58:20.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: mds.? [v2:172.21.15.28:6840/799558451,v1:172.21.15.28:6841/799558451] up:boot 2024-04-26T01:58:20.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T01:58:20.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:58:20.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:58:20.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:58:20.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:58:20.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:58:20.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:20 smithi028 ceph-mon[28247]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:58:20.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.53:6838/1396139895,v1:172.21.15.53:6839/1396139895] up:boot 2024-04-26T01:58:20.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6834/1088368811,v1:172.21.15.169:6835/1088368811] up:boot 2024-04-26T01:58:20.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6837/2625374233,v1:172.21.15.169:6838/2625374233] up:boot 2024-04-26T01:58:20.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.169:6836/851680291,v1:172.21.15.169:6839/851680291] up:boot 2024-04-26T01:58:20.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.28:6838/2353219779,v1:172.21.15.28:6839/2353219779] up:boot 2024-04-26T01:58:20.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: mds.? [v2:172.21.15.28:6840/799558451,v1:172.21.15.28:6841/799558451] up:boot 2024-04-26T01:58:20.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T01:58:20.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:58:20.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:58:20.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:58:20.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:58:20.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:58:20.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:20 smithi169 ceph-mon[32851]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:58:20.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.53:6838/1396139895,v1:172.21.15.53:6839/1396139895] up:boot 2024-04-26T01:58:20.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6834/1088368811,v1:172.21.15.169:6835/1088368811] up:boot 2024-04-26T01:58:20.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6837/2625374233,v1:172.21.15.169:6838/2625374233] up:boot 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.169:6836/851680291,v1:172.21.15.169:6839/851680291] up:boot 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.28:6838/2353219779,v1:172.21.15.28:6839/2353219779] up:boot 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: mds.? [v2:172.21.15.28:6840/799558451,v1:172.21.15.28:6841/799558451] up:boot 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:58:20.971 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:20 smithi053 ceph-mon[32593]: from='mgr.25034 172.21.15.28:0/2495197044' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:58:21.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:21 smithi028 ceph-mon[28247]: pgmap v549: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 6 op/s 2024-04-26T01:58:21.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:21 smithi169 ceph-mon[32851]: pgmap v549: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 6 op/s 2024-04-26T01:58:21.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:21 smithi053 ceph-mon[32593]: pgmap v549: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 6 op/s 2024-04-26T01:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 -- 172.21.15.28:0/2611138848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb328107b30 msgr2=0x7fb328105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 --2- 172.21.15.28:0/2611138848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb328107b30 0x7fb328105530 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb31800b0a0 tx=0x7fb31800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 -- 172.21.15.28:0/2611138848 shutdown_connections 2024-04-26T01:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 --2- 172.21.15.28:0/2611138848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb32806b180 0x7fb32806b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 --2- 172.21.15.28:0/2611138848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb328107b30 0x7fb328105530 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 --2- 172.21.15.28:0/2611138848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb328105a70 0x7fb328105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 -- 172.21.15.28:0/2611138848 >> 172.21.15.28:0/2611138848 conn(0x7fb328078740 msgr2=0x7fb328078b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 -- 172.21.15.28:0/2611138848 shutdown_connections 2024-04-26T01:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.279+0000 7fb32e000700 1 -- 172.21.15.28:0/2611138848 wait complete. 2024-04-26T01:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.280+0000 7fb32e000700 1 Processor -- start 2024-04-26T01:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb32e000700 1 -- start start 2024-04-26T01:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb32e000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb32806b180 0x7fb32819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb32e000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328105a70 0x7fb32819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb32e000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb328107b30 0x7fb3281a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb32e000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb328119280 con 0x7fb328105a70 2024-04-26T01:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb32e000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb328119100 con 0x7fb328107b30 2024-04-26T01:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb32e000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb328119400 con 0x7fb32806b180 2024-04-26T01:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb327400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb32806b180 0x7fb32819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb327400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb32806b180 0x7fb32819f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:35462/0 (socket says 172.21.15.28:35462) 2024-04-26T01:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb327400700 1 -- 172.21.15.28:0/2110879087 learned_addr learned my addr 172.21.15.28:0/2110879087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.281+0000 7fb31ea00700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328105a70 0x7fb32819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.282+0000 7fb327e00700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb328107b30 0x7fb3281a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.282+0000 7fb327400700 1 -- 172.21.15.28:0/2110879087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb328107b30 msgr2=0x7fb3281a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.282+0000 7fb327400700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb328107b30 0x7fb3281a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.282+0000 7fb327400700 1 -- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328105a70 msgr2=0x7fb32819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.282+0000 7fb327400700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328105a70 0x7fb32819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.282+0000 7fb327400700 1 -- 172.21.15.28:0/2110879087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3281a4720 con 0x7fb32806b180 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.282+0000 7fb327400700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb32806b180 0x7fb32819f710 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb31800d010 tx=0x7fb318009c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.283+0000 7fb324c00700 1 -- 172.21.15.28:0/2110879087 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318021070 con 0x7fb32806b180 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.283+0000 7fb324c00700 1 -- 172.21.15.28:0/2110879087 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb318007e30 con 0x7fb32806b180 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.283+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3281a49b0 con 0x7fb32806b180 2024-04-26T01:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.283+0000 7fb324c00700 1 -- 172.21.15.28:0/2110879087 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318004670 con 0x7fb32806b180 2024-04-26T01:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.283+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3281ac290 con 0x7fb32806b180 2024-04-26T01:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.284+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb328069730 con 0x7fb32806b180 2024-04-26T01:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.286+0000 7fb324c00700 1 -- 172.21.15.28:0/2110879087 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb318007980 con 0x7fb32806b180 2024-04-26T01:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.288+0000 7fb324c00700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb30c0a3db0 0x7fb30c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.288+0000 7fb324c00700 1 -- 172.21.15.28:0/2110879087 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8727+0+0 (secure 0 0 0) 0x7fb31800f020 con 0x7fb32806b180 2024-04-26T01:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.288+0000 7fb31ea00700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb30c0a3db0 0x7fb30c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.289+0000 7fb31ea00700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb30c0a3db0 0x7fb30c0a6270 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb328101a50 tx=0x7fb310009260 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:22.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.292+0000 7fb324c00700 1 -- 172.21.15.28:0/2110879087 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3180d3e20 con 0x7fb32806b180 2024-04-26T01:58:22.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.579+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb3281ac460 con 0x7fb32806b180 2024-04-26T01:58:22.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.642+0000 7fb324c00700 1 -- 172.21.15.28:0/2110879087 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) ==== 104+0+0 (secure 0 0 0) 0x7fb31802a020 con 0x7fb32806b180 2024-04-26T01:58:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb30c0a3db0 msgr2=0x7fb30c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb30c0a3db0 0x7fb30c0a6270 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb328101a50 tx=0x7fb310009260 comp rx=0 tx=0).stop 2024-04-26T01:58:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb32806b180 msgr2=0x7fb32819f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb32806b180 0x7fb32819f710 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb31800d010 tx=0x7fb318009c90 comp rx=0 tx=0).stop 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 shutdown_connections 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fb30c0a3db0 0x7fb30c0a6270 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb328107b30 0x7fb3281a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb32806b180 0x7fb32819f710 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 --2- 172.21.15.28:0/2110879087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328105a70 0x7fb32819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.645+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 >> 172.21.15.28:0/2110879087 conn(0x7fb328078740 msgr2=0x7fb3280ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.646+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 shutdown_connections 2024-04-26T01:58:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:22.646+0000 7fb32e000700 1 -- 172.21.15.28:0/2110879087 wait complete. 2024-04-26T01:58:22.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:22 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2110879087' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:58:22.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:22 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:58:22.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:22 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2110879087' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:58:22.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:22 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:58:23.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:22 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2110879087' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:58:23.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:22 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:58:23.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T01:58:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:23 smithi028 ceph-mon[28247]: pgmap v550: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 6 op/s 2024-04-26T01:58:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:23 smithi028 ceph-mon[28247]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T01:58:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:23 smithi028 ceph-mon[28247]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T01:58:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:23 smithi028 ceph-mon[28247]: Cluster is now healthy 2024-04-26T01:58:23.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:23 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:58:23.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:23 smithi028 ceph-mon[28247]: fsmap 12 up:standby 2024-04-26T01:58:23.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:23 smithi169 ceph-mon[32851]: pgmap v550: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 6 op/s 2024-04-26T01:58:23.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:23 smithi169 ceph-mon[32851]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T01:58:23.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:23 smithi169 ceph-mon[32851]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T01:58:23.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:23 smithi169 ceph-mon[32851]: Cluster is now healthy 2024-04-26T01:58:23.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:23 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:58:23.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:23 smithi169 ceph-mon[32851]: fsmap 12 up:standby 2024-04-26T01:58:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:23.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:23 smithi053 ceph-mon[32593]: pgmap v550: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 6 op/s 2024-04-26T01:58:23.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:23 smithi053 ceph-mon[32593]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T01:58:23.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:23 smithi053 ceph-mon[32593]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T01:58:23.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:23 smithi053 ceph-mon[32593]: Cluster is now healthy 2024-04-26T01:58:23.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:23 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:58:23.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:23 smithi053 ceph-mon[32593]: fsmap 12 up:standby 2024-04-26T01:58:25.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:25 smithi028 ceph-mon[28247]: pgmap v551: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 409 B/s wr, 4 op/s 2024-04-26T01:58:25.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:25 smithi169 ceph-mon[32851]: pgmap v551: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 409 B/s wr, 4 op/s 2024-04-26T01:58:25.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:25 smithi053 ceph-mon[32593]: pgmap v551: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 409 B/s wr, 4 op/s 2024-04-26T01:58:26.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 -- 172.21.15.28:0/2740834152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688104ec0 msgr2=0x7f96881052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 --2- 172.21.15.28:0/2740834152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688104ec0 0x7f96881052a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f96780099c0 tx=0x7f9678009cd0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 -- 172.21.15.28:0/2740834152 shutdown_connections 2024-04-26T01:58:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 --2- 172.21.15.28:0/2740834152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9688109fa0 0x7f9688111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 --2- 172.21.15.28:0/2740834152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688104ec0 0x7f96881052a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 --2- 172.21.15.28:0/2740834152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9688105870 0x7f96881098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 -- 172.21.15.28:0/2740834152 >> 172.21.15.28:0/2740834152 conn(0x7f9688100750 msgr2=0x7f9688102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 -- 172.21.15.28:0/2740834152 shutdown_connections 2024-04-26T01:58:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.211+0000 7f968f000700 1 -- 172.21.15.28:0/2740834152 wait complete. 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.212+0000 7f968f000700 1 Processor -- start 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.212+0000 7f968f000700 1 -- start start 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.212+0000 7f968f000700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9688104ec0 0x7f9688076f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.212+0000 7f968f000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688105870 0x7f9688077490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f968f000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9688109fa0 0x7f96880779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f968f000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9688114f00 con 0x7f9688105870 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f968f000700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9688114d80 con 0x7f9688109fa0 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f968f000700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9688115080 con 0x7f9688104ec0 2024-04-26T01:58:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f9687e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9688104ec0 0x7f9688076f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f9687e00700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9688104ec0 0x7f9688076f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.28:35482/0 (socket says 172.21.15.28:35482) 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f9687e00700 1 -- 172.21.15.28:0/2514769591 learned_addr learned my addr 172.21.15.28:0/2514769591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f9687400700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688105870 0x7f9688077490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f968d000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9688109fa0 0x7f96880779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f968d000700 1 -- 172.21.15.28:0/2514769591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9688104ec0 msgr2=0x7f9688076f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.213+0000 7f968d000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9688104ec0 0x7f9688076f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f968d000700 1 -- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688105870 msgr2=0x7f9688077490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f968d000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688105870 0x7f9688077490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f968d000700 1 -- 172.21.15.28:0/2514769591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96881abac0 con 0x7f9688109fa0 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f9687e00700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9688104ec0 0x7f9688076f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f9687400700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688105870 0x7f9688077490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f968d000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9688109fa0 0x7f96880779d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9680009f60 tx=0x7f9680009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f9684c00700 1 -- 172.21.15.28:0/2514769591 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96800092e0 con 0x7f9688109fa0 2024-04-26T01:58:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f9684c00700 1 -- 172.21.15.28:0/2514769591 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9680004bb0 con 0x7f9688109fa0 2024-04-26T01:58:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96881abd50 con 0x7f9688109fa0 2024-04-26T01:58:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f9684c00700 1 -- 172.21.15.28:0/2514769591 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680004020 con 0x7f9688109fa0 2024-04-26T01:58:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.214+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96881ac250 con 0x7f9688109fa0 2024-04-26T01:58:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.215+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96881063a0 con 0x7f9688109fa0 2024-04-26T01:58:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.218+0000 7f9684c00700 1 -- 172.21.15.28:0/2514769591 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f968001b030 con 0x7f9688109fa0 2024-04-26T01:58:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.219+0000 7f9684c00700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f966c0a3e30 0x7f966c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.219+0000 7f9684c00700 1 -- 172.21.15.28:0/2514769591 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8727+0+0 (secure 0 0 0) 0x7f96800d0c40 con 0x7f9688109fa0 2024-04-26T01:58:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.219+0000 7f9687e00700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f966c0a3e30 0x7f966c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.220+0000 7f9687e00700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f966c0a3e30 0x7f966c0a62f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f967800b910 tx=0x7f9678003720 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.222+0000 7f9684c00700 1 -- 172.21.15.28:0/2514769591 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9680022020 con 0x7f9688109fa0 2024-04-26T01:58:26.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.457+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f968802e020 con 0x7f9688109fa0 2024-04-26T01:58:26.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.655+0000 7f9684c00700 1 -- 172.21.15.28:0/2514769591 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v117) ==== 181+0+0 (secure 0 0 0) 0x7f9680010040 con 0x7f9688109fa0 2024-04-26T01:58:26.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f966c0a3e30 msgr2=0x7f966c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f966c0a3e30 0x7f966c0a62f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f967800b910 tx=0x7f9678003720 comp rx=0 tx=0).stop 2024-04-26T01:58:26.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9688109fa0 msgr2=0x7f96880779d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9688109fa0 0x7f96880779d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9680009f60 tx=0x7f9680009f90 comp rx=0 tx=0).stop 2024-04-26T01:58:26.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 shutdown_connections 2024-04-26T01:58:26.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7f966c0a3e30 0x7f966c0a62f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9688109fa0 0x7f96880779d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9688104ec0 0x7f9688076f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 --2- 172.21.15.28:0/2514769591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9688105870 0x7f9688077490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 >> 172.21.15.28:0/2514769591 conn(0x7f9688100750 msgr2=0x7f9688102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:26.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.660+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 shutdown_connections 2024-04-26T01:58:26.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:26.661+0000 7f968f000700 1 -- 172.21.15.28:0/2514769591 wait complete. 2024-04-26T01:58:26.661 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-04-26T01:58:26.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:26 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/2514769591' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:26.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:26 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:26.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:26 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/2514769591' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:26.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:26 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:26.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:26 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/2514769591' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:26.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:26 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:27.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid ead407d4-036b-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T01:58:27.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:27 smithi028 ceph-mon[28247]: pgmap v552: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 409 B/s wr, 2 op/s 2024-04-26T01:58:27.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:27 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T01:58:27.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:27 smithi028 ceph-mon[28247]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T01:58:27.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:27 smithi169 ceph-mon[32851]: pgmap v552: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 409 B/s wr, 2 op/s 2024-04-26T01:58:27.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:27 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T01:58:27.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:27 smithi169 ceph-mon[32851]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T01:58:27.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:27 smithi053 ceph-mon[32593]: pgmap v552: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 409 B/s wr, 2 op/s 2024-04-26T01:58:27.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:27 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T01:58:27.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:27 smithi053 ceph-mon[32593]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T01:58:28.022 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:29.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:29 smithi028 ceph-mon[28247]: pgmap v554: 65 pgs: 65 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:58:29.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:29 smithi169 ceph-mon[32851]: pgmap v554: 65 pgs: 65 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:58:29.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:29 smithi053 ceph-mon[32593]: pgmap v554: 65 pgs: 65 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:58:30.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 -- 172.21.15.28:0/1921966646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0601092d0 msgr2=0x7fd0601096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 --2- 172.21.15.28:0/1921966646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0601092d0 0x7fd0601096b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0500099c0 tx=0x7fd050009cd0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 -- 172.21.15.28:0/1921966646 shutdown_connections 2024-04-26T01:58:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 --2- 172.21.15.28:0/1921966646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd06010e3b0 0x7fd060115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 --2- 172.21.15.28:0/1921966646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0601092d0 0x7fd0601096b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 --2- 172.21.15.28:0/1921966646 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd060109c80 0x7fd06010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 -- 172.21.15.28:0/1921966646 >> 172.21.15.28:0/1921966646 conn(0x7fd060076210 msgr2=0x7fd060107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 -- 172.21.15.28:0/1921966646 shutdown_connections 2024-04-26T01:58:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.191+0000 7fd066600700 1 -- 172.21.15.28:0/1921966646 wait complete. 2024-04-26T01:58:30.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.192+0000 7fd066600700 1 Processor -- start 2024-04-26T01:58:30.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd066600700 1 -- start start 2024-04-26T01:58:30.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd066600700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0601092d0 0x7fd06019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:30.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd066600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd060109c80 0x7fd06019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:30.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd066600700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd06010e3b0 0x7fd0601a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:30.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd066600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd060119260 con 0x7fd060109c80 2024-04-26T01:58:30.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd066600700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd0601190e0 con 0x7fd0601092d0 2024-04-26T01:58:30.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd066600700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd0601193e0 con 0x7fd06010e3b0 2024-04-26T01:58:30.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd056a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd060109c80 0x7fd06019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:30.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd056a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd060109c80 0x7fd06019fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39948/0 (socket says 172.21.15.28:39948) 2024-04-26T01:58:30.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd056a00700 1 -- 172.21.15.28:0/3290205666 learned_addr learned my addr 172.21.15.28:0/3290205666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-26T01:58:30.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.193+0000 7fd05fe00700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd06010e3b0 0x7fd0601a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:30.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.194+0000 7fd05f400700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0601092d0 0x7fd06019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.194+0000 7fd05fe00700 1 -- 172.21.15.28:0/3290205666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0601092d0 msgr2=0x7fd06019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.194+0000 7fd05fe00700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0601092d0 0x7fd06019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.194+0000 7fd05fe00700 1 -- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd060109c80 msgr2=0x7fd06019fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.194+0000 7fd05fe00700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd060109c80 0x7fd06019fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.194+0000 7fd05fe00700 1 -- 172.21.15.28:0/3290205666 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0601a4740 con 0x7fd06010e3b0 2024-04-26T01:58:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.194+0000 7fd05fe00700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd06010e3b0 0x7fd0601a4000 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd05800efc0 tx=0x7fd05800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.195+0000 7fd05cc00700 1 -- 172.21.15.28:0/3290205666 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0580092e0 con 0x7fd06010e3b0 2024-04-26T01:58:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.195+0000 7fd05cc00700 1 -- 172.21.15.28:0/3290205666 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd058004bb0 con 0x7fd06010e3b0 2024-04-26T01:58:30.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.195+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0601a49d0 con 0x7fd06010e3b0 2024-04-26T01:58:30.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.195+0000 7fd05cc00700 1 -- 172.21.15.28:0/3290205666 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd058004080 con 0x7fd06010e3b0 2024-04-26T01:58:30.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.195+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0601ac2b0 con 0x7fd06010e3b0 2024-04-26T01:58:30.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.196+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd060069730 con 0x7fd06010e3b0 2024-04-26T01:58:30.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.198+0000 7fd05cc00700 1 -- 172.21.15.28:0/3290205666 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd05801b030 con 0x7fd06010e3b0 2024-04-26T01:58:30.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.200+0000 7fd05cc00700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0440a3e30 0x7fd0440a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:30.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.200+0000 7fd05cc00700 1 -- 172.21.15.28:0/3290205666 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8144+0+0 (secure 0 0 0) 0x7fd0580d0ad0 con 0x7fd06010e3b0 2024-04-26T01:58:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.200+0000 7fd05f400700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0440a3e30 0x7fd0440a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.201+0000 7fd05f400700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0440a3e30 0x7fd0440a62f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0500059d0 tx=0x7fd050005940 comp rx=0 tx=0).ready entity=mgr.25034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:30.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.202+0000 7fd05cc00700 1 -- 172.21.15.28:0/3290205666 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd058022020 con 0x7fd06010e3b0 2024-04-26T01:58:30.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.435+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd06002df80 con 0x7fd06010e3b0 2024-04-26T01:58:30.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.666+0000 7fd05cc00700 1 -- 172.21.15.28:0/3290205666 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v118) ==== 169+0+0 (secure 0 0 0) 0x7fd058010040 con 0x7fd06010e3b0 2024-04-26T01:58:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0440a3e30 msgr2=0x7fd0440a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0440a3e30 0x7fd0440a62f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0500059d0 tx=0x7fd050005940 comp rx=0 tx=0).stop 2024-04-26T01:58:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd06010e3b0 msgr2=0x7fd0601a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd06010e3b0 0x7fd0601a4000 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd05800efc0 tx=0x7fd05800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 shutdown_connections 2024-04-26T01:58:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:6800/4096350009,v1:172.21.15.28:6801/4096350009] conn(0x7fd0440a3e30 0x7fd0440a62f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd06010e3b0 0x7fd0601a4000 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0601092d0 0x7fd06019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 --2- 172.21.15.28:0/3290205666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd060109c80 0x7fd06019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 >> 172.21.15.28:0/3290205666 conn(0x7fd060076210 msgr2=0x7fd06010cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.673+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 shutdown_connections 2024-04-26T01:58:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-26T01:58:30.674+0000 7fd066600700 1 -- 172.21.15.28:0/3290205666 wait complete. 2024-04-26T01:58:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-04-26T01:58:30.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:30 smithi028 ceph-mon[28247]: from='client.? 172.21.15.28:0/3290205666' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:30.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:30 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:30.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:30 smithi053 ceph-mon[32593]: from='client.? 172.21.15.28:0/3290205666' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:30.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:30 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:31.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:30 smithi169 ceph-mon[32851]: from='client.? 172.21.15.28:0/3290205666' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:31.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:30 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T01:58:31.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:58:31.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:58:31.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:58:31.695 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T01:58:31.695 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T01:58:31.767 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T01:58:31.800 DEBUG:teuthology.orchestra.run.smithi169:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T01:58:31.822 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T01:58:31.823 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T01:58:31.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a 2024-04-26T01:58:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:31 smithi028 ceph-mon[28247]: pgmap v555: 65 pgs: 65 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:58:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:31 smithi028 ceph-mon[28247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T01:58:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:31 smithi028 ceph-mon[28247]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T01:58:31.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:31 smithi053 ceph-mon[32593]: pgmap v555: 65 pgs: 65 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:58:31.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:31 smithi053 ceph-mon[32593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T01:58:31.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:31 smithi053 ceph-mon[32593]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T01:58:32.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:31 smithi169 ceph-mon[32851]: pgmap v555: 65 pgs: 65 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:58:32.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:31 smithi169 ceph-mon[32851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T01:58:32.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:31 smithi169 ceph-mon[32851]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T01:58:32.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:31 smithi028 systemd[1]: Stopping Ceph mon.a for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:58:33.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a[28243]: 2024-04-26T01:58:32.564+0000 7fa873000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:58:33.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a[28243]: 2024-04-26T01:58:32.564+0000 7fa873000700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T01:58:33.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:32 smithi028 podman[98036]: 2024-04-26 01:58:32.670873958 +0000 UTC m=+0.303396569 container died 8553833424dfc95a9afec126ca28043a2ce913c7332b1a50934eab3365cce796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-26T01:58:33.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:33 smithi028 podman[98036]: 2024-04-26 01:58:33.153047626 +0000 UTC m=+0.785570249 container cleanup 8553833424dfc95a9afec126ca28043a2ce913c7332b1a50934eab3365cce796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T01:58:33.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:33 smithi028 bash[98036]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a 2024-04-26T01:58:33.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 26 01:58:33 smithi028 podman[98048]: 2024-04-26 01:58:33.396895565 +0000 UTC m=+0.714072283 container remove 8553833424dfc95a9afec126ca28043a2ce913c7332b1a50934eab3365cce796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T01:58:33.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T01:58:33.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:58:33.962 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T01:58:33.962 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T01:58:33.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.b 2024-04-26T01:58:34.405 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:34 smithi053 systemd[1]: Stopping Ceph mon.b for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:58:34.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:34 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b[32589]: 2024-04-26T01:58:34.406+0000 7fc8a03a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:58:34.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:34 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b[32589]: 2024-04-26T01:58:34.406+0000 7fc8a03a6700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T01:58:34.720 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:34 smithi053 podman[69481]: 2024-04-26 01:58:34.652368275 +0000 UTC m=+0.487277433 container died 2cf2123eb492f35494937a26e7ea11caac1d947d0255577099fd19e0ea4b81b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T01:58:35.315 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:35 smithi053 podman[69481]: 2024-04-26 01:58:35.073968241 +0000 UTC m=+0.908877406 container cleanup 2cf2123eb492f35494937a26e7ea11caac1d947d0255577099fd19e0ea4b81b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-26T01:58:35.315 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:35 smithi053 bash[69481]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b 2024-04-26T01:58:35.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T01:58:35.577 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:35 smithi053 podman[69495]: 2024-04-26 01:58:35.317624951 +0000 UTC m=+0.663098908 container remove 2cf2123eb492f35494937a26e7ea11caac1d947d0255577099fd19e0ea4b81b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:58:35.578 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:35 smithi053 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-26T01:58:35.578 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:35 smithi053 systemd[1]: Stopped Ceph mon.b for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:58:35.578 INFO:journalctl@ceph.mon.b.smithi053.stdout:Apr 26 01:58:35 smithi053 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.b.service: Consumed 17.581s CPU time. 2024-04-26T01:58:35.611 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:58:35.612 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T01:58:35.612 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T01:58:35.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.c 2024-04-26T01:58:35.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:35 smithi169 systemd[1]: Stopping Ceph mon.c for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:58:36.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:35 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-c[32847]: 2024-04-26T01:58:35.936+0000 7fb1d5a9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:58:36.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:35 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-c[32847]: 2024-04-26T01:58:35.936+0000 7fb1d5a9d700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T01:58:36.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:36 smithi169 podman[71423]: 2024-04-26 01:58:36.11076468 +0000 UTC m=+0.294241090 container died edd4d4fc4da948ab7eda8eeb57a818d66a21cc0ff0b9c22fefebcbe6cf4d6bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:58:36.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:36 smithi169 podman[71423]: 2024-04-26 01:58:36.480267077 +0000 UTC m=+0.663743486 container cleanup edd4d4fc4da948ab7eda8eeb57a818d66a21cc0ff0b9c22fefebcbe6cf4d6bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:58:36.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Apr 26 01:58:36 smithi169 bash[71423]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mon-c 2024-04-26T01:58:36.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T01:58:36.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:58:36.951 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T01:58:36.951 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T01:58:36.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.x 2024-04-26T01:58:37.284 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:58:37 smithi028 systemd[1]: Stopping Ceph mgr.x for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:58:37.947 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:58:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:58:37.556+0000 7f2f7fe00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:58:37.947 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:58:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x[28473]: 2024-04-26T01:58:37.556+0000 7f2f7fe00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T01:58:37.947 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:58:37 smithi028 podman[98152]: 2024-04-26 01:58:37.651012834 +0000 UTC m=+0.230949050 container died 2e5fee773f32d12ce66150ab282a951475ece1bbac1a17fa1b2dec77b2496710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T01:58:38.283 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:58:38 smithi028 podman[98152]: 2024-04-26 01:58:38.167597493 +0000 UTC m=+0.747533729 container cleanup 2e5fee773f32d12ce66150ab282a951475ece1bbac1a17fa1b2dec77b2496710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T01:58:38.283 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:58:38 smithi028 bash[98152]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x 2024-04-26T01:58:38.783 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 26 01:58:38 smithi028 podman[98166]: 2024-04-26 01:58:38.427407772 +0000 UTC m=+0.758318858 container remove 2e5fee773f32d12ce66150ab282a951475ece1bbac1a17fa1b2dec77b2496710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T01:58:38.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T01:58:39.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:58:39.078 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T01:58:39.078 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T01:58:39.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.y 2024-04-26T01:58:39.416 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:39 smithi053 systemd[1]: Stopping Ceph mgr.y for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:58:39.720 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:39 smithi053 podman[69600]: 2024-04-26 01:58:39.417430761 +0000 UTC m=+0.137590289 container stop 4c2228c33d22bb0a794a33ac5f2179da8e600d61460237d38688b42fb5118e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T01:58:39.720 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:39 smithi053 podman[69600]: 2024-04-26 01:58:39.443683426 +0000 UTC m=+0.163842949 container died 4c2228c33d22bb0a794a33ac5f2179da8e600d61460237d38688b42fb5118e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:58:40.173 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:39 smithi053 podman[69600]: 2024-04-26 01:58:39.98904151 +0000 UTC m=+0.709201032 container cleanup 4c2228c33d22bb0a794a33ac5f2179da8e600d61460237d38688b42fb5118e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:58:40.173 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:39 smithi053 bash[69600]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y 2024-04-26T01:58:40.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T01:58:40.433 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:40 smithi053 podman[69613]: 2024-04-26 01:58:40.175002337 +0000 UTC m=+0.751932425 container remove 4c2228c33d22bb0a794a33ac5f2179da8e600d61460237d38688b42fb5118e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:58:40.434 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:40 smithi053 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-26T01:58:40.434 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:40 smithi053 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-26T01:58:40.434 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:40 smithi053 systemd[1]: Stopped Ceph mgr.y for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:58:40.434 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Apr 26 01:58:40 smithi053 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.y.service: Consumed 23.314s CPU time. 2024-04-26T01:58:40.446 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:58:40.446 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T01:58:40.446 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T01:58:40.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.z 2024-04-26T01:58:40.912 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:58:40 smithi169 systemd[1]: Stopping Ceph mgr.z for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:58:41.727 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:58:41 smithi169 podman[71545]: 2024-04-26 01:58:41.417059944 +0000 UTC m=+0.772779755 container died caca5a9b12e1cc44a49f7bc801a1ef2ef8a276998077da28ff758690fbe128ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T01:58:42.151 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:58:41 smithi169 podman[71545]: 2024-04-26 01:58:41.729275105 +0000 UTC m=+1.084994908 container cleanup caca5a9b12e1cc44a49f7bc801a1ef2ef8a276998077da28ff758690fbe128ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T01:58:42.151 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:58:41 smithi169 bash[71545]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z 2024-04-26T01:58:42.151 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:58:41 smithi169 podman[71559]: 2024-04-26 01:58:41.961003068 +0000 UTC m=+0.545769509 container remove caca5a9b12e1cc44a49f7bc801a1ef2ef8a276998077da28ff758690fbe128ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-26T01:58:42.151 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Apr 26 01:58:41 smithi169 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-26T01:58:42.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T01:58:42.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:58:42.202 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T01:58:42.202 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T01:58:42.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.0 2024-04-26T01:58:42.534 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:42 smithi028 systemd[1]: Stopping Ceph osd.0 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:58:43.283 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:42 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0[40623]: 2024-04-26T01:58:42.814+0000 7f11c3800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:58:43.283 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:42 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0[40623]: 2024-04-26T01:58:42.814+0000 7f11c3800700 -1 osd.0 118 *** Got signal Terminated *** 2024-04-26T01:58:43.283 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:42 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0[40623]: 2024-04-26T01:58:42.814+0000 7f11c3800700 -1 osd.0 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:58:49.238 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:48 smithi028 podman[98272]: 2024-04-26 01:58:48.897631623 +0000 UTC m=+6.230825260 container died c6c032f6c583a60684101c0440a3c1d48109dd87cbd7990ef6942b526b3814fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-26T01:58:49.534 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:49 smithi028 podman[98272]: 2024-04-26 01:58:49.364847831 +0000 UTC m=+6.698041462 container cleanup c6c032f6c583a60684101c0440a3c1d48109dd87cbd7990ef6942b526b3814fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-26T01:58:49.534 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:49 smithi028 bash[98272]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0 2024-04-26T01:58:50.033 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:49 smithi028 podman[98292]: 2024-04-26 01:58:49.595347005 +0000 UTC m=+0.684457089 container remove c6c032f6c583a60684101c0440a3c1d48109dd87cbd7990ef6942b526b3814fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T01:58:50.620 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:50 smithi028 podman[98381]: 2024-04-26 01:58:50.26772283 +0000 UTC m=+0.035628674 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:58:50.620 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:50 smithi028 podman[98381]: 2024-04-26 01:58:50.383889302 +0000 UTC m=+0.151795175 container create 3b5a23e377038bd41b3e08899226d41f3e708fc47ef2034ffa64e318a3415161 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:58:51.033 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:50 smithi028 podman[98381]: 2024-04-26 01:58:50.729974828 +0000 UTC m=+0.497880707 container init 3b5a23e377038bd41b3e08899226d41f3e708fc47ef2034ffa64e318a3415161 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-26T01:58:51.034 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:50 smithi028 podman[98381]: 2024-04-26 01:58:50.741732016 +0000 UTC m=+0.509637915 container start 3b5a23e377038bd41b3e08899226d41f3e708fc47ef2034ffa64e318a3415161 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:58:51.034 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:50 smithi028 podman[98381]: 2024-04-26 01:58:50.851543337 +0000 UTC m=+0.619449209 container attach 3b5a23e377038bd41b3e08899226d41f3e708fc47ef2034ffa64e318a3415161 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T01:58:51.533 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 26 01:58:51 smithi028 podman[98381]: 2024-04-26 01:58:51.214737843 +0000 UTC m=+0.982643746 container died 3b5a23e377038bd41b3e08899226d41f3e708fc47ef2034ffa64e318a3415161 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-26T01:58:51.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T01:58:52.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:58:52.108 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T01:58:52.109 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T01:58:52.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.1 2024-04-26T01:58:52.443 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:52 smithi053 systemd[1]: Stopping Ceph osd.1 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:58:52.720 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1[37222]: 2024-04-26T01:58:52.444+0000 7fd5478ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:58:52.727 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1[37222]: 2024-04-26T01:58:52.444+0000 7fd5478ed700 -1 osd.1 118 *** Got signal Terminated *** 2024-04-26T01:58:52.727 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1[37222]: 2024-04-26T01:58:52.444+0000 7fd5478ed700 -1 osd.1 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:58:58.376 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:58 smithi053 podman[69727]: 2024-04-26 01:58:58.077079903 +0000 UTC m=+5.760987738 container died 61b26e19c9dd4d490cb69bc08279d5029347ed9eaa08b3fa98bac5aab5bbd57b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-26T01:58:58.720 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:58 smithi053 podman[69727]: 2024-04-26 01:58:58.478086685 +0000 UTC m=+6.161994520 container cleanup 61b26e19c9dd4d490cb69bc08279d5029347ed9eaa08b3fa98bac5aab5bbd57b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T01:58:58.720 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:58 smithi053 bash[69727]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1 2024-04-26T01:58:59.095 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:58 smithi053 podman[69744]: 2024-04-26 01:58:58.747445439 +0000 UTC m=+0.666970550 container remove 61b26e19c9dd4d490cb69bc08279d5029347ed9eaa08b3fa98bac5aab5bbd57b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T01:58:59.348 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:59 smithi053 podman[69831]: 2024-04-26 01:58:59.097568962 +0000 UTC m=+0.112496218 container create cd8dba88533562e2e48270dc35961a1887140f70b68bfbcc9e26b6c2969f5475 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T01:58:59.349 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:59 smithi053 podman[69831]: 2024-04-26 01:58:59.004057024 +0000 UTC m=+0.018984295 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:58:59.654 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:59 smithi053 podman[69831]: 2024-04-26 01:58:59.350739669 +0000 UTC m=+0.365666926 container init cd8dba88533562e2e48270dc35961a1887140f70b68bfbcc9e26b6c2969f5475 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-26T01:58:59.654 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:59 smithi053 podman[69831]: 2024-04-26 01:58:59.354654678 +0000 UTC m=+0.369581941 container start cd8dba88533562e2e48270dc35961a1887140f70b68bfbcc9e26b6c2969f5475 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-26T01:58:59.654 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:59 smithi053 podman[69831]: 2024-04-26 01:58:59.442778649 +0000 UTC m=+0.457705921 container attach cd8dba88533562e2e48270dc35961a1887140f70b68bfbcc9e26b6c2969f5475 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T01:58:59.970 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 26 01:58:59 smithi053 podman[69831]: 2024-04-26 01:58:59.656427379 +0000 UTC m=+0.671354636 container died cd8dba88533562e2e48270dc35961a1887140f70b68bfbcc9e26b6c2969f5475 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-04-26T01:59:00.346 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T01:59:00.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:59:00.394 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T01:59:00.394 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T01:59:00.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.2 2024-04-26T01:59:00.707 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:00 smithi169 systemd[1]: Stopping Ceph osd.2 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:59:01.162 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:00 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2[38152]: 2024-04-26T01:59:00.708+0000 7eff0bb88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:59:01.162 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:00 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2[38152]: 2024-04-26T01:59:00.708+0000 7eff0bb88700 -1 osd.2 118 *** Got signal Terminated *** 2024-04-26T01:59:01.162 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:00 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2[38152]: 2024-04-26T01:59:00.708+0000 7eff0bb88700 -1 osd.2 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:59:06.616 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:06 smithi169 podman[71681]: 2024-04-26 01:59:06.336800799 +0000 UTC m=+5.739825057 container died f562c3a0d468fe0bca9a403ecd2ebb681ee0eb2df719e17c3c61f45f71837069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:59:06.912 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:06 smithi169 podman[71681]: 2024-04-26 01:59:06.693249674 +0000 UTC m=+6.096273933 container cleanup f562c3a0d468fe0bca9a403ecd2ebb681ee0eb2df719e17c3c61f45f71837069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:59:06.912 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:06 smithi169 bash[71681]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2 2024-04-26T01:59:07.304 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:06 smithi169 podman[71699]: 2024-04-26 01:59:06.9614959 +0000 UTC m=+0.623797009 container remove f562c3a0d468fe0bca9a403ecd2ebb681ee0eb2df719e17c3c61f45f71837069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T01:59:07.602 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:07 smithi169 podman[71786]: 2024-04-26 01:59:07.209585221 +0000 UTC m=+0.018046866 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:59:07.602 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:07 smithi169 podman[71786]: 2024-04-26 01:59:07.312360595 +0000 UTC m=+0.120822267 container create d1a258db8433941a3c45b55c8081a3347b9d2b567e5b2b1ea90b9cb14f669203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-26T01:59:07.602 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:07 smithi169 podman[71786]: 2024-04-26 01:59:07.604452889 +0000 UTC m=+0.412914519 container init d1a258db8433941a3c45b55c8081a3347b9d2b567e5b2b1ea90b9cb14f669203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T01:59:07.893 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:07 smithi169 podman[71786]: 2024-04-26 01:59:07.607779648 +0000 UTC m=+0.416241283 container start d1a258db8433941a3c45b55c8081a3347b9d2b567e5b2b1ea90b9cb14f669203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-26T01:59:07.893 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:07 smithi169 podman[71786]: 2024-04-26 01:59:07.723458101 +0000 UTC m=+0.531919742 container attach d1a258db8433941a3c45b55c8081a3347b9d2b567e5b2b1ea90b9cb14f669203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:59:08.162 INFO:journalctl@ceph.osd.2.smithi169.stdout:Apr 26 01:59:07 smithi169 podman[71786]: 2024-04-26 01:59:07.895355813 +0000 UTC m=+0.703817442 container died d1a258db8433941a3c45b55c8081a3347b9d2b567e5b2b1ea90b9cb14f669203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-26T01:59:08.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T01:59:08.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:59:08.565 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T01:59:08.566 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T01:59:08.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.3 2024-04-26T01:59:09.033 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:08 smithi028 systemd[1]: Stopping Ceph osd.3 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:59:09.425 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:09 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3[47524]: 2024-04-26T01:59:09.156+0000 7f74f5600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:59:09.425 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:09 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3[47524]: 2024-04-26T01:59:09.156+0000 7f74f5600700 -1 osd.3 118 *** Got signal Terminated *** 2024-04-26T01:59:09.425 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:09 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3[47524]: 2024-04-26T01:59:09.156+0000 7f74f5600700 -1 osd.3 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:59:09.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:09 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:09.425+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:10.162 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:09 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:09.677+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:10.432 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:10 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:10.157+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:10.775 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:10 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:10.468+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:10.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:10.628+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:11.033 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:10 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3[47524]: 2024-04-26T01:59:10.781+0000 7f74f0e00700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.266386+0000 front 2024-04-26T01:58:47.266529+0000 (oldest deadline 2024-04-26T01:59:10.766158+0000) 2024-04-26T01:59:11.418 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:11 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:11.148+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:11.728 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:11 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:11.418+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:11.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:11.640+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:12.033 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:11 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3[47524]: 2024-04-26T01:59:11.759+0000 7f74f0e00700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.266386+0000 front 2024-04-26T01:58:47.266529+0000 (oldest deadline 2024-04-26T01:59:10.766158+0000) 2024-04-26T01:59:12.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:11.837+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:12.395 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:12 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:12.142+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:12.756 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:12 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:12.395+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:13.033 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:12 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3[47524]: 2024-04-26T01:59:12.757+0000 7f74f0e00700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.266386+0000 front 2024-04-26T01:58:47.266529+0000 (oldest deadline 2024-04-26T01:59:10.766158+0000) 2024-04-26T01:59:13.162 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:12 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:12.690+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:13.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:12 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:12.829+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:13.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:13 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:13.161+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:13.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:13 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:13.371+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:13.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:13.650+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:13.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:13.821+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:14.033 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:13 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3[47524]: 2024-04-26T01:59:13.708+0000 7f74f0e00700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.266386+0000 front 2024-04-26T01:58:47.266529+0000 (oldest deadline 2024-04-26T01:59:10.766158+0000) 2024-04-26T01:59:14.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:13 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:13.833+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:14.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:14 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:14.198+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:14.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:14 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:14.325+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:14.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:14.603+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:14.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:14.816+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:15.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:14 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:14.832+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:15.283 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:15 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:15.182+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:15.283 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:15 smithi028 podman[98485]: 2024-04-26 01:59:15.03405078 +0000 UTC m=+5.996428976 container died d5ba3546dd6d213dcc24f039b32467efb3ea3a7bf0962100b65739f8af7cb0d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T01:59:15.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:15 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:15.328+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:15.784 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:15 smithi028 podman[98485]: 2024-04-26 01:59:15.526830413 +0000 UTC m=+6.489208606 container cleanup d5ba3546dd6d213dcc24f039b32467efb3ea3a7bf0962100b65739f8af7cb0d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T01:59:15.784 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:15 smithi028 bash[98485]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3 2024-04-26T01:59:15.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:15.592+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:15.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:15.826+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:16.202 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:15 smithi028 podman[98517]: 2024-04-26 01:59:15.786661808 +0000 UTC m=+0.745383778 container remove d5ba3546dd6d213dcc24f039b32467efb3ea3a7bf0962100b65739f8af7cb0d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T01:59:16.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:15 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:15.829+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:16.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:16 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:16.203+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:16.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:16 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:16.292+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:16.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:16.570+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:16.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:16.802+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:17.033 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:16 smithi028 podman[98604]: 2024-04-26 01:59:16.579333393 +0000 UTC m=+0.040524191 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:59:17.034 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:16 smithi028 podman[98604]: 2024-04-26 01:59:16.703900712 +0000 UTC m=+0.165091534 container create 9288014a6329b5f589771fa92c80ba683b7beea3d14cf78297cca03975d6cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-04-26T01:59:17.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:16 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:16.852+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:17.534 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:17 smithi028 podman[98604]: 2024-04-26 01:59:17.075060072 +0000 UTC m=+0.536250901 container init 9288014a6329b5f589771fa92c80ba683b7beea3d14cf78297cca03975d6cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T01:59:17.534 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:17 smithi028 podman[98604]: 2024-04-26 01:59:17.08711587 +0000 UTC m=+0.548306674 container start 9288014a6329b5f589771fa92c80ba683b7beea3d14cf78297cca03975d6cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T01:59:17.534 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:17 smithi028 podman[98604]: 2024-04-26 01:59:17.190346512 +0000 UTC m=+0.651537391 container attach 9288014a6329b5f589771fa92c80ba683b7beea3d14cf78297cca03975d6cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T01:59:17.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:17 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:17.211+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:17.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:17 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:17.278+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:17.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:17.595+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:17.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:17.771+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:18.005 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:17 smithi028 podman[98604]: 2024-04-26 01:59:17.566906251 +0000 UTC m=+1.028097101 container died 9288014a6329b5f589771fa92c80ba683b7beea3d14cf78297cca03975d6cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T01:59:18.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:17 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:17.860+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:18.260 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:18.231+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:18.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T01:59:18.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:18 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:18.261+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:18.533 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:18 smithi028 podman[98630]: 2024-04-26 01:59:18.410833468 +0000 UTC m=+0.825510859 container remove 9288014a6329b5f589771fa92c80ba683b7beea3d14cf78297cca03975d6cbf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T01:59:18.533 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:18 smithi028 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-26T01:59:18.533 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:18 smithi028 systemd[1]: Stopped Ceph osd.3 for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T01:59:18.533 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 26 01:59:18 smithi028 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.3.service: Consumed 8min 52.081s CPU time. 2024-04-26T01:59:18.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:59:18.594 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T01:59:18.594 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T01:59:18.594 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.4 2024-04-26T01:59:18.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:18.613+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:18.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:18.776+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:18.932 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:18 smithi053 systemd[1]: Stopping Ceph osd.4 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:59:18.932 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:18 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:18.847+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:19.221 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:18 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4[44022]: 2024-04-26T01:59:18.933+0000 7f665646f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:59:19.222 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:18 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4[44022]: 2024-04-26T01:59:18.933+0000 7f665646f700 -1 osd.4 118 *** Got signal Terminated *** 2024-04-26T01:59:19.222 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:18 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4[44022]: 2024-04-26T01:59:18.933+0000 7f665646f700 -1 osd.4 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:59:19.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:19 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:19.212+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:19.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:19 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:19.224+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:19.590 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:19.173+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:19.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:19.758+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:19.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:19.592+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:20.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:19 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:19.876+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:20.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:19 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:19.876+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:20.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:20.161+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:20.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:20 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:20.234+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:20.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:20 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:20.225+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:20.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:20 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:20.225+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:20.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:20.714+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:20.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:20.638+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:20.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:20.638+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:21.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:20 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:20.953+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:21.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:20 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:20.879+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:21.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:20 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:20.879+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:21.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:21 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:21.276+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:21.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:21 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:21.241+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:21.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:21 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:21.241+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:21.657 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:21.178+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:21.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:21.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:21.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:21.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:21.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:21.659+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:21.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:21.659+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:22.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:21 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:21.989+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:22.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:21 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:21.919+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:22.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:21 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:21.919+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:22.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:22 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:22.259+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:22.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:22 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:22.283+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:22.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:22 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:22.283+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:22.616 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:22.213+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:22.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:22.618+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:22.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:22.618+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:22.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:22.680+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:22.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:22.680+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:23.220 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:23 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4[44022]: 2024-04-26T01:59:23.048+0000 7f6652a63700 -1 osd.4 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:57.106696+0000 front 2024-04-26T01:58:57.106742+0000 (oldest deadline 2024-04-26T01:59:23.006930+0000) 2024-04-26T01:59:23.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:23 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:23.012+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:23.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:22 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:22.887+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:23.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:22 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:22.887+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:23.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:23 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:23.227+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:23.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:23 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:23.326+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:23.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:23 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:23.326+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:23.627 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:23 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:23.238+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:23.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:23 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:23.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:23.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:23 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:23.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:23.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:23 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:23.629+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:23.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:23 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:23.629+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:24.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:24 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:24.023+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:24.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:23 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:23.871+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:24.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:23 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:23.871+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:24.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:24 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:24.227+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:24.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:24 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:24.227+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:24.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:24 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:24.352+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:24.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:24 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:24.352+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:24.602 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:24 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:24.196+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:24.825 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:24 smithi053 podman[69951]: 2024-04-26 01:59:24.511433547 +0000 UTC m=+5.720713590 container died 01d309ad14b9d04e8c9065a8575036e73d6e0eb51b0d698e5c9c37ca2958c104 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T01:59:24.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:24 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:24.735+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:24.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:24 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:24.735+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:24.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:24 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:24.604+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:24.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:24 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:24.604+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:25.159 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:25 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:25.006+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:25.159 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:24 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:24.862+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:25.159 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:24 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:24.862+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:25.160 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:24 smithi053 podman[69951]: 2024-04-26 01:59:24.951296385 +0000 UTC m=+6.160576437 container cleanup 01d309ad14b9d04e8c9065a8575036e73d6e0eb51b0d698e5c9c37ca2958c104 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T01:59:25.160 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:24 smithi053 bash[69951]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4 2024-04-26T01:59:25.470 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:25 smithi053 podman[69968]: 2024-04-26 01:59:25.161454936 +0000 UTC m=+0.651013624 container remove 01d309ad14b9d04e8c9065a8575036e73d6e0eb51b0d698e5c9c37ca2958c104 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T01:59:25.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:25.213+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:25.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:25.213+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:25.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:25.379+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:25.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:25 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:25.379+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:25.554 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:25 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:25.202+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:25.903 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:25 smithi053 podman[70057]: 2024-04-26 01:59:25.476653663 +0000 UTC m=+0.018664821 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:59:25.904 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:25 smithi053 podman[70057]: 2024-04-26 01:59:25.587393607 +0000 UTC m=+0.129404745 container create ccaac26ca918eb7f6f3055aac4cea3aecf9255646b13391eedf043d14c66cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T01:59:25.904 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:25 smithi053 podman[70057]: 2024-04-26 01:59:25.815095779 +0000 UTC m=+0.357106918 container init ccaac26ca918eb7f6f3055aac4cea3aecf9255646b13391eedf043d14c66cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-26T01:59:25.904 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:25 smithi053 podman[70057]: 2024-04-26 01:59:25.818339574 +0000 UTC m=+0.360350721 container start ccaac26ca918eb7f6f3055aac4cea3aecf9255646b13391eedf043d14c66cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:59:25.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:25 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:25.557+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:25.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:25 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:25.557+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:25.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:25 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:25.741+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:25.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:25 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:25.741+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:26.221 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:25 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:25.984+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:26.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:25 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:25.903+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:26.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:25 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:25.903+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:26.222 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:25 smithi053 podman[70057]: 2024-04-26 01:59:25.948404783 +0000 UTC m=+0.490415928 container attach ccaac26ca918eb7f6f3055aac4cea3aecf9255646b13391eedf043d14c66cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T01:59:26.222 INFO:journalctl@ceph.osd.4.smithi053.stdout:Apr 26 01:59:26 smithi053 podman[70057]: 2024-04-26 01:59:26.123907492 +0000 UTC m=+0.665918641 container died ccaac26ca918eb7f6f3055aac4cea3aecf9255646b13391eedf043d14c66cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T01:59:26.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:26 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:26.159+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:26.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:26.173+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:26.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:26.173+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:26.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:26.340+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:26.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:26.340+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:26.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T01:59:26.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:59:26.822 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T01:59:26.822 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T01:59:26.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.5 2024-04-26T01:59:26.850 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:26 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:26.598+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:26.850 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:26 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:26.598+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:26.850 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:26 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:26.598+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.747561+0000 front 2024-04-26T01:59:03.747511+0000 (oldest deadline 2024-04-26T01:59:26.548150+0000) 2024-04-26T01:59:26.850 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:26 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:26.693+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:26.850 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:26 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:26.693+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:27.136 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:26 smithi169 systemd[1]: Stopping Ceph osd.5 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:59:27.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:26 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:26.946+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:27.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:26 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:26.898+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:27.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:26 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:26.898+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:27.389 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:27 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:27.138+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:27.389 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:27 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:27.138+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:27.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:27.152+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:27.412 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:27.138+0000 7fb14a2fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:59:27.412 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:27.138+0000 7fb14a2fa700 -1 osd.5 118 *** Got signal Terminated *** 2024-04-26T01:59:27.412 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:27.138+0000 7fb14a2fa700 -1 osd.5 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:59:27.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:27 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:27.390+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:27.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:27 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:27.390+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:27.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:27.625+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:27.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:27.625+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:27.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:27.625+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.747561+0000 front 2024-04-26T01:59:03.747511+0000 (oldest deadline 2024-04-26T01:59:26.548150+0000) 2024-04-26T01:59:27.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:27.650+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:27.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:27.650+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:27.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:27 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:27.650+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:28.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:27 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:27.947+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:28.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:27 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:27.917+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:28.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:27 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:27.917+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:28.363 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:28 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:28.101+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:28.363 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:28 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:28.101+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:28.363 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:28 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:28.101+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:28.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:28 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:28.174+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:28.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:28 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:28.174+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:28.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:28 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:28.363+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:28.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:28 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:28.363+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:28.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:28 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:28.635+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:28.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:28 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:28.635+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:28.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:28 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:28.635+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.747561+0000 front 2024-04-26T01:59:03.747511+0000 (oldest deadline 2024-04-26T01:59:26.548150+0000) 2024-04-26T01:59:28.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:28 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:28.661+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:28.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:28 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:28.661+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:28.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:28 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:28.661+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:29.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:28 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:28.938+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:29.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:28 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:28.913+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:29.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:28 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:28.913+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:29.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:29 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:29.327+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:29.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:29 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:29.327+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:29.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:29 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:29.107+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:29.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:29 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:29.107+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:29.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:29 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:29.107+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:29.653 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:29 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:29.185+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:29.653 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:29 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:29.185+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:29.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:29 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:29.655+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:29.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:29 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:29.655+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:29.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:29 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:29.655+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:29.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:29 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:29.660+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:29.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:29 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:29.660+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:29.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:29 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:29.660+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.747561+0000 front 2024-04-26T01:59:03.747511+0000 (oldest deadline 2024-04-26T01:59:26.548150+0000) 2024-04-26T01:59:30.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:29 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:29.961+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:30.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:29 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:29.961+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:30.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:29 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:29.917+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:30.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:29 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:29.917+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:30.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:29 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:29.917+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:30.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:30 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:30.079+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:30.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:30 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:30.079+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:30.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:30 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:30.079+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:30.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:30 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:30.285+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:30.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:30 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:30.285+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:30.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:30 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:30.285+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:30.653 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:30 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:30.218+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:30.653 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:30 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:30.218+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:30.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:30 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:30.708+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:30.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:30 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:30.708+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:30.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:30 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:30.708+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.747561+0000 front 2024-04-26T01:59:03.747511+0000 (oldest deadline 2024-04-26T01:59:26.548150+0000) 2024-04-26T01:59:30.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:30 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:30.654+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:30.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:30 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:30.654+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:30.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:30 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:30.654+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:31.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:30 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:30.926+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:31.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:30 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:30.926+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:31.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:30 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:30.887+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:31.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:30 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:30.887+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:31.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:30 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:30.887+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:31.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:31.052+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:31.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:31.052+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:31.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:31.052+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:31.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:31.237+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:31.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:31.237+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:31.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:31 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:31.237+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:31.660 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:31 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:31.261+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:31.660 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:31 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:31.261+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:31.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:31 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:31.709+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:43.443757+0000 front 2024-04-26T01:58:43.443482+0000 (oldest deadline 2024-04-26T01:59:09.343329+0000) 2024-04-26T01:59:31.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:31 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:31.709+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.845671+0000 front 2024-04-26T01:58:53.845815+0000 (oldest deadline 2024-04-26T01:59:19.745696+0000) 2024-04-26T01:59:31.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:31 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5[44939]: 2024-04-26T01:59:31.709+0000 7fb1428e6700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.747561+0000 front 2024-04-26T01:59:03.747511+0000 (oldest deadline 2024-04-26T01:59:26.548150+0000) 2024-04-26T01:59:31.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:31 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:31.661+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:31.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:31 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:31.661+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:31.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:31 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:31.661+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:32.136 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:31 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:31.922+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:32.136 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:31 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:31.922+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:32.136 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:31 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:31.886+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:32.136 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:31 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:31.886+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:32.136 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:31 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:31.886+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:32.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:32.059+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:32.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:32.059+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:32.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:32.059+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:32.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:32.255+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:32.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:32.255+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:32.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:32.255+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:32.648 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:32 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:32.220+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:32.649 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:32 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:32.220+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:32.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:32 smithi169 podman[71903]: 2024-04-26 01:59:32.650755488 +0000 UTC m=+5.629472035 container died 2f286c8b402bf0bf15a223311aba64c9659075ac92ff69eea5bb6282c1e3e588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:59:32.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:32 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:32.703+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:32.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:32 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:32.703+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:32.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:32 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:32.703+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:33.197 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:33 smithi169 podman[71903]: 2024-04-26 01:59:33.013586461 +0000 UTC m=+5.992303005 container cleanup 2f286c8b402bf0bf15a223311aba64c9659075ac92ff69eea5bb6282c1e3e588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:59:33.197 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:33 smithi169 bash[71903]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5 2024-04-26T01:59:33.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:32 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:32.920+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:33.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:32 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:32.920+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:33.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:32 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:32.920+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:33.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:32 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:32.881+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:33.221 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:32 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:32.881+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:33.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:33.060+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:33.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:33.060+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:33.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:33.061+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:33.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:33.302+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:33.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:33.302+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:33.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:33 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:33.302+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:33.626 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:33 smithi169 podman[71920]: 2024-04-26 01:59:33.281518575 +0000 UTC m=+0.629073635 container remove 2f286c8b402bf0bf15a223311aba64c9659075ac92ff69eea5bb6282c1e3e588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:59:33.627 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:33 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:33.197+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:33.627 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:33 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:33.197+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:33.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:33 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:33.663+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:33.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:33 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:33.663+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:33.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:33 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:33.663+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:33.912 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:33 smithi169 podman[72010]: 2024-04-26 01:59:33.532213382 +0000 UTC m=+0.018292912 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:59:33.913 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:33 smithi169 podman[72010]: 2024-04-26 01:59:33.673984389 +0000 UTC m=+0.160063904 container create fcf05f7649a2d4c39b0193cc56ecb25e9f97249ecd6c50114e051bad320aca48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T01:59:34.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:33 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:33.874+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:34.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:33 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:33.874+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:34.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:33 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:33.946+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:34.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:33 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:33.946+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:34.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:33 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:33.946+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:34.234 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:33 smithi169 podman[72010]: 2024-04-26 01:59:33.94353569 +0000 UTC m=+0.429615202 container init fcf05f7649a2d4c39b0193cc56ecb25e9f97249ecd6c50114e051bad320aca48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T01:59:34.234 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:33 smithi169 podman[72010]: 2024-04-26 01:59:33.946792492 +0000 UTC m=+0.432872005 container start fcf05f7649a2d4c39b0193cc56ecb25e9f97249ecd6c50114e051bad320aca48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:59:34.234 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:34 smithi169 podman[72010]: 2024-04-26 01:59:34.026906047 +0000 UTC m=+0.512985575 container attach fcf05f7649a2d4c39b0193cc56ecb25e9f97249ecd6c50114e051bad320aca48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T01:59:34.519 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:34 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:34.240+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:34.519 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:34 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:34.240+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:34.519 INFO:journalctl@ceph.osd.5.smithi169.stdout:Apr 26 01:59:34 smithi169 podman[72010]: 2024-04-26 01:59:34.236912178 +0000 UTC m=+0.722991693 container died fcf05f7649a2d4c39b0193cc56ecb25e9f97249ecd6c50114e051bad320aca48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-26T01:59:34.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:34 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:34.099+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:34.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:34 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:34.099+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:34.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:34 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:34.099+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:34.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:34 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:34.283+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:34.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:34 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:34.283+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:34.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:34 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:34.283+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:34.820 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:34 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:34.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:34.821 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:34 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:34.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:34.821 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:34 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:34.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:34.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T01:59:34.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:59:34.922 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T01:59:34.922 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T01:59:34.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.6 2024-04-26T01:59:35.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:34 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:34.903+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:35.221 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:34 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:34.903+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:35.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:34 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:34.985+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:35.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:34 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:34.985+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:35.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:34 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:34.985+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:35.258 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:35 smithi028 systemd[1]: Stopping Ceph osd.6 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:59:35.258 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:35.067+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:35.259 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:35.067+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:35.259 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:35.067+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:35.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:35.259+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:35.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:35.259+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:35.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:35.259+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:35.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:35.502+0000 7f0470e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:59:35.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:35.502+0000 7f0470e00700 -1 osd.6 118 *** Got signal Terminated *** 2024-04-26T01:59:35.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:35 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:35.502+0000 7f0470e00700 -1 osd.6 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:59:35.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:35 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:35.229+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:35.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:35 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:35.229+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:36.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:35 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:35.672+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:36.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:35 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:35.672+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:36.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:35 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:35.672+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:36.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:35 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:35.950+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:36.221 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:35 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:35.950+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:36.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:35 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:35.968+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:36.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:35 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:35.968+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:36.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:35 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:35.968+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:36.306 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:36 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:36.039+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:36.306 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:36 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:36.039+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:36.306 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:36 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:36.039+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:36.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:36 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:36.254+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:36.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:36 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:36.254+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:36.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:36 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:36.307+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:36.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:36 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:36.307+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:36.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:36 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:36.307+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:36.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:36 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:36.307+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:37.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:36 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:36.664+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:37.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:36 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:36.664+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:37.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:36 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:36.664+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:37.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:36 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:36.972+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:37.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:36 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:36.972+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:37.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:36 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:36.966+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:37.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:36 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:36.966+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:37.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:36 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:36.966+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:37.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:37.284+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:37.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:37.284+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:37.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:37.284+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:37.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:37.284+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:37.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:37.040+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:37.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:37.040+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:37.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:37.040+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:37.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:37.040+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:10.846977+0000 front 2024-04-26T01:59:10.846605+0000 (oldest deadline 2024-04-26T01:59:36.746522+0000) 2024-04-26T01:59:37.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:37 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:37.227+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:37.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:37 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:37.227+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:37.634 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:37 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:37.227+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:37.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:37 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:37.634+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:37.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:37 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:37.634+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:37.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:37 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:37.634+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:38.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:37 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:37.959+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:38.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:37 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:37.959+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:38.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:38 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:38.009+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:38.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:38 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:38.009+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:38.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:38 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:38.009+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:38.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:38 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:38.009+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:38.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:38 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:38.051+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:38.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:38 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:38.051+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:38.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:38 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:38.051+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:38.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:38 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:38.052+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:10.846977+0000 front 2024-04-26T01:59:10.846605+0000 (oldest deadline 2024-04-26T01:59:36.746522+0000) 2024-04-26T01:59:38.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:38 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:38.276+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:38.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:38 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:38.276+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:38.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:38 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:38.276+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:38.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:38 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:38.276+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:38.612 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:38 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:38.202+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:38.612 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:38 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:38.202+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:38.612 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:38 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:38.202+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:38.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:38 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:38.613+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:38.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:38 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:38.613+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:38.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:38 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:38.613+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:39.303 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:39 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:39.050+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:39.303 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:39 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:39.050+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:39.303 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:39 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:39.050+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:39.303 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:39 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:39.050+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:10.846977+0000 front 2024-04-26T01:59:10.846605+0000 (oldest deadline 2024-04-26T01:59:36.746522+0000) 2024-04-26T01:59:39.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:39.052+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:39.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:39.052+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:39.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:39.052+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:39.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:39.052+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:39.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:39.003+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:39.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:39.003+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:39.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:39.003+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:39.619 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:39 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:39.185+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:39.620 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:39 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:39.185+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:39.620 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:39 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:39.185+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:39.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:39 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:39.303+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:39.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:39 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:39.303+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:39.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:39 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:39.303+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:39.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:39 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:39.303+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:39.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:39 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:39.621+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:39.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:39 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:39.621+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:39.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:39 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:39.621+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:40.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:40 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:40.138+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:40.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:40 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:40.138+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:40.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:40 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:40.138+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:40.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:39.985+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:40.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:39.985+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:40.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:39 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:39.985+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:40.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:40.017+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:40.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:40.017+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:40.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:40.017+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:40.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:40.017+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:40.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:40.099+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:47.637350+0000 front 2024-04-26T01:58:47.637315+0000 (oldest deadline 2024-04-26T01:59:09.337521+0000) 2024-04-26T01:59:40.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:40.099+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.340870+0000 front 2024-04-26T01:58:53.340730+0000 (oldest deadline 2024-04-26T01:59:23.342356+0000) 2024-04-26T01:59:40.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:40.099+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:03.343142+0000 front 2024-04-26T01:59:03.343461+0000 (oldest deadline 2024-04-26T01:59:27.443161+0000) 2024-04-26T01:59:40.533 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6[54042]: 2024-04-26T01:59:40.099+0000 7f0467600700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:10.846977+0000 front 2024-04-26T01:59:10.846605+0000 (oldest deadline 2024-04-26T01:59:36.746522+0000) 2024-04-26T01:59:40.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:40.288+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:40.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:40.288+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:40.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:40.288+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:40.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:40.288+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:40.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:40 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:40.585+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:40.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:40 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:40.585+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:40.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:40 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:40.585+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:41.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:41 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:41.133+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:41.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:41 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:41.133+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:41.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:41 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:41.133+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:41.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:40.986+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:41.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:40.986+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:41.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:40 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:40.986+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:41.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:41 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:41.002+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:41.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:41 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:41.002+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:41.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:41 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:41.002+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:41.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:41 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:41.002+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:41.634 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:41 smithi028 podman[98709]: 2024-04-26 01:59:41.347423487 +0000 UTC m=+5.988794441 container died 2ecfbd5a52befa93841167b0340160bc6b8e644429eeb8e1d786f20f04dc10ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:59:41.634 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:41 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:41.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:41.634 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:41 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:41.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:41.634 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:41 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:41.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:41.634 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:41 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:41.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:41.685 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:41 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:41.581+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:41.685 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:41 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:41.581+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:41.685 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:41 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:41.581+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:41.975 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:41 smithi028 podman[98709]: 2024-04-26 01:59:41.763556487 +0000 UTC m=+6.404927485 container cleanup 2ecfbd5a52befa93841167b0340160bc6b8e644429eeb8e1d786f20f04dc10ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:59:41.975 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:41 smithi028 bash[98709]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6 2024-04-26T01:59:42.283 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:42 smithi028 podman[98739]: 2024-04-26 01:59:42.05663558 +0000 UTC m=+0.692228888 container remove 2ecfbd5a52befa93841167b0340160bc6b8e644429eeb8e1d786f20f04dc10ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T01:59:42.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:42 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:42.155+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:42.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:42 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:42.155+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:42.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:42 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:42.155+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:42.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:42 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:42.031+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:42.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:42 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:42.031+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:42.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:42 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:42.031+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:42.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:42 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:42.050+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:42.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:42 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:42.050+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:42.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:42 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:42.050+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:42.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:42 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:42.050+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:42.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:42 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:42.295+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:42.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:42 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:42.295+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:42.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:42 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:42.295+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:42.783 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:42 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:42.295+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:42.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:42 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:42.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:42.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:42 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:42.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:42.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:42 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:42.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:43.193 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:42 smithi028 podman[98828]: 2024-04-26 01:59:42.834468758 +0000 UTC m=+0.034930789 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:59:43.193 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:42 smithi028 podman[98828]: 2024-04-26 01:59:42.94932697 +0000 UTC m=+0.149789038 container create 24f5c0184eeb21a3a054bf832f47dbfe176fd0fced29dafed4f979d3a8bd5867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T01:59:43.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:43 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:43.063+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:43.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:43 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:43.063+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:43.221 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:43 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:43.063+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:43.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:43 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:43.088+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:43.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:43 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:43.088+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:43.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:43 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:43.088+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:43.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:43 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:43.088+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:43.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:43 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:43.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:43.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:43 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:43.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:43.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:43 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:43.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:43.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:43 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:43.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:43.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:43 smithi028 podman[98828]: 2024-04-26 01:59:43.286088423 +0000 UTC m=+0.486550535 container init 24f5c0184eeb21a3a054bf832f47dbfe176fd0fced29dafed4f979d3a8bd5867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:59:43.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:43 smithi028 podman[98828]: 2024-04-26 01:59:43.297390807 +0000 UTC m=+0.497852881 container start 24f5c0184eeb21a3a054bf832f47dbfe176fd0fced29dafed4f979d3a8bd5867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:59:43.534 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:43 smithi028 podman[98828]: 2024-04-26 01:59:43.401694067 +0000 UTC m=+0.602156179 container attach 24f5c0184eeb21a3a054bf832f47dbfe176fd0fced29dafed4f979d3a8bd5867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-04-26T01:59:43.640 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:43 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:43.181+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:43.640 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:43 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:43.181+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:43.640 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:43 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:43.181+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:43.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:43 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:43.641+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:43.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:43 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:43.641+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:43.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:43 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:43.641+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:44.033 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 26 01:59:43 smithi028 podman[98828]: 2024-04-26 01:59:43.768511766 +0000 UTC m=+0.968973857 container died 24f5c0184eeb21a3a054bf832f47dbfe176fd0fced29dafed4f979d3a8bd5867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T01:59:44.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:44.112+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:44.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:44.112+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:44.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:44.112+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:44.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:44.112+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:44.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:44.111+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:44.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:44.111+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:44.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:44.111+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:44.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:44 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:44.248+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:44.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:44 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:44.248+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:44.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:44 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:44.248+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:44.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:44 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:44.248+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:44.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T01:59:44.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:44 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:44.202+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:44.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:44 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:44.202+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:44.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:44 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:44.202+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:44.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:59:44.704 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T01:59:44.704 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T01:59:44.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.7 2024-04-26T01:59:45.038 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:44 smithi053 systemd[1]: Stopping Ceph osd.7 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:59:45.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:44 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:44.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:45.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:44 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:44.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:45.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:44 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:44.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:45.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:45.072+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:45.072+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:45.072+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:45.072+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:45.039+0000 7fdd1eafa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:45.039+0000 7fdd1eafa700 -1 osd.7 118 *** Got signal Terminated *** 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:45.039+0000 7fdd1eafa700 -1 osd.7 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:45.113+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:45.113+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:45.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:45 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:45.113+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:45.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:45 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:45.246+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:45.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:45 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:45.246+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:45.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:45 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:45.246+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:45.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:45 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:45.246+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:45.643 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:45 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:45.245+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:45.643 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:45 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:45.245+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:45.644 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:45 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:45.245+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:45.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:45 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:45.645+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:45.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:45 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:45.645+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:45.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:45 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:45.645+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:46.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:46.164+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:46.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:46.164+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:46.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:46.164+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:46.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:46.164+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:22.892393+0000 front 2024-04-26T01:59:22.892443+0000 (oldest deadline 2024-04-26T01:59:45.792584+0000) 2024-04-26T01:59:46.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:46.030+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:46.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:46.030+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:46.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:46.030+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:46.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:46 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:46.030+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:46.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:46 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:46.251+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:46.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:46 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:46.251+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:46.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:46 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:46.251+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:46.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:46 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:46.251+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:46.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:46 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:46.251+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:46.607 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:46.275+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:46.607 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:46.275+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:46.607 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:46.275+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:46.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:46.608+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:46.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:46.608+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:46.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:46 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:46.608+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:47.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:47.187+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:47.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:47.187+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:47.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:47.187+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:47.471 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:47.187+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:22.892393+0000 front 2024-04-26T01:59:22.892443+0000 (oldest deadline 2024-04-26T01:59:45.792584+0000) 2024-04-26T01:59:47.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.004+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:47.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.004+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:47.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.004+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:47.471 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.004+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:47.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:47 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:47.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:47.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:47 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:47.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:47.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:47 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:47.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:47.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:47 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:47.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:47.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:47 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:47.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:47.657 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:47.318+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:47.657 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:47.319+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:47.657 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:47.319+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:47.657 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:47.319+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:47.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:47.658+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:47.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:47.658+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:47.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:47.658+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:47.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:47 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:47.658+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:48.232 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.973+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:48.233 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.973+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:48.233 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.973+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:48.233 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.973+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:48.233 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:47.973+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:48.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:48.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:48.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:48.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:48.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:48.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:48.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:48.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:48.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:48 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:48.258+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:48.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:48.280+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:48.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:48.280+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:48.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:48.280+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:48.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:48.280+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:48.720 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:48.234+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:48.720 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:48.234+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:48.720 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:48.234+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:48.720 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:48.234+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:22.892393+0000 front 2024-04-26T01:59:22.892443+0000 (oldest deadline 2024-04-26T01:59:45.792584+0000) 2024-04-26T01:59:49.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:48.685+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:49.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:48.685+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:49.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:48.685+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:49.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:48 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:48.685+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:49.220 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:49.204+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.787755+0000 front 2024-04-26T01:58:56.787625+0000 (oldest deadline 2024-04-26T01:59:20.887798+0000) 2024-04-26T01:59:49.221 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:49.204+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.588779+0000 front 2024-04-26T01:59:05.588719+0000 (oldest deadline 2024-04-26T01:59:29.688839+0000) 2024-04-26T01:59:49.221 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:49.204+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.689882+0000 front 2024-04-26T01:59:09.689682+0000 (oldest deadline 2024-04-26T01:59:38.390407+0000) 2024-04-26T01:59:49.221 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7[50667]: 2024-04-26T01:59:49.204+0000 7fdd170e6700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:22.892393+0000 front 2024-04-26T01:59:22.892443+0000 (oldest deadline 2024-04-26T01:59:45.792584+0000) 2024-04-26T01:59:49.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:48.957+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:49.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:48.957+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:49.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:48.957+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:49.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:48.957+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:49.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:48 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:48.957+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:49.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:49.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:49.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:49.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:49.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:49.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:49.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:49.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:49.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:49 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:49.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:49.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:49.248+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:49.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:49.248+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:49.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:49.248+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:49.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:49.248+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:50.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:49.724+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:50.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:49.724+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:50.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:49.724+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:50.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:49 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:49.724+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:50.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:49.921+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:50.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:49.921+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:50.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:49.921+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:50.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:49.921+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:50.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:49 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:49.921+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:50.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:50 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:50.207+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:50.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:50 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:50.207+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:50.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:50 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:50.207+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:50.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:50 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:50.207+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:50.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:50 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:50.207+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:50.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:50.216+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:50.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:50.216+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:50.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:50.216+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:50.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:50.216+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:50.854 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:50 smithi053 podman[70175]: 2024-04-26 01:59:50.544625715 +0000 UTC m=+5.629284908 container died 77b4b24cc1e653e9e14a84962ca2442af7357768137f4b2f2c3fa7475099361f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T01:59:51.147 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:50 smithi053 podman[70175]: 2024-04-26 01:59:50.955964849 +0000 UTC m=+6.040624041 container cleanup 77b4b24cc1e653e9e14a84962ca2442af7357768137f4b2f2c3fa7475099361f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-26T01:59:51.147 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:50 smithi053 bash[70175]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7 2024-04-26T01:59:51.147 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:50.948+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:51.147 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:50.948+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:51.147 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:50.948+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:51.147 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:50.948+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:51.147 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:50 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:50.948+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:51.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:50.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:51.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:50.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:51.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:50.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:51.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:50 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:50.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:51.470 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:51 smithi053 podman[70193]: 2024-04-26 01:59:51.14881544 +0000 UTC m=+0.607970181 container remove 77b4b24cc1e653e9e14a84962ca2442af7357768137f4b2f2c3fa7475099361f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:59:51.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:51 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:51.226+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:51.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:51 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:51.226+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:51.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:51 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:51.226+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:51.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:51 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:51.226+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:51.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:51 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:51.226+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:51.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:51.174+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:51.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:51.174+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:51.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:51.174+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:51.662 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:51.174+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:51.829 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:51 smithi053 podman[70285]: 2024-04-26 01:59:51.482045964 +0000 UTC m=+0.018489866 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:59:51.830 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:51 smithi053 podman[70285]: 2024-04-26 01:59:51.591746711 +0000 UTC m=+0.128190618 container create 19890d955d714a9a5f57105973789d25bada10e68d1d924ad31b299a00b11c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-26T01:59:51.830 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:51 smithi053 podman[70285]: 2024-04-26 01:59:51.828160034 +0000 UTC m=+0.364603946 container init 19890d955d714a9a5f57105973789d25bada10e68d1d924ad31b299a00b11c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T01:59:52.127 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:51.780+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:52.127 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:51.780+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:52.127 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:51.780+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:52.127 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:51 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:51.780+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:52.135 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:51 smithi053 podman[70285]: 2024-04-26 01:59:51.831931625 +0000 UTC m=+0.368375529 container start 19890d955d714a9a5f57105973789d25bada10e68d1d924ad31b299a00b11c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T01:59:52.135 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:51 smithi053 podman[70285]: 2024-04-26 01:59:51.953068615 +0000 UTC m=+0.489512529 container attach 19890d955d714a9a5f57105973789d25bada10e68d1d924ad31b299a00b11c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-26T01:59:52.135 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:51.911+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:52.135 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:51.911+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:52.136 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:51.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:52.136 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:51.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:52.136 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:51.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:52.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:52.165+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:52.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:52.165+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:52.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:52.165+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:52.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:52.165+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:52.418 INFO:journalctl@ceph.osd.7.smithi053.stdout:Apr 26 01:59:52 smithi053 podman[70285]: 2024-04-26 01:59:52.137948497 +0000 UTC m=+0.674392408 container died 19890d955d714a9a5f57105973789d25bada10e68d1d924ad31b299a00b11c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:59:52.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:52.264+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:52.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:52.264+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:52.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:52.264+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:52.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:52.264+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:52.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:52 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:52.264+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:52.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T01:59:52.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T01:59:52.870 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T01:59:52.870 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T01:59:52.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.8 2024-04-26T01:59:53.131 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:52 smithi169 systemd[1]: Stopping Ceph osd.8 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T01:59:53.131 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:52.799+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:53.131 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:52.799+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:53.131 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:52.799+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:53.131 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:52 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:52.799+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:53.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:52.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:53.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:52.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:53.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:52.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:53.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:52.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:53.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:52.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:53.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:52 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:52.912+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T01:59:53.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:53.132+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:53.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:53.132+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:53.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:53.132+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:53.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:53.132+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:53.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:53.183+0000 7f6dedf0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:59:53.413 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:53.183+0000 7f6dedf0d700 -1 osd.8 118 *** Got signal Terminated *** 2024-04-26T01:59:53.413 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:53.183+0000 7f6dedf0d700 -1 osd.8 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T01:59:53.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:53 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:53.231+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:53.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:53 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:53.231+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:53.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:53 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:53.231+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:53.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:53 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:53.231+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:53.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:53 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:53.231+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:54.143 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:53.782+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:54.143 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:53.782+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:54.143 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:53.782+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:54.143 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:53.782+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:54.143 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:53 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:53.782+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T01:59:54.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:53.879+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:54.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:53.879+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:54.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:53.879+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:54.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:53.879+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:54.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:53.879+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:54.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:53 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:53.879+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T01:59:54.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:54.144+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:54.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:54.144+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:54.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:54.144+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:54.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:54.144+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:54.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:54.144+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:29.945558+0000 front 2024-04-26T01:59:29.945504+0000 (oldest deadline 2024-04-26T01:59:54.045559+0000) 2024-04-26T01:59:54.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:54.240+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:54.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:54.240+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:54.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:54.240+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:54.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:54.240+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:54.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:54.240+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:54.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:54 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:54.240+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T01:59:55.130 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:54.754+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:55.130 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:54.754+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:55.130 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:54.754+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:55.130 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:54.754+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:55.130 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:54 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:54.754+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T01:59:55.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:54.836+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:55.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:54.836+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:55.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:54.836+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:55.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:54.836+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:55.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:54.836+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:55.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:54 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:54.836+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T01:59:55.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:55.131+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:55.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:55.131+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:55.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:55.131+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:55.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:55.131+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:55.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:55.131+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:29.945558+0000 front 2024-04-26T01:59:29.945504+0000 (oldest deadline 2024-04-26T01:59:54.045559+0000) 2024-04-26T01:59:55.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:55.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:55.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:55.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:55.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:55.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:55.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:55.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:55.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:55.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:55.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:55 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:55.232+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T01:59:56.107 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:55.741+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:56.107 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:55.741+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:56.107 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:55.741+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:56.107 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:55.741+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:56.107 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:55 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:55.741+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T01:59:56.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:55.801+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:56.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:55.801+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:56.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:55.801+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:56.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:55.801+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:56.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:55.801+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:56.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:55 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:55.801+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T01:59:56.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:56.108+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:56.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:56.108+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:56.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:56.108+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:56.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:56.108+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:56.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:56.108+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:29.945558+0000 front 2024-04-26T01:59:29.945504+0000 (oldest deadline 2024-04-26T01:59:54.045559+0000) 2024-04-26T01:59:56.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:56.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:56.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:56.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:56.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:56.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:56.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:56.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:56.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:56.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:56.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:56 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:56.223+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T01:59:57.061 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:56.792+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:57.061 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:56.792+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:57.061 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:56.792+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:57.061 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:56.792+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:57.061 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:56 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:56.792+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T01:59:57.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:56.808+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:57.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:56.808+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:57.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:56.808+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:57.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:56.808+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:57.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:56.808+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:57.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:56.808+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T01:59:57.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:57.061+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:57.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:57.061+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:57.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:57.061+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:57.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:57.061+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:57.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:57.061+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:29.945558+0000 front 2024-04-26T01:59:29.945504+0000 (oldest deadline 2024-04-26T01:59:54.045559+0000) 2024-04-26T01:59:57.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:57 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:57.271+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:57.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:57 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:57.271+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:57.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:57 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:57.271+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:57.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:57 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:57.271+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:57.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:57 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:57.271+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:57.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:57 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:57.271+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T01:59:58.096 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:57.812+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:58.097 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:57.812+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:58.097 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:57.812+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:58.097 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:57.812+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:58.097 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:57 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:57.812+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T01:59:58.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:57 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:57.793+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:58.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:57 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:57.793+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:58.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:57 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:57.793+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:58.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:57 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:57.793+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:58.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:57 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:57.793+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:58.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:57 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:57.793+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T01:59:58.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:58.098+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:53.839224+0000 front 2024-04-26T01:58:53.839267+0000 (oldest deadline 2024-04-26T01:59:19.039977+0000) 2024-04-26T01:59:58.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:58.098+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:02.541263+0000 front 2024-04-26T01:59:02.541295+0000 (oldest deadline 2024-04-26T01:59:27.241464+0000) 2024-04-26T01:59:58.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:58.098+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:13.043231+0000 front 2024-04-26T01:59:13.043009+0000 (oldest deadline 2024-04-26T01:59:36.542741+0000) 2024-04-26T01:59:58.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:58.098+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:18.843984+0000 front 2024-04-26T01:59:18.844119+0000 (oldest deadline 2024-04-26T01:59:47.044671+0000) 2024-04-26T01:59:58.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8[51660]: 2024-04-26T01:59:58.098+0000 7f6dea501700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:29.945558+0000 front 2024-04-26T01:59:29.945504+0000 (oldest deadline 2024-04-26T01:59:54.045559+0000) 2024-04-26T01:59:58.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:58.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:58.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:58.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:58.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:58.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:58.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:58.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:58.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:58.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:58.784 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:58 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:58.290+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T01:59:59.022 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:58 smithi169 podman[72129]: 2024-04-26 01:59:58.710439109 +0000 UTC m=+5.645943712 container died 44d8ccfc3774194cee3e8cd0fe90b2e7d461378b4f5877544598f478bb6b34ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T01:59:59.022 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:59 smithi169 podman[72129]: 2024-04-26 01:59:59.024624815 +0000 UTC m=+5.960129415 container cleanup 44d8ccfc3774194cee3e8cd0fe90b2e7d461378b4f5877544598f478bb6b34ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T01:59:59.022 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:58.769+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:59.023 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:58.769+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:59.023 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:58.769+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:59.023 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:58.769+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:59.023 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:58 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:58.769+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T01:59:59.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:58 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:58.777+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T01:59:59.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:58 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:58.777+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T01:59:59.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:58 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:58.777+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T01:59:59.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:58 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:58.777+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T01:59:59.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:58 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:58.777+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T01:59:59.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:58 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:58.777+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T01:59:59.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:59 smithi169 bash[72129]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8 2024-04-26T01:59:59.412 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:59 smithi169 podman[72146]: 2024-04-26 01:59:59.242980029 +0000 UTC m=+0.530778472 container remove 44d8ccfc3774194cee3e8cd0fe90b2e7d461378b4f5877544598f478bb6b34ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:59:59.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:59 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:59.261+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T01:59:59.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:59 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:59.261+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T01:59:59.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:59 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:59.261+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T01:59:59.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:59 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:59.261+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T01:59:59.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:59 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:59.261+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T01:59:59.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 01:59:59 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T01:59:59.261+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T01:59:59.852 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:59 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:59.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:59 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:59.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:59 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:59.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:59 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:59.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 01:59:59 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T01:59:59.733+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:59 smithi169 podman[72237]: 2024-04-26 01:59:59.439959857 +0000 UTC m=+0.018478986 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:59 smithi169 podman[72237]: 2024-04-26 01:59:59.560592657 +0000 UTC m=+0.139111818 container create e502533a4d15e967e0b5cd61714a70b7d2ce07b5d0df0d58e5ba9b5a926d0610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:59 smithi169 podman[72237]: 2024-04-26 01:59:59.77958355 +0000 UTC m=+0.358102668 container init e502533a4d15e967e0b5cd61714a70b7d2ce07b5d0df0d58e5ba9b5a926d0610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:59 smithi169 podman[72237]: 2024-04-26 01:59:59.78285678 +0000 UTC m=+0.361375896 container start e502533a4d15e967e0b5cd61714a70b7d2ce07b5d0df0d58e5ba9b5a926d0610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T01:59:59.853 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 01:59:59 smithi169 podman[72237]: 2024-04-26 01:59:59.854391611 +0000 UTC m=+0.432910739 container attach e502533a4d15e967e0b5cd61714a70b7d2ce07b5d0df0d58e5ba9b5a926d0610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:00:00.162 INFO:journalctl@ceph.osd.8.smithi169.stdout:Apr 26 02:00:00 smithi169 podman[72237]: 2024-04-26 02:00:00.076813648 +0000 UTC m=+0.655332765 container died e502533a4d15e967e0b5cd61714a70b7d2ce07b5d0df0d58e5ba9b5a926d0610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:00:00.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:59 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:59.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:00.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:59 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:59.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:00.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:59 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:59.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:00.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:59 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:59.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:00.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:59 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:59.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:00.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 01:59:59 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T01:59:59.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:00.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:00.225+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T02:00:00.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:00.225+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T02:00:00.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:00.225+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T02:00:00.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:00.225+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T02:00:00.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:00.225+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T02:00:00.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:00 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:00.225+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T02:00:00.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T02:00:00.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:00.729 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T02:00:00.730 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T02:00:00.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.9 2024-04-26T02:00:01.122 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:00 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:00.735+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:01.122 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:00 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:00.735+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:01.122 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:00 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:00.735+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:01.122 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:00 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:00.735+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:01.122 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:00 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:00.735+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:01.122 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:00 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:00.735+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:01.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:00 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:00.709+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:01.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:00 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:00.709+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:01.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:00 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:00.709+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:01.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:00 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:00.709+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:01.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:00 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:00.709+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:01.201 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:00 smithi028 systemd[1]: Stopping Ceph osd.9 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:01.201 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.201+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T02:00:01.201 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.201+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T02:00:01.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.201+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T02:00:01.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.201+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T02:00:01.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.201+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T02:00:01.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.201+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T02:00:01.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.314+0000 7f62ef200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:01.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.314+0000 7f62ef200700 -1 osd.9 118 *** Got signal Terminated *** 2024-04-26T02:00:01.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:01 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:01.314+0000 7f62ef200700 -1 osd.9 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:00:02.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:01 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:01.722+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:02.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:01 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:01.722+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:02.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:01 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:01.722+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:02.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:01 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:01.722+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:02.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:01 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:01.722+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:02.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:01 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:01.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:02.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:01 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:01.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:02.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:01 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:01.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:02.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:01 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:01.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:02.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:01 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:01.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:02.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:01 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:01.748+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:02.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:02 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:02.152+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T02:00:02.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:02 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:02.152+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T02:00:02.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:02 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:02.152+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T02:00:02.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:02 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:02.152+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T02:00:02.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:02 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:02.152+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T02:00:02.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:02 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:02.152+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T02:00:02.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:02 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:02.152+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.829739+0000 front 2024-04-26T01:59:38.829919+0000 (oldest deadline 2024-04-26T02:00:01.729259+0000) 2024-04-26T02:00:03.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:02 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:02.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:03.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:02 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:02.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:03.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:02 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:02.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:03.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:02 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:02.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:03.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:02 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:02.689+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:03.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:02 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:02.768+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:03.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:02 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:02.768+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:03.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:02 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:02.768+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:03.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:02 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:02.768+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:03.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:02 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:02.768+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:03.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:02 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:02.768+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:03.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:03 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:03.128+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T02:00:03.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:03 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:03.128+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T02:00:03.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:03 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:03.128+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T02:00:03.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:03 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:03.128+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T02:00:03.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:03 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:03.128+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T02:00:03.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:03 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:03.128+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T02:00:03.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:03 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:03.128+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.829739+0000 front 2024-04-26T01:59:38.829919+0000 (oldest deadline 2024-04-26T02:00:01.729259+0000) 2024-04-26T02:00:03.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:03 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:03.654+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:03.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:03 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:03.654+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:03.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:03 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:03.654+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:03.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:03 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:03.654+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:03.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:03 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:03.654+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:04.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:03 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:03.746+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:04.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:03 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:03.746+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:04.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:03 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:03.746+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:04.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:03 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:03.746+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:04.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:03 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:03.746+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:04.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:03 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:03.746+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:04.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:04 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:04.126+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T02:00:04.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:04 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:04.126+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T02:00:04.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:04 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:04.126+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T02:00:04.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:04 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:04.126+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T02:00:04.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:04 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:04.126+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T02:00:04.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:04 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:04.126+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T02:00:04.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:04 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:04.126+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.829739+0000 front 2024-04-26T01:59:38.829919+0000 (oldest deadline 2024-04-26T02:00:01.729259+0000) 2024-04-26T02:00:05.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:04 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:04.668+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:05.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:04 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:04.668+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:05.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:04 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:04.668+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:05.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:04 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:04.668+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:05.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:04 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:04.668+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:05.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:04 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:04.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:05.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:04 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:04.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:05.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:04 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:04.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:05.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:04 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:04.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:05.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:04 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:04.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:05.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:04 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:04.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:05.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:04 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:04.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:05.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:05 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:05.165+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T02:00:05.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:05 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:05.165+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T02:00:05.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:05 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:05.165+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T02:00:05.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:05 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:05.165+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T02:00:05.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:05 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:05.165+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T02:00:05.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:05 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:05.165+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T02:00:05.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:05 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:05.165+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.829739+0000 front 2024-04-26T01:59:38.829919+0000 (oldest deadline 2024-04-26T02:00:01.729259+0000) 2024-04-26T02:00:06.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:05 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:05.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:06.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:05 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:05.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:06.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:05 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:05.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:06.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:05 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:05.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:06.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:05 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:05.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:06.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:05 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:05.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:06.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:05 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:05.816+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:06.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:05 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:05.816+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:06.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:05 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:05.816+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:06.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:05 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:05.816+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:06.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:05 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:05.816+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:06.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:05 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:05.816+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:06.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:05 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:05.816+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:06.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:06 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:06.131+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.412121+0000 front 2024-04-26T01:58:46.411719+0000 (oldest deadline 2024-04-26T01:59:09.311847+0000) 2024-04-26T02:00:06.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:06 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:06.131+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.316050+0000 front 2024-04-26T01:58:56.315644+0000 (oldest deadline 2024-04-26T01:59:19.815702+0000) 2024-04-26T02:00:06.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:06 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:06.131+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.019174+0000 front 2024-04-26T01:59:05.019009+0000 (oldest deadline 2024-04-26T01:59:29.718985+0000) 2024-04-26T02:00:06.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:06 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:06.131+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:09.720163+0000 front 2024-04-26T01:59:09.720412+0000 (oldest deadline 2024-04-26T01:59:35.620443+0000) 2024-04-26T02:00:06.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:06 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:06.131+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.124695+0000 front 2024-04-26T01:59:23.124596+0000 (oldest deadline 2024-04-26T01:59:46.024672+0000) 2024-04-26T02:00:06.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:06 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:06.131+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.926800+0000 front 2024-04-26T01:59:28.926746+0000 (oldest deadline 2024-04-26T01:59:53.527792+0000) 2024-04-26T02:00:06.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:06 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9[61057]: 2024-04-26T02:00:06.131+0000 7f62eaa00700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.829739+0000 front 2024-04-26T01:59:38.829919+0000 (oldest deadline 2024-04-26T02:00:01.729259+0000) 2024-04-26T02:00:07.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:06 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:06.701+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:07.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:06 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:06.701+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:07.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:06 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:06.701+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:07.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:06 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:06.702+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:07.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:06 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:06.702+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:07.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:06 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:06.702+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:07.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:06 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:06.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:07.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:06 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:06.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:07.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:06 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:06.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:07.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:06 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:06.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:07.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:06 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:06.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:07.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:06 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:06.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:07.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:06 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:06.785+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:07.348 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:07 smithi028 podman[98934]: 2024-04-26 02:00:07.089681217 +0000 UTC m=+5.928137900 container died c9c117b985fc0f82b18f497a4305e19320953bb6499f46a609cb13ec64dbbadf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T02:00:07.608 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:07 smithi028 podman[98934]: 2024-04-26 02:00:07.449476307 +0000 UTC m=+6.287933010 container cleanup c9c117b985fc0f82b18f497a4305e19320953bb6499f46a609cb13ec64dbbadf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T02:00:07.608 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:07 smithi028 bash[98934]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9 2024-04-26T02:00:08.033 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:07 smithi028 podman[98954]: 2024-04-26 02:00:07.608587362 +0000 UTC m=+0.504772692 container remove c9c117b985fc0f82b18f497a4305e19320953bb6499f46a609cb13ec64dbbadf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T02:00:08.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:07 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:07.686+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:08.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:07 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:07.686+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:08.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:07 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:07.686+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:08.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:07 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:07.686+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:08.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:07 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:07.686+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:08.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:07 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:07.686+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:08.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:07 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:07.795+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:08.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:07 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:07.795+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:08.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:07 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:07.795+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:08.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:07 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:07.795+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:08.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:07 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:07.795+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:08.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:07 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:07.795+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:08.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:07 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:07.795+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:08.754 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:08 smithi028 podman[99043]: 2024-04-26 02:00:08.390396911 +0000 UTC m=+0.036136053 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:00:08.754 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:08 smithi028 podman[99043]: 2024-04-26 02:00:08.509594917 +0000 UTC m=+0.155334085 container create 61ede65de080a117ae2637fa1cf8fbb3cc039f3266bc9e598bbf2b4f36e0bdbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T02:00:08.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:08 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:08.651+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:08.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:08 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:08.651+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:08.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:08 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:08.651+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:08.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:08 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:08.651+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:08.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:08 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:08.651+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:08.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:08 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:08.651+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:09.033 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:08 smithi028 podman[99043]: 2024-04-26 02:00:08.754935746 +0000 UTC m=+0.400674957 container init 61ede65de080a117ae2637fa1cf8fbb3cc039f3266bc9e598bbf2b4f36e0bdbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-26T02:00:09.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:08 smithi028 podman[99043]: 2024-04-26 02:00:08.767980416 +0000 UTC m=+0.413719557 container start 61ede65de080a117ae2637fa1cf8fbb3cc039f3266bc9e598bbf2b4f36e0bdbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T02:00:09.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:08 smithi028 podman[99043]: 2024-04-26 02:00:08.836956464 +0000 UTC m=+0.482695678 container attach 61ede65de080a117ae2637fa1cf8fbb3cc039f3266bc9e598bbf2b4f36e0bdbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T02:00:09.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:08 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:08.783+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:09.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:08 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:08.783+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:09.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:08 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:08.783+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:09.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:08 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:08.783+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:09.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:08 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:08.783+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:09.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:08 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:08.783+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:09.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:08 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:08.783+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:09.533 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 26 02:00:09 smithi028 podman[99043]: 2024-04-26 02:00:09.235750308 +0000 UTC m=+0.881489440 container died 61ede65de080a117ae2637fa1cf8fbb3cc039f3266bc9e598bbf2b4f36e0bdbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:00:09.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T02:00:10.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:10.000 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T02:00:10.001 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T02:00:10.001 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.10 2024-04-26T02:00:10.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:09 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:09.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:10.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:09 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:09.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:10.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:09 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:09.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:10.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:09 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:09.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:10.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:09 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:09.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:10.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:09 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:09.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:10.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:09.824+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:10.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:09.824+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:10.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:09.824+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:10.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:09.824+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:10.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:09.824+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:10.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:09.824+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:10.222 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:09 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:09.824+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:10.222 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 systemd[1]: Stopping Ceph osd.10 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:10.720 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.326+0000 7f905482f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:10.720 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.326+0000 7f905482f700 -1 osd.10 118 *** Got signal Terminated *** 2024-04-26T02:00:10.720 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.326+0000 7f905482f700 -1 osd.10 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:00:10.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:10.662+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:10.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:10.662+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:10.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:10.662+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:10.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:10.662+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:10.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:10.662+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:10.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:10 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:10.662+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:11.177 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.812+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:11.177 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.812+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:11.177 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.812+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:11.177 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.812+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:11.177 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.812+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:11.177 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.812+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:11.177 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:10 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:10.812+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:12.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:11.699+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:12.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:11.699+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:12.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:11.699+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:12.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:11.699+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:12.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:11.699+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:12.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:11.699+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:12.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:11.699+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:12.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:11.798+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:12.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:11.798+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:12.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:11.798+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:12.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:11.798+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:12.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:11.798+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:12.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:11.798+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:12.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:11 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:11.798+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:13.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:12 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:12.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:13.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:12 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:12.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:13.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:12 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:12.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:13.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:12 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:12.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:13.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:12 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:12.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:13.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:12 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:12.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:13.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:12 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:12.681+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:13.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:12 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:12.807+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:13.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:12 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:12.807+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:13.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:12 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:12.807+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:13.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:12 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:12.807+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:13.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:12 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:12.807+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:13.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:12 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:12.807+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:13.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:12 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:12.807+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:14.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:13.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:14.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:13.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:14.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:13.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:14.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:13.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:14.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:13.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:14.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:13.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:14.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:13 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:13.690+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:14.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:13 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:13.763+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:14.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:13 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:13.763+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:14.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:13 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:13.763+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:14.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:13 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:13.763+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:14.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:13 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:13.763+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:14.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:13 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:13.763+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:14.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:13 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:13.763+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:15.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:14.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:15.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:14.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:15.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:14.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:15.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:14.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:15.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:14.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:15.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:14.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:15.163 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:14 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:14.671+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:15.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:14 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:14.767+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.904481+0000 front 2024-04-26T01:58:46.904724+0000 (oldest deadline 2024-04-26T01:59:10.905071+0000) 2024-04-26T02:00:15.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:14 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:14.767+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.007606+0000 front 2024-04-26T01:58:56.007707+0000 (oldest deadline 2024-04-26T01:59:18.907735+0000) 2024-04-26T02:00:15.220 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:14 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:14.767+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:05.309552+0000 front 2024-04-26T01:59:05.309442+0000 (oldest deadline 2024-04-26T01:59:29.409273+0000) 2024-04-26T02:00:15.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:14 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:14.767+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-26T01:59:11.710251+0000 front 2024-04-26T01:59:11.710620+0000 (oldest deadline 2024-04-26T01:59:37.010346+0000) 2024-04-26T02:00:15.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:14 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:14.767+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:21.711495+0000 front 2024-04-26T01:59:21.711542+0000 (oldest deadline 2024-04-26T01:59:47.611622+0000) 2024-04-26T02:00:15.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:14 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:14.767+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:27.612200+0000 front 2024-04-26T01:59:27.612242+0000 (oldest deadline 2024-04-26T01:59:52.911961+0000) 2024-04-26T02:00:15.221 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:14 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10[57290]: 2024-04-26T02:00:14.767+0000 7f904ce1b700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.213027+0000 front 2024-04-26T01:59:38.213163+0000 (oldest deadline 2024-04-26T02:00:04.112681+0000) 2024-04-26T02:00:15.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:15.657+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:15.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:15.657+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:15.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:15.657+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:15.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:15.657+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:15.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:15.657+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:15.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:15.657+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:15.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:15 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:15.657+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:16.174 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:15 smithi053 podman[70406]: 2024-04-26 02:00:15.873524158 +0000 UTC m=+5.674454227 container died e6c45ab53b47493a9c19c9488f73fe2abacc8406dd1f46b30a2f5434e157509e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:00:16.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:16 smithi053 podman[70406]: 2024-04-26 02:00:16.259574285 +0000 UTC m=+6.060504363 container cleanup e6c45ab53b47493a9c19c9488f73fe2abacc8406dd1f46b30a2f5434e157509e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-26T02:00:16.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:16 smithi053 bash[70406]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10 2024-04-26T02:00:16.788 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:16 smithi053 podman[70423]: 2024-04-26 02:00:16.477291073 +0000 UTC m=+0.602472230 container remove e6c45ab53b47493a9c19c9488f73fe2abacc8406dd1f46b30a2f5434e157509e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-26T02:00:17.070 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:16 smithi053 podman[70513]: 2024-04-26 02:00:16.693728347 +0000 UTC m=+0.018690292 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:00:17.070 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:16 smithi053 podman[70513]: 2024-04-26 02:00:16.803189298 +0000 UTC m=+0.128151242 container create 1c7e78692f263616903b172e4635b67b9670ffe22f5179d7730820bf924e4ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T02:00:17.070 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:16 smithi053 podman[70513]: 2024-04-26 02:00:16.997538725 +0000 UTC m=+0.322500672 container init 1c7e78692f263616903b172e4635b67b9670ffe22f5179d7730820bf924e4ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T02:00:17.070 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:17 smithi053 podman[70513]: 2024-04-26 02:00:17.000795722 +0000 UTC m=+0.325757674 container start 1c7e78692f263616903b172e4635b67b9670ffe22f5179d7730820bf924e4ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:00:17.070 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:17 smithi053 podman[70513]: 2024-04-26 02:00:17.072302789 +0000 UTC m=+0.397264743 container attach 1c7e78692f263616903b172e4635b67b9670ffe22f5179d7730820bf924e4ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:00:17.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:16.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:17.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:16.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:17.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:16.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:17.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:16.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:17.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:16.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:17.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:16.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:17.162 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:16 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:16.674+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:17.470 INFO:journalctl@ceph.osd.10.smithi053.stdout:Apr 26 02:00:17 smithi053 podman[70513]: 2024-04-26 02:00:17.311425037 +0000 UTC m=+0.636386982 container died 1c7e78692f263616903b172e4635b67b9670ffe22f5179d7730820bf924e4ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T02:00:17.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:17.656+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:17.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:17.656+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:17.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:17.656+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:17.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:17.656+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:17.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:17.656+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:17.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:17.656+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:17.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:17 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:17.656+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:18.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T02:00:18.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:18.064 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T02:00:18.064 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T02:00:18.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.11 2024-04-26T02:00:18.384 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 systemd[1]: Stopping Ceph osd.11 for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:18.662 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.384+0000 7fcc0e0fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:18.662 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.384+0000 7fcc0e0fa700 -1 osd.11 118 *** Got signal Terminated *** 2024-04-26T02:00:18.662 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.384+0000 7fcc0e0fa700 -1 osd.11 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:00:18.662 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:18.662 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:18.662 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:18.662 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:18.663 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:18.663 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:18.663 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:18 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:18.620+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:19.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:19.594+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:19.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:19.594+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:19.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:19.594+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:19.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:19.594+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:19.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:19.594+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:19.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:19.594+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:19.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:19 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:19.594+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:20.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:20.546+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:20.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:20.546+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:20.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:20.546+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:20.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:20.546+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:20.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:20.546+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:20.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:20.546+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:20.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:20.546+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:20.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:20 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:20.546+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6820 osd.8 since back 2024-04-26T01:59:54.569987+0000 front 2024-04-26T01:59:54.569949+0000 (oldest deadline 2024-04-26T02:00:20.469939+0000) 2024-04-26T02:00:21.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:21.502+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:21.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:21.502+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:21.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:21.502+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:21.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:21.502+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:21.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:21.502+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:21.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:21.502+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:21.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:21.502+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:21.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:21 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:21.502+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6820 osd.8 since back 2024-04-26T01:59:54.569987+0000 front 2024-04-26T01:59:54.569949+0000 (oldest deadline 2024-04-26T02:00:20.469939+0000) 2024-04-26T02:00:22.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:22.521+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-26T01:58:46.360257+0000 front 2024-04-26T01:58:46.360314+0000 (oldest deadline 2024-04-26T01:59:13.360730+0000) 2024-04-26T02:00:22.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:22.521+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-04-26T01:58:56.862257+0000 front 2024-04-26T01:58:56.861980+0000 (oldest deadline 2024-04-26T01:59:20.961842+0000) 2024-04-26T02:00:22.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:22.521+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-04-26T01:59:00.962219+0000 front 2024-04-26T01:59:00.962244+0000 (oldest deadline 2024-04-26T01:59:26.862249+0000) 2024-04-26T02:00:22.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:22.521+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-04-26T01:59:23.066030+0000 front 2024-04-26T01:59:23.066113+0000 (oldest deadline 2024-04-26T01:59:47.165992+0000) 2024-04-26T02:00:22.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:22.521+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-04-26T01:59:28.266695+0000 front 2024-04-26T01:59:28.266778+0000 (oldest deadline 2024-04-26T01:59:52.966890+0000) 2024-04-26T02:00:22.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:22.521+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-26T01:59:38.868230+0000 front 2024-04-26T01:59:38.868063+0000 (oldest deadline 2024-04-26T02:00:05.268341+0000) 2024-04-26T02:00:22.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:22.521+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-04-26T01:59:45.269009+0000 front 2024-04-26T01:59:45.269102+0000 (oldest deadline 2024-04-26T02:00:11.168810+0000) 2024-04-26T02:00:22.913 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:22 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11[58725]: 2024-04-26T02:00:22.521+0000 7fcc066e6700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.169:6820 osd.8 since back 2024-04-26T01:59:54.569987+0000 front 2024-04-26T01:59:54.569949+0000 (oldest deadline 2024-04-26T02:00:20.469939+0000) 2024-04-26T02:00:24.217 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:23 smithi169 podman[72358]: 2024-04-26 02:00:23.947448621 +0000 UTC m=+5.692453578 container stop 93a3436c6667bf7462246de56c38bc94544d54ec6cd031557f0ddecaa7b46c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T02:00:24.217 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:23 smithi169 podman[72358]: 2024-04-26 02:00:23.947897665 +0000 UTC m=+5.692902617 container died 93a3436c6667bf7462246de56c38bc94544d54ec6cd031557f0ddecaa7b46c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-26T02:00:24.500 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:24 smithi169 podman[72358]: 2024-04-26 02:00:24.318144637 +0000 UTC m=+6.063149588 container cleanup 93a3436c6667bf7462246de56c38bc94544d54ec6cd031557f0ddecaa7b46c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-26T02:00:24.500 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:24 smithi169 bash[72358]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11 2024-04-26T02:00:24.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:24 smithi169 podman[72375]: 2024-04-26 02:00:24.502638391 +0000 UTC m=+0.544965283 container remove 93a3436c6667bf7462246de56c38bc94544d54ec6cd031557f0ddecaa7b46c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-26T02:00:25.237 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:24 smithi169 podman[72468]: 2024-04-26 02:00:24.825520392 +0000 UTC m=+0.018310019 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:00:25.237 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:24 smithi169 podman[72468]: 2024-04-26 02:00:24.944939378 +0000 UTC m=+0.137728999 container create a25cd69bc2f29e803dbe9a46e0302d924984388d0c61f05ee205390390f2f28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-26T02:00:25.237 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:25 smithi169 podman[72468]: 2024-04-26 02:00:25.155798285 +0000 UTC m=+0.348587905 container init a25cd69bc2f29e803dbe9a46e0302d924984388d0c61f05ee205390390f2f28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T02:00:25.237 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:25 smithi169 podman[72468]: 2024-04-26 02:00:25.158807902 +0000 UTC m=+0.351597523 container start a25cd69bc2f29e803dbe9a46e0302d924984388d0c61f05ee205390390f2f28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-26T02:00:25.237 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:25 smithi169 podman[72468]: 2024-04-26 02:00:25.238950264 +0000 UTC m=+0.431739915 container attach a25cd69bc2f29e803dbe9a46e0302d924984388d0c61f05ee205390390f2f28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T02:00:25.662 INFO:journalctl@ceph.osd.11.smithi169.stdout:Apr 26 02:00:25 smithi169 podman[72468]: 2024-04-26 02:00:25.455142662 +0000 UTC m=+0.647932286 container died a25cd69bc2f29e803dbe9a46e0302d924984388d0c61f05ee205390390f2f28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-26T02:00:26.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T02:00:26.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:26.104 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T02:00:26.104 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T02:00:26.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.a 2024-04-26T02:00:26.533 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 02:00:26 smithi028 systemd[1]: Stopping Ceph mds.a for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:27.033 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 02:00:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a[68175]: 2024-04-26T02:00:26.708+0000 7fbb7c000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:27.033 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 02:00:26 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a[68175]: 2024-04-26T02:00:26.708+0000 7fbb7c000700 -1 mds.a *** got signal Terminated *** 2024-04-26T02:00:31.033 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 02:00:30 smithi028 podman[99161]: 2024-04-26 02:00:30.7265759 +0000 UTC m=+4.166016694 container died 984a7114bd058e87596344d50f76734e525bdca4ff87647edee5a445c454c1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-26T02:00:31.391 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 02:00:31 smithi028 podman[99161]: 2024-04-26 02:00:31.166401281 +0000 UTC m=+4.605842068 container cleanup 984a7114bd058e87596344d50f76734e525bdca4ff87647edee5a445c454c1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T02:00:31.391 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 02:00:31 smithi028 bash[99161]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a 2024-04-26T02:00:31.783 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 26 02:00:31 smithi028 podman[99175]: 2024-04-26 02:00:31.392492888 +0000 UTC m=+0.642616585 container remove 984a7114bd058e87596344d50f76734e525bdca4ff87647edee5a445c454c1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:00:31.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T02:00:31.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:31.990 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T02:00:31.990 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T02:00:31.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.d 2024-04-26T02:00:32.533 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 02:00:32 smithi028 systemd[1]: Stopping Ceph mds.d for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:33.033 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 02:00:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d[66943]: 2024-04-26T02:00:32.635+0000 7f6667e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:33.033 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 02:00:32 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d[66943]: 2024-04-26T02:00:32.635+0000 7f6667e00700 -1 mds.d *** got signal Terminated *** 2024-04-26T02:00:36.033 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 02:00:35 smithi028 podman[99280]: 2024-04-26 02:00:35.651675061 +0000 UTC m=+3.187556729 container died 83f859f873459a737475812aa86d9743bc2251216fa4678613cc5b7af5a1d076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T02:00:36.428 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 02:00:36 smithi028 podman[99280]: 2024-04-26 02:00:36.136703169 +0000 UTC m=+3.672584843 container cleanup 83f859f873459a737475812aa86d9743bc2251216fa4678613cc5b7af5a1d076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T02:00:36.428 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 02:00:36 smithi028 bash[99280]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d 2024-04-26T02:00:36.678 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 26 02:00:36 smithi028 podman[99294]: 2024-04-26 02:00:36.429020624 +0000 UTC m=+0.761212860 container remove 83f859f873459a737475812aa86d9743bc2251216fa4678613cc5b7af5a1d076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T02:00:36.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T02:00:36.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:36.809 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T02:00:36.809 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T02:00:36.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.g 2024-04-26T02:00:37.283 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 02:00:36 smithi028 systemd[1]: Stopping Ceph mds.g for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:37.783 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 02:00:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g[67762]: 2024-04-26T02:00:37.455+0000 7f365f800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:37.784 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 02:00:37 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g[67762]: 2024-04-26T02:00:37.455+0000 7f365f800700 -1 mds.g *** got signal Terminated *** 2024-04-26T02:00:39.033 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 02:00:38 smithi028 podman[99399]: 2024-04-26 02:00:38.744750057 +0000 UTC m=+1.425140105 container died 5127c778750182e1025f0b05561b3da4a7a0e3cc8700dd8e295a5fd731aa7dcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T02:00:39.448 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 02:00:39 smithi028 podman[99399]: 2024-04-26 02:00:39.156696252 +0000 UTC m=+1.837086293 container cleanup 5127c778750182e1025f0b05561b3da4a7a0e3cc8700dd8e295a5fd731aa7dcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-26T02:00:39.448 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 02:00:39 smithi028 bash[99399]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g 2024-04-26T02:00:39.783 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 26 02:00:39 smithi028 podman[99414]: 2024-04-26 02:00:39.449591073 +0000 UTC m=+0.685324096 container remove 5127c778750182e1025f0b05561b3da4a7a0e3cc8700dd8e295a5fd731aa7dcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T02:00:39.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T02:00:40.070 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:40.071 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T02:00:40.071 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T02:00:40.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.j 2024-04-26T02:00:40.534 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 02:00:40 smithi028 systemd[1]: Stopping Ceph mds.j for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:41.033 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 02:00:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j[66526]: 2024-04-26T02:00:40.684+0000 7ff187000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:41.033 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 02:00:40 smithi028 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j[66526]: 2024-04-26T02:00:40.684+0000 7ff187000700 -1 mds.j *** got signal Terminated *** 2024-04-26T02:00:42.823 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 02:00:42 smithi028 podman[99521]: 2024-04-26 02:00:42.570350577 +0000 UTC m=+2.018120426 container died 2492a7c9273af25763b1028870af812524a614b776c14e623bfd895ab0f4e742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:00:43.193 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 02:00:42 smithi028 podman[99521]: 2024-04-26 02:00:42.980104481 +0000 UTC m=+2.427874324 container cleanup 2492a7c9273af25763b1028870af812524a614b776c14e623bfd895ab0f4e742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-26T02:00:43.193 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 02:00:42 smithi028 bash[99521]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j 2024-04-26T02:00:43.533 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 26 02:00:43 smithi028 podman[99546]: 2024-04-26 02:00:43.194770452 +0000 UTC m=+0.606304405 container remove 2492a7c9273af25763b1028870af812524a614b776c14e623bfd895ab0f4e742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-26T02:00:43.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T02:00:43.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:43.792 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T02:00:43.792 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T02:00:43.792 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.b 2024-04-26T02:00:44.099 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 02:00:43 smithi053 systemd[1]: Stopping Ceph mds.b for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:44.373 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 02:00:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b[62669]: 2024-04-26T02:00:44.101+0000 7fa4a11f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:44.374 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 02:00:44 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b[62669]: 2024-04-26T02:00:44.101+0000 7fa4a11f1700 -1 mds.b *** got signal Terminated *** 2024-04-26T02:00:46.970 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 02:00:46 smithi053 podman[70618]: 2024-04-26 02:00:46.681217406 +0000 UTC m=+2.692948318 container died b9e03549be77c1d36178e58f7ea1f62a7b31450ca5c6949049635f1662a91c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-26T02:00:47.470 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 02:00:47 smithi053 podman[70618]: 2024-04-26 02:00:47.126227443 +0000 UTC m=+3.137958403 container cleanup b9e03549be77c1d36178e58f7ea1f62a7b31450ca5c6949049635f1662a91c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:00:47.470 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 02:00:47 smithi053 bash[70618]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b 2024-04-26T02:00:47.470 INFO:journalctl@ceph.mds.b.smithi053.stdout:Apr 26 02:00:47 smithi053 podman[70644]: 2024-04-26 02:00:47.302432888 +0000 UTC m=+0.614872711 container remove b9e03549be77c1d36178e58f7ea1f62a7b31450ca5c6949049635f1662a91c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:00:47.535 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T02:00:47.571 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:47.572 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T02:00:47.572 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T02:00:47.572 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.e 2024-04-26T02:00:47.903 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 02:00:47 smithi053 systemd[1]: Stopping Ceph mds.e for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:48.220 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 02:00:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e[62039]: 2024-04-26T02:00:47.904+0000 7f2f543bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:48.220 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 02:00:47 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e[62039]: 2024-04-26T02:00:47.904+0000 7f2f543bd700 -1 mds.e *** got signal Terminated *** 2024-04-26T02:00:50.778 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 02:00:50 smithi053 podman[70749]: 2024-04-26 02:00:50.523065538 +0000 UTC m=+2.726001048 container died 998214597ce2f89b16d390535398e8b7defa799ca2111d2174aa95b9272bd75c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-26T02:00:51.154 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 02:00:50 smithi053 podman[70749]: 2024-04-26 02:00:50.880238631 +0000 UTC m=+3.083174149 container cleanup 998214597ce2f89b16d390535398e8b7defa799ca2111d2174aa95b9272bd75c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-26T02:00:51.154 INFO:journalctl@ceph.mds.e.smithi053.stdout:Apr 26 02:00:50 smithi053 bash[70749]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-e 2024-04-26T02:00:51.305 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T02:00:51.338 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:51.339 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T02:00:51.339 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T02:00:51.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.h 2024-04-26T02:00:51.666 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 02:00:51 smithi053 systemd[1]: Stopping Ceph mds.h for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:51.666 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 02:00:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h[62354]: 2024-04-26T02:00:51.667+0000 7f52d4318700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:51.666 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 02:00:51 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h[62354]: 2024-04-26T02:00:51.667+0000 7f52d4318700 -1 mds.h *** got signal Terminated *** 2024-04-26T02:00:55.674 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 02:00:55 smithi053 podman[70870]: 2024-04-26 02:00:55.354066713 +0000 UTC m=+3.793748067 container died 57bcc154dc479d8ce882e006502be1226f040bf840187c0198afcccfc8e5fe6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:00:55.705 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 02:00:55 smithi053 podman[70870]: 2024-04-26 02:00:55.675970514 +0000 UTC m=+4.115651867 container cleanup 57bcc154dc479d8ce882e006502be1226f040bf840187c0198afcccfc8e5fe6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T02:00:55.970 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 02:00:55 smithi053 bash[70870]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h 2024-04-26T02:00:55.970 INFO:journalctl@ceph.mds.h.smithi053.stdout:Apr 26 02:00:55 smithi053 podman[70884]: 2024-04-26 02:00:55.835482364 +0000 UTC m=+0.473371020 container remove 57bcc154dc479d8ce882e006502be1226f040bf840187c0198afcccfc8e5fe6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T02:00:56.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T02:00:56.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:00:56.092 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T02:00:56.092 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T02:00:56.092 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.k 2024-04-26T02:00:56.403 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 02:00:56 smithi053 systemd[1]: Stopping Ceph mds.k for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:00:56.720 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 02:00:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k[61715]: 2024-04-26T02:00:56.404+0000 7f28b413f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:00:56.720 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 02:00:56 smithi053 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k[61715]: 2024-04-26T02:00:56.404+0000 7f28b413f700 -1 mds.k *** got signal Terminated *** 2024-04-26T02:01:00.686 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 02:01:00 smithi053 podman[70989]: 2024-04-26 02:01:00.418548263 +0000 UTC m=+4.109952575 container died 7498e416a85af0e8868acabd2f2675d2ea35014c7ff6f6e57a9fd624685f1dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T02:01:00.970 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 02:01:00 smithi053 podman[70989]: 2024-04-26 02:01:00.787716699 +0000 UTC m=+4.479121018 container cleanup 7498e416a85af0e8868acabd2f2675d2ea35014c7ff6f6e57a9fd624685f1dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:01:00.970 INFO:journalctl@ceph.mds.k.smithi053.stdout:Apr 26 02:01:00 smithi053 bash[70989]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-k 2024-04-26T02:01:01.223 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T02:01:01.255 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:01:01.255 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T02:01:01.255 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T02:01:01.256 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.c 2024-04-26T02:01:01.533 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:01 smithi169 systemd[1]: Stopping Ceph mds.c for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:01:01.801 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:01 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c[63770]: 2024-04-26T02:01:01.650+0000 7ff6471cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:01:01.801 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:01 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c[63770]: 2024-04-26T02:01:01.650+0000 7ff6471cb700 -1 mds.c *** got signal Terminated *** 2024-04-26T02:01:02.939 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:02 smithi169 podman[72574]: 2024-04-26 02:01:02.688791119 +0000 UTC m=+1.246986433 container died 813d4dceafefe51e333d6517c5ac0bc542f7ea0f0125856ce819b3a029a83f3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:01:03.207 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:03 smithi169 podman[72574]: 2024-04-26 02:01:03.024845724 +0000 UTC m=+1.583041087 container cleanup 813d4dceafefe51e333d6517c5ac0bc542f7ea0f0125856ce819b3a029a83f3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T02:01:03.207 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:03 smithi169 bash[72574]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c 2024-04-26T02:01:03.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T02:01:03.474 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:03 smithi169 podman[72614]: 2024-04-26 02:01:03.209263814 +0000 UTC m=+0.516230931 container remove 813d4dceafefe51e333d6517c5ac0bc542f7ea0f0125856ce819b3a029a83f3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T02:01:03.474 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:03 smithi169 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.c.service: Deactivated successfully. 2024-04-26T02:01:03.474 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:03 smithi169 systemd[1]: Stopped Ceph mds.c for ead407d4-036b-11ef-bc93-c7b262605968. 2024-04-26T02:01:03.474 INFO:journalctl@ceph.mds.c.smithi169.stdout:Apr 26 02:01:03 smithi169 systemd[1]: ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.c.service: Consumed 9.715s CPU time. 2024-04-26T02:01:03.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:01:03.489 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T02:01:03.489 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T02:01:03.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.f 2024-04-26T02:01:03.912 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 02:01:03 smithi169 systemd[1]: Stopping Ceph mds.f for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:01:04.412 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 02:01:03 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f[63461]: 2024-04-26T02:01:03.919+0000 7f3d62b37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:01:04.412 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 02:01:03 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f[63461]: 2024-04-26T02:01:03.919+0000 7f3d62b37700 -1 mds.f *** got signal Terminated *** 2024-04-26T02:01:06.986 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 02:01:06 smithi169 podman[72721]: 2024-04-26 02:01:06.685189011 +0000 UTC m=+2.982167681 container died a2f9a35ac466d75335266c47bc8b24f8712d0462cabf409077c8d76462ad6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:01:07.286 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 02:01:07 smithi169 podman[72721]: 2024-04-26 02:01:07.078445067 +0000 UTC m=+3.375423736 container cleanup a2f9a35ac466d75335266c47bc8b24f8712d0462cabf409077c8d76462ad6a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T02:01:07.286 INFO:journalctl@ceph.mds.f.smithi169.stdout:Apr 26 02:01:07 smithi169 bash[72721]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-f 2024-04-26T02:01:07.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T02:01:07.455 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:01:07.456 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T02:01:07.456 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T02:01:07.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.i 2024-04-26T02:01:07.662 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 02:01:07 smithi169 systemd[1]: Stopping Ceph mds.i for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:01:08.162 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 02:01:07 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: 2024-04-26T02:01:07.776+0000 7f2011f3a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:01:08.162 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 02:01:07 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i[64080]: 2024-04-26T02:01:07.776+0000 7f2011f3a700 -1 mds.i *** got signal Terminated *** 2024-04-26T02:01:11.031 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 02:01:10 smithi169 podman[72842]: 2024-04-26 02:01:10.698068039 +0000 UTC m=+3.028053639 container died 36c61d61a6700436930f17f4cdb0757369d037fd27b72677cacde837364dcf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:01:11.306 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 02:01:11 smithi169 podman[72842]: 2024-04-26 02:01:11.090360811 +0000 UTC m=+3.420346407 container cleanup 36c61d61a6700436930f17f4cdb0757369d037fd27b72677cacde837364dcf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:01:11.306 INFO:journalctl@ceph.mds.i.smithi169.stdout:Apr 26 02:01:11 smithi169 bash[72842]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-i 2024-04-26T02:01:11.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T02:01:11.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:01:11.457 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T02:01:11.457 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T02:01:11.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.l 2024-04-26T02:01:11.662 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 02:01:11 smithi169 systemd[1]: Stopping Ceph mds.l for ead407d4-036b-11ef-bc93-c7b262605968... 2024-04-26T02:01:12.162 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 02:01:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l[63139]: 2024-04-26T02:01:11.775+0000 7f9551e67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:01:12.162 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 02:01:11 smithi169 ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l[63139]: 2024-04-26T02:01:11.775+0000 7f9551e67700 -1 mds.l *** got signal Terminated *** 2024-04-26T02:01:13.912 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 02:01:13 smithi169 podman[72962]: 2024-04-26 02:01:13.658195656 +0000 UTC m=+1.979497594 container died 5dc46c50602caa0a5fab21da8be9c037ce7224623950ab1bd75018b71caddefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:01:14.296 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 02:01:13 smithi169 podman[72962]: 2024-04-26 02:01:13.960590308 +0000 UTC m=+2.281892254 container cleanup 5dc46c50602caa0a5fab21da8be9c037ce7224623950ab1bd75018b71caddefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T02:01:14.296 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 02:01:13 smithi169 bash[72962]: ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l 2024-04-26T02:01:14.296 INFO:journalctl@ceph.mds.l.smithi169.stdout:Apr 26 02:01:14 smithi169 podman[72976]: 2024-04-26 02:01:14.178008505 +0000 UTC m=+0.515039655 container remove 5dc46c50602caa0a5fab21da8be9c037ce7224623950ab1bd75018b71caddefc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-ead407d4-036b-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T02:01:14.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ead407d4-036b-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T02:01:14.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:01:14.341 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T02:01:14.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ead407d4-036b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:01:14.675 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T02:01:19.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ead407d4-036b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:01:19.803 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T02:01:22.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ead407d4-036b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:01:22.462 INFO:teuthology.orchestra.run.smithi169.stdout:Deleting cluster with fsid: ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T02:01:24.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:01:24.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:01:24.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:01:24.910 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T02:01:24.912 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi028/crash 2024-04-26T02:01:24.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:01:25.012 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:01:25.012 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:01:25.018 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi053/crash 2024-04-26T02:01:25.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:01:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:tar: /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:01:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:01:25.058 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi169/crash 2024-04-26T02:01:25.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:01:25.099 INFO:teuthology.orchestra.run.smithi169.stderr:tar: /var/lib/ceph/ead407d4-036b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:01:25.099 INFO:teuthology.orchestra.run.smithi169.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:01:25.100 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T02:01:25.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-26T02:01:25.208 INFO:tasks.cephadm:Compressing logs... 2024-04-26T02:01:25.208 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:01:25.213 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:01:25.216 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:01:25.243 INFO:teuthology.orchestra.run.smithi169.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:01:25.243 INFO:teuthology.orchestra.run.smithi169.stderr:: No such file or directory 2024-04-26T02:01:25.243 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:01:25.243 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T02:01:25.243 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T02:01:25.251 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:01:25.251 INFO:teuthology.orchestra.run.smithi053.stderr:: No such file or directory 2024-04-26T02:01:25.251 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:01:25.251 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr: 90.4%/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.h.log: -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log 87.1% 2024-04-26T02:01:25.252 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.c.log: /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.1% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T02:01:25.254 INFO:teuthology.orchestra.run.smithi169.stderr: 90.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:01:25.255 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T02:01:25.255 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T02:01:25.255 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:01:25.255 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T02:01:25.255 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T02:01:25.255 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:01:25.259 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.10.log: 94.3% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T02:01:25.259 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T02:01:25.260 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:01:25.260 INFO:teuthology.orchestra.run.smithi053.stderr: 83.6% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:01:25.260 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T02:01:25.260 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log: 92.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T02:01:25.264 INFO:teuthology.orchestra.run.smithi053.stderr: 94.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T02:01:25.265 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T02:01:25.266 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T02:01:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:01:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:: No such file or directory 2024-04-26T02:01:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:01:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:01:25.272 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:01:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T02:01:25.273 INFO:teuthology.orchestra.run.smithi053.stderr: 94.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T02:01:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T02:01:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T02:01:25.276 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:01:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T02:01:25.276 INFO:teuthology.orchestra.run.smithi028.stderr: 90.3% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:01:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T02:01:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T02:01:25.278 INFO:teuthology.orchestra.run.smithi028.stderr: 93.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:01:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T02:01:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T02:01:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.6.log: /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log: 94.5% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T02:01:25.284 INFO:teuthology.orchestra.run.smithi028.stderr: 83.8% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:01:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T02:01:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:01:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.a.log: /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T02:01:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T02:01:25.293 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:01:25.294 INFO:teuthology.orchestra.run.smithi169.stderr: 94.5% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T02:01:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.3% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T02:01:25.316 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:01:25.584 INFO:teuthology.orchestra.run.smithi028.stderr: 93.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T02:01:25.652 INFO:teuthology.orchestra.run.smithi028.stderr: 93.7% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T02:01:25.781 INFO:teuthology.orchestra.run.smithi169.stderr: 93.7% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T02:01:25.808 INFO:teuthology.orchestra.run.smithi028.stderr: 90.1% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T02:01:25.858 INFO:teuthology.orchestra.run.smithi169.stderr: 91.7% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T02:01:25.964 INFO:teuthology.orchestra.run.smithi053.stderr: 91.9% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T02:01:27.048 INFO:teuthology.orchestra.run.smithi028.stderr: 90.1% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T02:01:27.294 INFO:teuthology.orchestra.run.smithi169.stderr: 94.6% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T02:01:27.952 INFO:teuthology.orchestra.run.smithi053.stderr: 94.0% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T02:01:29.759 INFO:teuthology.orchestra.run.smithi169.stderr: 93.6% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T02:01:59.812 INFO:teuthology.orchestra.run.smithi169.stderr: 91.3% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T02:01:59.900 INFO:teuthology.orchestra.run.smithi028.stderr: 91.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T02:02:07.957 INFO:teuthology.orchestra.run.smithi053.stderr: 91.8% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T02:02:11.622 INFO:teuthology.orchestra.run.smithi028.stderr: 91.3% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T02:02:11.741 INFO:teuthology.orchestra.run.smithi169.stderr: 91.3% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T02:02:11.950 INFO:teuthology.orchestra.run.smithi169.stderr: 91.5% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T02:02:13.383 INFO:teuthology.orchestra.run.smithi169.stderr: 91.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T02:02:13.384 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-26T02:02:13.385 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m48.155s 2024-04-26T02:02:13.385 INFO:teuthology.orchestra.run.smithi169.stderr:user 2m57.891s 2024-04-26T02:02:13.385 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m5.442s 2024-04-26T02:02:14.206 INFO:teuthology.orchestra.run.smithi053.stderr: 91.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T02:02:16.633 INFO:teuthology.orchestra.run.smithi028.stderr: 91.2% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T02:02:17.705 INFO:teuthology.orchestra.run.smithi053.stderr: 91.4% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T02:02:19.988 INFO:teuthology.orchestra.run.smithi053.stderr: 91.3% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T02:02:19.991 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-26T02:02:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m54.759s 2024-04-26T02:02:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:user 3m16.286s 2024-04-26T02:02:19.991 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m5.660s 2024-04-26T02:02:31.660 INFO:teuthology.orchestra.run.smithi028.stderr: 91.2% -- replaced with /var/log/ceph/ead407d4-036b-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T02:02:31.664 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-26T02:02:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:real 1m6.419s 2024-04-26T02:02:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:user 3m7.860s 2024-04-26T02:02:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m13.365s 2024-04-26T02:02:31.665 INFO:tasks.cephadm:Archiving logs... 2024-04-26T02:02:31.665 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi028/log 2024-04-26T02:02:31.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:03:06.115 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi053/log 2024-04-26T02:03:06.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:03:39.314 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi169/log 2024-04-26T02:03:39.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:04:09.659 INFO:tasks.cephadm:Removing cluster... 2024-04-26T02:04:09.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ead407d4-036b-11ef-bc93-c7b262605968 --force 2024-04-26T02:04:10.003 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T02:04:11.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ead407d4-036b-11ef-bc93-c7b262605968 --force 2024-04-26T02:04:11.541 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T02:04:12.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ead407d4-036b-11ef-bc93-c7b262605968 --force 2024-04-26T02:04:12.287 INFO:teuthology.orchestra.run.smithi169.stdout:Deleting cluster with fsid: ead407d4-036b-11ef-bc93-c7b262605968 2024-04-26T02:04:12.799 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T02:04:12.799 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:04:12.862 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:04:12.878 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:04:12.893 INFO:tasks.cephadm:Teardown complete 2024-04-26T02:04:12.894 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T02:04:12.906 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T02:04:12.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:04:12.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:04:12.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:04:13.079 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T02:04:13.090 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T02:04:13.090 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:04:13.098 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:04:13.100 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:04:13.113 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:04:13.114 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:04:13.117 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:04:13.117 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-04-26T02:04:13.117 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv01.front.sepia.ceph.com 3 6 377 29 +1725us[+1729us] +/- 45ms 2024-04-26T02:04:13.117 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 -1817us[-1817us] +/- 66ms 2024-04-26T02:04:13.117 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -662us[ -662us] +/- 75ms 2024-04-26T02:04:13.117 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:04:13.119 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:04:13.119 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-04-26T02:04:13.119 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +1685us[+1694us] +/- 45ms 2024-04-26T02:04:13.119 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 -1868us[-1859us] +/- 66ms 2024-04-26T02:04:13.119 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -712us[ -703us] +/- 75ms 2024-04-26T02:04:13.119 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:04:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:04:13.159 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:04:13.159 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-26T02:04:13.160 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 +1811us[+1836us] +/- 45ms 2024-04-26T02:04:13.160 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 95 -1701us[-1656us] +/- 65ms 2024-04-26T02:04:13.160 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 28 -533us[ -509us] +/- 76ms 2024-04-26T02:04:13.160 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:04:13.161 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T02:04:13.172 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T02:04:13.173 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T02:04:13.275 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:04:13.508 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:04:13.578 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:04:13.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:04:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:04:13.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:04:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:04:13.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:04:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:04:13.800 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T02:04:13.811 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T02:04:13.822 INFO:teuthology.task.internal:Duration was 2744.806206 seconds 2024-04-26T02:04:13.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T02:04:13.834 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T02:04:13.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:04:13.842 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:04:13.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:04:13.882 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:04:13.883 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:04:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:04:14.261 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T02:04:14.261 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-04-26T02:04:14.262 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:04:14.340 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-04-26T02:04:14.340 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:04:14.376 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-04-26T02:04:14.376 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:04:14.396 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T02:04:14.396 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:04:14.404 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:04:14.419 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:04:14.494 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T02:04:14.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:04:14.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:04:14.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:04:15.296 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T02:04:15.309 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T02:04:15.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:04:15.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:04:15.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:04:15.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T02:04:15.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:04:15.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:04:15.459 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:04:15.481 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-04-26T02:04:15.499 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-04-26T02:04:15.549 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-04-26T02:04:15.592 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:04:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:04:15.642 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:04:15.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:04:15.659 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:04:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:04:15.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T02:04:15.688 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T02:04:15.689 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi028 2024-04-26T02:04:15.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:04:15.809 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi053 2024-04-26T02:04:15.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:04:15.851 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235/remote/smithi169 2024-04-26T02:04:15.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:04:15.888 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T02:04:15.889 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:04:15.897 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:04:15.900 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:04:15.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T02:04:15.964 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T02:04:15.964 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T02:04:15.981 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T02:04:15.981 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:04:15.988 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:04:15.991 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:04:16.006 INFO:teuthology.orchestra.run.smithi169.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:04 /home/ubuntu/cephtest 2024-04-26T02:04:16.006 INFO:teuthology.orchestra.run.smithi053.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:04 /home/ubuntu/cephtest 2024-04-26T02:04:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:04 /home/ubuntu/cephtest 2024-04-26T02:04:16.044 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T02:04:16.057 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T02:04:16.083 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T02:04:16.109 DEBUG:teuthology.nuke:shortname: smithi028 2024-04-26T02:04:16.109 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:04:16.133 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:09:24.805897', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLo5ySqCtwejdXjowKaoioj1jhXs8OyavPeXWNu4uesAxD9iJ1xy2bkqb3bS9sAzr4/vsiz6LdgOzBgl8GVvhVc='} 2024-04-26T02:04:16.137 DEBUG:teuthology.nuke:shortname: smithi053 2024-04-26T02:04:16.137 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:04:16.165 DEBUG:teuthology.nuke:shortname: smithi169 2024-04-26T02:04:16.165 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:04:16.168 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:09:24.807718', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBH0CAbYukTmxxBRGplu/YJpHfq0PpyXRbfWS68BntPvvyKiRPX6vA7kuP0E79O/udz5E9lyE7KtnL2D/nuPCT4='} 2024-04-26T02:04:16.191 INFO:teuthology.orchestra.console.smithi028:Power off 2024-04-26T02:04:16.192 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:04:16.205 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:09:24.809314', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApA1IpWDbjAwBm0DgsXFeZ9YfP569BS90v2+vnZ1P7gaIKfjb192gH4qyIbw7WXNbySvGj6eiqUkItuhEzV3I4='} 2024-04-26T02:04:16.214 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-04-26T02:04:16.214 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:16.238 INFO:teuthology.orchestra.console.smithi053:Power off 2024-04-26T02:04:16.238 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:04:16.248 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-26T02:04:16.265 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-04-26T02:04:16.265 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:16.281 INFO:teuthology.orchestra.console.smithi169:Power off 2024-04-26T02:04:16.281 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:04:16.290 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-04-26T02:04:16.305 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-04-26T02:04:16.305 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:16.330 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-04-26T02:04:20.249 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:20.291 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:20.332 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:20.361 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-26T02:04:20.401 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-04-26T02:04:20.442 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-04-26T02:04:24.367 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:24.402 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:24.445 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:24.477 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-04-26T02:04:24.512 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-04-26T02:04:24.554 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-04-26T02:04:24.578 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-04-26T02:04:24.613 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-04-26T02:04:28.559 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:04:28.669 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-04-26T02:04:28.770 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-04-26T02:04:28.875 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} duration: 2744.8062064647675 failure_reason: 'error during quiesce thrashing: Error releasing set ''6288c6a'': 1 (EPERM)' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-04-26T02:04:28.876 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T02:04:28.962 INFO:teuthology.run:FAIL